Azure IoT Device Provisioning Service (DPS) over MQTT

Continuing the theme of “doing things on Azure IoT without using our SDKs”, this article describes how to provision IOT devices with Azure IoT’s Device Provisioning Service over raw MQTT.

Previously, I wrote an article that describes how to leverage Azure IoT’s Device Provisioning Service over its REST API, as well as an article about connecting to IoT Hub/Edge over raw MQTT.  Where possible, I do recommend using our SDKs, as they provide a nice abstraction layer over the supported transport protocols and frees you from all that protocol-level detailed work.  However, we understand there are times and reasons where it’s just a better fit to do things over the raw protocols.

To support this, the Azure IoT DPS engineering team has documented the necessary technical details to register your device via MQTT.   This document may provide enough details for you to figure out how to do it, but since I needed to test it for a customer anyway, I thought I’d capture a real-world example in hopes it can help others.

To make the scenario simpler, I chose to just use symmetric key attestation, but this would still work with any of the attestation methods supported by DPS.

Create individual enrollment

The first step is to create the enrollment in DPS.  In the Azure portal, in your DPS instance, from the ‘overview’ tab, grab your Scope ID from the upper right of the ‘overview’ tab as shown below (I’ve blacked out part of my details, for obvious reasons)

dps-scope-id

Once you have that, copy it somewhere like notepad or equivalent, we’ll use it later.  Once we have that, we can create our enrollment.  On the left nav, click on “Manage Enrollments” and then “Add Individual Enrollment”.  For “Mechanism”, choose Symmetric Key, enter a registration ID of your choosing (for the example further below, I used ‘my-mqtt-dev01’)

create-individual-enrollment

Click Save.  Then drill back into your enrollment in the portal and copy the “Primary Key”  and save it for later use.

Generate SAS token

Once you’ve created the enrollment and gotten the device key, we need to generate a SAS token for authentication to the DPS service.  A description of the SAS token, and several code samples for generating one in various languages can be found here.  Some of the inputs (discussed below) will be different for DPS versus IoT Hub, but the basic structure of the SAS token is the same.

For my purposes, I used this python code to generate mine:

————-

from base64 import b64encode, b64decode
from hashlib import sha256
from time import time
from urllib import quote_plus, urlencode
from hmac import HMAC

def generate_sas_token(uri, key, policy_name, expiry=3600000000):
ttl = time() + expiry
sign_key = “%s\n%d” % ((quote_plus(uri)), int(ttl))
print(sign_key)
signature = b64encode(HMAC(b64decode(key), sign_key, sha256).digest())

rawtoken = {
‘sr’ :  uri,
‘sig’: signature,
‘se’ : str(int(ttl))
}

if policy_name is not None:
rawtoken[‘skn’] = policy_name

return ‘SharedAccessSignature ‘ + urlencode(rawtoken)

uri = ‘[dps URI]’
key = ‘[device key]’
expiry = [SAS token duration]
policy=’registration’

print(generate_sas_token(uri, key, policy, expiry))

——–

where:

  • [dps URI] is of the form [DPS scope id]/registrations/[registration id]
  • [device key] is the primary key you saved earlier
  • [SAS token duration] is the number of seconds you want the token to be valid for
  • policy is required to be ‘registration’ for DPS SAS tokens

running this code will give you a SAS token that looks something like this (changing a few random characters to protect my DPS):

SharedAccessSignature sr=0ne00055505%2Fregistrations%2Fmy-mqtt-dev01&skn=registration&sig=gMpllKo7qS1VR31vyfsT6JAcc4%2BHIu2gQSyai0Uz0KM%3D&se=1579698526

Now that we have our authentication credentials, we are ready to make our MQTT call.

Example call

The documentation does a decent job of showing the MQTT parameters and flow (read it first!), so I’m not going to repeat that here.  What I will show is an example call with screenshots to ‘make it real’.   For my testing, I used mqtt.fx, which is a pretty nice little interactive MQTT test client.

Once you download and install it,  click on the little lightning bolt to switch from localhost to allow you to create a new connection to an MQTT server.

mqtt-lightning

After that, click on the settings symbol next to the edit box to open the settings dialog that lets you edit the various connection profiles:

mqttfx-settings-icon

On the “Edit Connection Profiles” dialog, in the very bottom left hand corner, click the “+” symbol to create a new connection profile.

Give your connection a name and choose MQTT Broker as the Profile Type

mqtt-profile-settings-general

Enter the following settings in the top half of the dialog:

  • for “Broker Address”, use ‘global.azure-devices-provisioning.net’
  • for “Broker Port”, use “8883”
  • for Client ID, enter your registration ID you used in the portal for your device

Click on the General ‘tab’ at the bottom.  As in the screenshot above, for MQTT Version, uncheck the “Use Default” button and explicitly choose version 3.1.1.  Leave other settings on this tab alone.

click on the “User Credentials” tab’

  • for “User Name”, enter [DPS Scope Id]/registrations/[registration id]/api-version=2019-03-31  (replacing the scope id and registration id with your values)
  • for “Password”, copy/paste in your SAS token you generated earlier

mqtt-profile-user-creds

Move to the SSL/TLS tab.   Check the box for “Enable SSL/TLS” and make sure that TLSv1.2 is chosen as the protocol

mqtt-profile-tls

leave the proxy and LWT tabs alone.

Click Ok to save the settings and return to the main screen

Click on the Connect button and you should get a successful connection (you can verify by looking at the “log” tab)

Once connected, navigate to the “Subscribe” tab.  We will set up a subscription on the dps ‘response’ MQTT topic to receive responses to our registration attempts from DPS.  On the “Subscribe” tab, enter ‘$dps/registrations/res/#’ into the subscriptions box, choose “QoS1” from the buttons on the right, and click “Subscribe”.  You should see an active subscription get set up and waiting on responses.

mqtt-subscription-setup

Click back over on the “Publish” tab and we will make our registration attempt.  In the publish edit box, enter $dps/registrations/PUT/iotdps-register/?$rid={request_id}

replace {request_id} with an integer of your choosing (1 is fine to start with).  This lets us correlate requests with responses when we get responses back from the service.  For example, I entered:

$dps/registrations/PUT/iotdps-register/?$rid=1

in the big edit box beneath the publish edit box, we need to enter a ‘payload’ for the request.  For DPS registration requests, the payload takes the form of a JSON document like this:  {“registrationId”:”<registration id>”}

for example, for my sample it’s:

{“registrationId”: “my-mqtt-dev01”}

mqtt-reg-publish

Hit the “Publish button”

Flip back over to the Subscribe tab and you should see on the right hand side of the screen that we’ve received a response from DPS.  You should see something like this:

mqtt-registration-assigning

This indicates that DPS is in the process of ‘assigning’ and registering our device to an IoT Hub.  This is a potentially long running operation, so to get the status of it, we have to query for that status.  To do that, we are going to publish another MQTT message to check on the status.  For that, we need the ‘operationId’ from the message we just received.  In the screenshot above, mine looks like this:

4.22724a0213a69c4d.9750f5e6-b4c3-4760-9b15-4e74d6120bd1

Copy that ID as we’ll use it in the next step.

To check on the status of the operation, switch back over to the Publish tab and replace the values in the publish edit box with this

$dps/registrations/GET/iotdps-get-operationstatus/?$rid={request_id}&operationId={operationId}

replacing {request_id} with a new request id (2 in my case) and the {operationId} with the operationId you just copied. For example, with my sample values and the response received above, my request looks like this:

$dps/registrations/GET/iotdps-get-operationstatus/?$rid=2&operationId=4.22724a0213a69c4d.9750f5e6-b4c3-4760-9b15-4e74d6120bd1

Delete the JSON in the payload box and click “publish”

Switch back over to the Subscribe tab and you should notice that you’ve received a response to your operational status query, similar to this:

mqtt-registration-status

Notice the status of “assigned”, as well as details like “assignedHub” that gives the state of the successful registration and connection details.

If you navigate back over to the azure portal and look at the enrollment record for your device (refresh the page.. you may have to exit and re-enter), you should see something like this:

mqtt-registration-success

This indicates that our DPS registration was successful.

In the “real world”, in your application, you’ll make the registration attempt and then poll the operational status until it gets to the state of ‘assigned’.  There will be intermediate states while it is being assigned, but doing this manually through a GUI, I’m not fast enough to catch them Smile

Enjoy – and let me know in the comments if you have any questions or issues.

Azure IoT Device Provisioning Service via REST–part 2

This is part 2 of a two part post on provisioning IoT devices to Azure IoT Hub via the Azure IoT Device Provisioning Service (DPS) via its REST API.  Part 1 described the process for doing it with x.509 certificate attestation from devices and this part will describe doing it with Symmetric Key attestation. 

I won’t repeat all the introduction, caveats, etc. that accompanied part 1, but you may want to take a quick peek at them so you know what I will, and will not, be covering here.

If you don’t fully understand the Symmetric Key attestation options for DPS, I recommend you go read the docs here first and then come back…

Ok, welcome back!

So let’s just jump right in.  Similarly to part 1, there will be a couple of sections of ‘setup’, depending on whether you choose to go with Individual Enrollments or Group Enrollments in DPS.  Once that is done, and the accompanying attestation tokens are generated, the actual API calls are identical between the two.

Therefore, for the first two sections, you can choose the one that matches your desired enrollment type and read it for the required setup (or read them both if you are the curious type), then you can just jump to the bottom section for the actual API calls.

But, before we start with setup, there’s a little prep work to do.

Prep work (aka – how do I generate the SAS tokens?)

Symmetric Key attestation in DPS works, just like pretty much all the rest of Azure, on the concept of SAS tokens.  In Azure IoT, these tokens are typically derived from a cryptographic key tied to a device or service-access level.  As mentioned in the overview link above (in case you didn’t read it), DPS have two options for these keys.  One option is an individual key per device, as specified or auto-generated in the individual enrollments.  The other option is to have a group enrollment key, from which you derive a device-specific key, that you leverage for your SAS token generation.

Generating SAS tokens

So first, let’s talk about and prep for the generation of our SAS tokens, independent of what kind of key we use.   The use of, and generation of, SAS tokens is generally the same for both DPS and IoT Hub, so you can see the process and sample code in various languages here.  For my testing, I pretty much shamelessly stole re-used the python example from that page, which I slightly modified (to actually call the generate_sas_token method).

from base64 import b64encode, b64decode
from hashlib import sha256
from time import time
from urllib import quote_plus, urlencode
from hmac import HMAC

def generate_sas_token(uri, key, policy_name, expiry=3600):
     ttl = time() + expiry
     sign_key = “%s\n%d” % ((quote_plus(uri)), int(ttl))
     print sign_key
     signature = b64encode(HMAC(b64decode(key), sign_key, sha256).digest())

    rawtoken = {
         ‘sr’ :  uri,
         ‘sig’: signature,
         ‘se’ : str(int(ttl))
     }

    if policy_name is not None:
         rawtoken[‘skn’] = policy_name

    return ‘SharedAccessSignature ‘ + urlencode(rawtoken)

uri = ‘[resource_uri]’
key = ‘[device_key]’
expiry = [expiry_in_seconds]
policy=’[policy]’

print generate_sas_token(uri, key, policy, expiry)

the parameters at the bottom of the script, which I hardcoded because I am lazy busy, are as follows:

  • [resource_uri] – this is the URI of the resource you are trying to reach with this token.  For DPS, it is of the form ‘[dps_scope_id]/registrations/[dps_registration_id]’, where [dps_scope_id] is the scope id associated with your DPS instance, found on the overview blade of your DPS instance in the Azure portal, and [dps_registration_id] is the registration_id you want to use for your device.  It will be whatever you specified in an individual enrollment in DPS, or can be anything you want in a group enrollment as long as it is unique.  Frequently used ideas here are combinations of serial numbers, MAC addresses, GUIDs, etc
  • [device_key] is the device key associated with your device.  This is either the one specified or auto-generated for you in an individual enrollment, or a derived key for a group enrollment, as explained a little further below
  • [expiry_in_seconds] the validity period of this SAS token in sec…   ok, not going to insult your intelligence here
  • [policy] the policy with which the key above is associated.  For DPS device registration, this is hard coded to ‘registration’

So an example set of inputs for a device called ‘dps-sym-key-test01’ might look like this (with the scope id and key modified to protect my DPS instance from the Russians!)

uri = ‘0ne00057505/registrations/dps-sym-key-test01’
key = ‘gPD2SOUYSOMXygVZA+pupNvWckqaS3Qnu+BUBbw7TbIZU7y2UZ5ksp4uMJfdV+nTIBayN+fZIZco4tS7oeVR/A==’
expiry = 3600000
policy=’registration’

Save the script above to a *.py file.   (obviously, you’ll need to install python 2.7 or better if you don’t have it to run the script)

If you are only doing individual enrollments, you can skip the next section, unless you are just curious.

Generating derived keys

For group enrollments, you don’t have individual enrollment records for devices in the DPS enrollments, so therefore you don’t have individual device keys.  To make this work, we take the enrollment-group level key and, from it, cryptographically derive a device specific key.  This is done by essentially hashing the registration id for the device with the enrollment-group level key.  The DPS team has provided some scripts/commands for doing this for both bash and Powershell here.  I’ll repeat the bash command below just to demonstrate.

KEY=[group enrollment key]
REG_ID=[registration id]

keybytes=$(echo $KEY | base64 –decode | xxd -p -u -c 1000)
echo -n $REG_ID | openssl sha256 -mac HMAC -macopt hexkey:$keybytes -binary | base64

where [group enrollment key] is the key from your group enrollment in DPS.  this will generate a cryptographic key that uniquely represents the device specified by your registration id.  We can then use that key as the ‘[device_key]’ in the python script above to generate a SAS key specific to that device within the group enrollment.

Ok – enough prep, let’s get to it.  The next section shows the DPS setup for an Individual Enrollment.  Skip to the section beneath it for Group Enrollment.

DPS Individual Enrollment – setup

The setup for an individual device enrollment for symmetric key in DPS is pretty straightforward.  Navigate to the “manage enrollments” blade from the left nav underneath your DPS instance and click “Add Individual Enrollment”.  On the ‘Add Enrollment’ blade, for Mechanism, choose “Symmetric Key”, then below, enter in your desired registration Id (and an option device id for iot hub if you want it to be different).  It should look similar to the below (click on the pic for a bigger version).

dps-symkey-individual-setup

Click Save.   Once saved, drill back into the device and copy the Primary Key and remember your registration id, we’ll need both later.

That’s it for now.  You can skip to the “call DPS REST APIs” section below, or read on if you want to know how to do this with a group enrollment.

DPS Group Enrollment – setup

The setup for an group enrollment for symmetric key is only slightly more complicated than individual.  On the portal side, it’s fairly simple.  In the Azure portal, under your DPS instance, on the left nav click on ‘manage enrollments’ and then “Add Group Enrollment”.  On the Add Enrollment page, give the enrollment a meaningful name and set Attestation Type to Symmetric Key, like the screenshot below.

dps-symkey-group-setup

Once you do that, click Save, and then drill back down into the enrollment and copy the “Primary Key” that got generated.  This is the group key referenced above, from which we will derive the individual device keys.

In fact, let’s do that before the next section.  Recall the bash command given above for deriving the device key, below is an example using the group key from my ‘dps-test-sym-group1’ group enrollment above and I’ll just ‘dps-test-sym-device01’ as my registration id

dps-symkey-derive-device-key

You can see from the picture that the script generated a device-specific key (by hashing the registration id with the group key).

Just like with the individual enrollment above, we now have the pieces we need to generate our SAS key and call the DPS registration REST APIs

call DPS REST APIs

Now that we have everything setup, enrolled, and our device-specific keys ready, we can set up to call the APIs.  First we need to generate our SAS tokens to authenticate.  Plug in the values from your DPS instance into the python script you saved earlier.  For the [device key] parameter, be sure and plug in either the individual device key you copied earlier, or for the group enrollment, make sure and use the derived key you just created and not the group enrollment key.

Below is an example of a run with my keys, etc

dps-symkey-generate-sas

the very last line is the one we need.  In my case, it was (with a couple of characters changed to protect my DPS):

SharedAccessSignature sr=0ne00052505%2Fregistrations%2Fdps-test-sym-device01&skn=registration&sig=FKOnylJndmpPYgJ5CXkw1pw3kiywt%2FcJIi9eu4xJAEY%3D&se=1568718116

So we now have the pieces we need for the API call. 

The CURL command for the registration API looks like this (with the variable parts bolded).

curl -L -i -X PUT -H ‘Content-Type: application/json’ -H ‘Content-Encoding:  utf-8’ -H ‘Authorization: [sas_token]‘ -d ‘{“registrationId”: “[registration_id]“}’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/register?api-version=2019-03-31

where

  • [sas_token]  is the one we just generated
  • [dps_scope_id] is the one we grabbed earlier from the azure portal
  • [registration_id] is the the one we chose for our device.
  • the –L tells CURL to follow redirects
  • -i tells CURL to output response headers so we can see them
  • -X PUT makes this a put command
  • -H ‘Content-Type:  application/json’ and –H ‘Content-Encoding: utf-8’ are required and tells DPS we are sending utf-8 encoded json in the body  (change the encoding to whatever matches what you are sending)

dps-symkey-registration-call_results

Above is an example of my call and the results returned.

Note two things.. One is the operationId.  DPS enrollment in an IoT Hub is a (potentially) long running operation, and thus is done asynchronously.  So to see the status of your IoT Hub provisioning, we’ll need to poll for status.  I’ll get to that in a minute.  The second thing is the “status” field, which begins in the ‘assigning’ status.

The next API call we need to make is get the status.  You’ll basically do this in a loop until you either get a success or failure status.  The valid status values for DPS are:

    • assigned
      – the return value from the status call will indicate what IoT Hub the device was assigned to
    • assigning
    • disabled
      – the device enrollment record is disabled in DPS, so we can’t assigned
    • failed
      – assignment failed.  There will be an errorCode and errorMessage returned in an registrationState record in the returned JSON to indicate what failed.
    • unassigned – ummm..  no clue.

To make the afore-mentioned status call, you need to copy the operationId from the return status above.  The CURL command for that call is (with variables bolded):

curl -L -i -X GET -H ‘Content-Type: application/json’ -H ‘Content-Encoding:  utf-8’ -H ‘Authorization: [sas_token]’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/operations/[operation_id]?api-version=2019-03-31

use the same sas_token and registration_id as before and the operation_id you just copied.

A successful call looks like this:

dps-symkey-operation-status

Unfortunately, I’m not a fast enough copy/paste-r to catch it in a status other than ‘assigned’  (DPS is just too fast for me).  But you can try this all programmatically or in a script to do it.

Viola

That’s it.  Done.  You can check the status of your registration in the azure portal and see that the device was assigned.

dps-symkey-done

enjoy, and as always, if you have questions or even suggested topics (remember, it has to be complex, technical, and not well covered in the docs), hit me up in the comments