Uncategorized | Busbyland - Azure IoT Playground

Changing roles…

It’s been a while since I wrote anything here. Life, the holidays, and now a job change have gotten in the way.

I’ve changed roles from my previous role as an IoT Global Black Belt, which was part of Microsoft’s field based technical sales org to a role in the Azure IoT engineering team itself. I’m now a Principal Program Manager on our Azure IoT Internet Business Acceleration team, helping partners and customers build those early lighthouse and high impact solutions using our newest technology. I’ve updated my “About me” page accordingly

It will evolve over time, but my initial focus is on our Azure Digital Twins solution!

I’ll continue to write general purpose Azure IoT articles, but you may see some slant towards ADT 🙂

Run Azure Digital Twins ADT-Explorer in Docker

This post walks you through running the ADT-Explorer tool, part of our Azure Digital Twins solution, in a docker container.

My apologies, but it’s been a while since I blogged. Life’s been getting in the way.

I’ve been doing more and more lately with our Azure Digital Twins solution. The ADT engineering team has a nice visualization tool for visualizing your models, twins, relationships, and query results. One of the biggest challenges for me and my customers, has been getting my environment set up: right version of node, right version of the code, authentication, etc.

Conveniently, the ADT team has some instructions for running the ADT-Explorer tool in a docker container, so you don’t have to worry about the pre-reqs. It generally works great. However, there’s one downside. The ADT-Explorer tool uses the Microsoft.Identity provider, and specifically the DefaultAzureCredential setting to do your authentication. This is usually a great thing, in that it just automatically picks up any cached local azure credentials you may already be logged in with (Azure CLI, VS Code, Visual Studio, ENV vars, etc) and uses them.

The challenge, however, is that ADT-Explorer running in a docker container cannot leverage those local cached credentials. So, what do you do?

The easiest way to handle this is to install the azure cli inside the docker container, do an ‘az login’ in it, and then start ADT explorer. The rest of this post walks you through this. This post assumes you have docker installed on your machine with linux containers enabled.

The first step is to clone the repo with git.


git clone https://github.com/Azure-Samples/digital-twins-explorer

After you do that, navigate to the digital-twins-explorer and create the adt-explorer docker container (Note the dot/period at the end of the command – don’t forget it!)


docker build -t adt-explorer .

So far this follows the instructions provided by the ADT team, but this is where we will deviate. The ADT team provided instructions have you just run the container as-is. However, if you do that, the DefaultAzureCredentials won’t work. Instead, we need to install the azure cli and login into azure before we start the adt-explorer app. The first thing we need to do is run the adt-explorer docker container, but we are going to start in a bash command prompt. Run the following command


docker run -it -p3000:3000 adt-explorer bash

Note that we added ‘bash’ to the end of the command line, to start the container but override the default entrypoint and give us a bash prompt. It should look like this:

Now that we have a prompt, we need to install the azure cli. To install it, run this command


curl -sL https://aka.ms/InstallAzureCLIDeb | bash

This should install the CLI for us. After a successful install, we can now do an ‘az login’ to authenticate to Azure. You will get a ‘code’ to use, and a URL to visit in your browser as shown below

Open your browser, navigate to http://microsoft.com/devicelogin, enter the code, then your azure subscription credentials as shown below

The authentication process will be slightly different for every environment based on your IT department’s set up (i.e. 2-factor auth, etc), but the result should eventually be a successful login

After a successful login, you’ll see a list of your azure subscriptions in the docker container.

Now we can start the adt-explorer app. To do so, run


npm run start

You’ll see this in your container.

You can now open up a browser, navigate to http://localhost:3000. Click on the People icon in the upper right corner, enter your ADT instance URL, and off you go.

Enjoy, and as always, hit me up in the comments if you have any issues.

Azure IoT Device Provisioning Service (DPS) over MQTT

Continuing the theme of “doing things on Azure IoT without using our SDKs”, this article describes how to provision IOT devices with Azure IoT’s Device Provisioning Service over raw MQTT.

Previously, I wrote an article that describes how to leverage Azure IoT’s Device Provisioning Service over its REST API, as well as an article about connecting to IoT Hub/Edge over raw MQTT. Where possible, I do recommend using our SDKs, as they provide a nice abstraction layer over the supported transport protocols and frees you from all that protocol-level detailed work. However, we understand there are times and reasons where it’s just a better fit to do things over the raw protocols.

To support this, the Azure IoT DPS engineering team has documented the necessary technical details to register your device via MQTT. This document may provide enough details for you to figure out how to do it, but since I needed to test it for a customer anyway, I thought I’d capture a real-world example in hopes it can help others.

To make the scenario simpler, I chose to just use symmetric key attestation, but this would still work with any of the attestation methods supported by DPS.

Create individual enrollment

The first step is to create the enrollment in DPS. In the Azure portal, in your DPS instance, from the ‘overview’ tab, grab your Scope ID from the upper right of the ‘overview’ tab as shown below (I’ve blacked out part of my details, for obvious reasons)

Once you have that, copy it somewhere like notepad or equivalent, we’ll use it later. Once we have that, we can create our enrollment. On the left nav, click on “Manage Enrollments” and then “Add Individual Enrollment”. For “Mechanism”, choose Symmetric Key, enter a registration ID of your choosing (for the example further below, I used ‘my-mqtt-dev01’)

Click Save. Then drill back into your enrollment in the portal and copy the “Primary Key” and save it for later use.

Generate SAS token

Once you’ve created the enrollment and gotten the device key, we need to generate a SAS token for authentication to the DPS service. A description of the SAS token, and several code samples for generating one in various languages can be found here. Some of the inputs (discussed below) will be different for DPS versus IoT Hub, but the basic structure of the SAS token is the same.

For my purposes, I used this python code to generate mine:

————-

from base64 import b64encode, b64decode
from hashlib import sha256
from time import time
from urllib import quote_plus, urlencode
from hmac import HMAC

def generate_sas_token(uri, key, policy_name, expiry=3600000000):
ttl = time() + expiry
sign_key = “%s\n%d” % ((quote_plus(uri)), int(ttl))
print(sign_key)
signature = b64encode(HMAC(b64decode(key), sign_key, sha256).digest())

rawtoken = {
‘sr’ : uri,
‘sig’: signature,
‘se’ : str(int(ttl))
}

if policy_name is not None:
rawtoken[‘skn’] = policy_name

return ‘SharedAccessSignature ‘ + urlencode(rawtoken)

uri = ‘[dps URI]’
key = ‘[device key]’
expiry = [SAS token duration]
policy=’registration’

print(generate_sas_token(uri, key, policy, expiry))

——–

where:

[dps URI] is of the form [DPS scope id]/registrations/[registration id]
[device key] is the primary key you saved earlier
[SAS token duration] is the number of seconds you want the token to be valid for
policy is required to be ‘registration’ for DPS SAS tokens

running this code will give you a SAS token that looks something like this (changing a few random characters to protect my DPS):

SharedAccessSignature sr=0ne00055505%2Fregistrations%2Fmy-mqtt-dev01&skn=registration&sig=gMpllKo7qS1VR31vyfsT6JAcc4%2BHIu2gQSyai0Uz0KM%3D&se=1579698526

Now that we have our authentication credentials, we are ready to make our MQTT call.

Example call

The documentation does a decent job of showing the MQTT parameters and flow (read it first!), so I’m not going to repeat that here. What I will show is an example call with screenshots to ‘make it real’. For my testing, I used mqtt.fx, which is a pretty nice little interactive MQTT test client.

Once you download and install it, click on the little lightning bolt to switch from localhost to allow you to create a new connection to an MQTT server.

After that, click on the settings symbol next to the edit box to open the settings dialog that lets you edit the various connection profiles:

On the “Edit Connection Profiles” dialog, in the very bottom left hand corner, click the “+” symbol to create a new connection profile.

Give your connection a name and choose MQTT Broker as the Profile Type

Enter the following settings in the top half of the dialog:

for “Broker Address”, use ‘global.azure-devices-provisioning.net’
for “Broker Port”, use “8883”
for Client ID, enter your registration ID you used in the portal for your device

Click on the General ‘tab’ at the bottom. As in the screenshot above, for MQTT Version, uncheck the “Use Default” button and explicitly choose version 3.1.1. Leave other settings on this tab alone.

click on the “User Credentials” tab’

for “User Name”, enter [DPS Scope Id]/registrations/[registration id]/api-version=2019-03-31 (replacing the scope id and registration id with your values)
for “Password”, copy/paste in your SAS token you generated earlier

Move to the SSL/TLS tab. Check the box for “Enable SSL/TLS” and make sure that TLSv1.2 is chosen as the protocol

leave the proxy and LWT tabs alone.

Click Ok to save the settings and return to the main screen

Click on the Connect button and you should get a successful connection (you can verify by looking at the “log” tab)

Once connected, navigate to the “Subscribe” tab. We will set up a subscription on the dps ‘response’ MQTT topic to receive responses to our registration attempts from DPS. On the “Subscribe” tab, enter ‘$dps/registrations/res/#’ into the subscriptions box, choose “QoS1” from the buttons on the right, and click “Subscribe”. You should see an active subscription get set up and waiting on responses.

Click back over on the “Publish” tab and we will make our registration attempt. In the publish edit box, enter $dps/registrations/PUT/iotdps-register/?$rid={request_id}

replace {request_id} with an integer of your choosing (1 is fine to start with). This lets us correlate requests with responses when we get responses back from the service. For example, I entered:

$dps/registrations/PUT/iotdps-register/?$rid=1

in the big edit box beneath the publish edit box, we need to enter a ‘payload’ for the request. For DPS registration requests, the payload takes the form of a JSON document like this: {“registrationId”:”<registration id>”}

for example, for my sample it’s:

{“registrationId”: “my-mqtt-dev01”}

Hit the “Publish button”

Flip back over to the Subscribe tab and you should see on the right hand side of the screen that we’ve received a response from DPS. You should see something like this:

This indicates that DPS is in the process of ‘assigning’ and registering our device to an IoT Hub. This is a potentially long running operation, so to get the status of it, we have to query for that status. To do that, we are going to publish another MQTT message to check on the status. For that, we need the ‘operationId’ from the message we just received. In the screenshot above, mine looks like this:

4.22724a0213a69c4d.9750f5e6-b4c3-4760-9b15-4e74d6120bd1

Copy that ID as we’ll use it in the next step.

To check on the status of the operation, switch back over to the Publish tab and replace the values in the publish edit box with this

$dps/registrations/GET/iotdps-get-operationstatus/?$rid={request_id}&operationId={operationId}

replacing {request_id} with a new request id (2 in my case) and the {operationId} with the operationId you just copied. For example, with my sample values and the response received above, my request looks like this:

$dps/registrations/GET/iotdps-get-operationstatus/?$rid=2&operationId=4.22724a0213a69c4d.9750f5e6-b4c3-4760-9b15-4e74d6120bd1

Delete the JSON in the payload box and click “publish”

Switch back over to the Subscribe tab and you should notice that you’ve received a response to your operational status query, similar to this:

Notice the status of “assigned”, as well as details like “assignedHub” that gives the state of the successful registration and connection details.

If you navigate back over to the azure portal and look at the enrollment record for your device (refresh the page.. you may have to exit and re-enter), you should see something like this:

This indicates that our DPS registration was successful.

In the “real world”, in your application, you’ll make the registration attempt and then poll the operational status until it gets to the state of ‘assigned’. There will be intermediate states while it is being assigned, but doing this manually through a GUI, I’m not fast enough to catch them Smile

Enjoy – and let me know in the comments if you have any questions or issues.

Connect MXChip DevKit to Azure IoT Edge

A customer of mine who is working on an IoT POC to show their management wanted to connect the MXChip Devkit to IoT Hub via IoT Edge. This turned out to be trickier than it should be, as the connection between the MXChip and the IoT Edge box is, like all IoT Hub/Edge connections, TLS-encrypted. So you have to get the MXChip to trust the “tls server” certificate that IoT Edge returns when a client tries to connect. Thanks to some great ground-laying work by Arthur Ma, I wrote a hands-on lab walking you through this scenario. The lab can be found on my team’s github site here

Enjoy, and let me know if you have any problems.

Azure IoT Device Provisioning Service via REST–part 2

This is part 2 of a two part post on provisioning IoT devices to Azure IoT Hub via the Azure IoT Device Provisioning Service (DPS) via its REST API. Part 1 described the process for doing it with x.509 certificate attestation from devices and this part will describe doing it with Symmetric Key attestation.

I won’t repeat all the introduction, caveats, etc. that accompanied part 1, but you may want to take a quick peek at them so you know what I will, and will not, be covering here.

If you don’t fully understand the Symmetric Key attestation options for DPS, I recommend you go read the docs here first and then come back…

Ok, welcome back!

So let’s just jump right in. Similarly to part 1, there will be a couple of sections of ‘setup’, depending on whether you choose to go with Individual Enrollments or Group Enrollments in DPS. Once that is done, and the accompanying attestation tokens are generated, the actual API calls are identical between the two.

Therefore, for the first two sections, you can choose the one that matches your desired enrollment type and read it for the required setup (or read them both if you are the curious type), then you can just jump to the bottom section for the actual API calls.

But, before we start with setup, there’s a little prep work to do.

Prep work (aka – how do I generate the SAS tokens?)

Symmetric Key attestation in DPS works, just like pretty much all the rest of Azure, on the concept of SAS tokens. In Azure IoT, these tokens are typically derived from a cryptographic key tied to a device or service-access level. As mentioned in the overview link above (in case you didn’t read it), DPS have two options for these keys. One option is an individual key per device, as specified or auto-generated in the individual enrollments. The other option is to have a group enrollment key, from which you derive a device-specific key, that you leverage for your SAS token generation.

Generating SAS tokens

So first, let’s talk about and prep for the generation of our SAS tokens, independent of what kind of key we use. The use of, and generation of, SAS tokens is generally the same for both DPS and IoT Hub, so you can see the process and sample code in various languages here. For my testing, I pretty much ~~shamelessly stole~~ re-used the python example from that page, which I slightly modified (to actually call the generate_sas_token method).

from base64 import b64encode, b64decode
from hashlib import sha256
from time import time
from urllib import quote_plus, urlencode
from hmac import HMAC

def generate_sas_token(uri, key, policy_name, expiry=3600):
     ttl = time() + expiry
     sign_key = “%s\n%d” % ((quote_plus(uri)), int(ttl))
     print sign_key
     signature = b64encode(HMAC(b64decode(key), sign_key, sha256).digest())

    rawtoken = {
         ‘sr’ : uri,
         ‘sig’: signature,
         ‘se’ : str(int(ttl))
     }

if policy_name is not None:
rawtoken[‘skn’] = policy_name

return ‘SharedAccessSignature ‘ + urlencode(rawtoken)

uri = ‘[resource_uri]’
key = ‘[device_key]’
expiry = [expiry_in_seconds]
policy=’[policy]’

print generate_sas_token(uri, key, policy, expiry)

the parameters at the bottom of the script, which I hardcoded because I am ~~lazy~~ busy, are as follows:

[resource_uri] – this is the URI of the resource you are trying to reach with this token. For DPS, it is of the form ‘[dps_scope_id]/registrations/[dps_registration_id]’, where [dps_scope_id] is the scope id associated with your DPS instance, found on the overview blade of your DPS instance in the Azure portal, and [dps_registration_id] is the registration_id you want to use for your device. It will be whatever you specified in an individual enrollment in DPS, or can be anything you want in a group enrollment as long as it is unique. Frequently used ideas here are combinations of serial numbers, MAC addresses, GUIDs, etc
[device_key] is the device key associated with your device. This is either the one specified or auto-generated for you in an individual enrollment, or a derived key for a group enrollment, as explained a little further below
[expiry_in_seconds] the validity period of this SAS token in sec… ok, not going to insult your intelligence here
[policy] the policy with which the key above is associated. For DPS device registration, this is hard coded to ‘registration’

So an example set of inputs for a device called ‘dps-sym-key-test01’ might look like this (with the scope id and key modified to protect my DPS instance from the Russians!)

uri = ‘0ne00057505/registrations/dps-sym-key-test01’
key = ‘gPD2SOUYSOMXygVZA+pupNvWckqaS3Qnu+BUBbw7TbIZU7y2UZ5ksp4uMJfdV+nTIBayN+fZIZco4tS7oeVR/A==’
expiry = 3600000
policy=’registration’

Save the script above to a *.py file. (obviously, you’ll need to install python 2.7 or better if you don’t have it to run the script)

If you are only doing individual enrollments, you can skip the next section, unless you are just curious.

Generating derived keys

For group enrollments, you don’t have individual enrollment records for devices in the DPS enrollments, so therefore you don’t have individual device keys. To make this work, we take the enrollment-group level key and, from it, cryptographically derive a device specific key. This is done by essentially hashing the registration id for the device with the enrollment-group level key. The DPS team has provided some scripts/commands for doing this for both bash and Powershell here. I’ll repeat the bash command below just to demonstrate.

KEY=[group enrollment key]
REG_ID=[registration id]

keybytes=$(echo $KEY | base64 –decode | xxd -p -u -c 1000)
echo -n $REG_ID | openssl sha256 -mac HMAC -macopt hexkey:$keybytes -binary | base64

where [group enrollment key] is the key from your group enrollment in DPS. this will generate a cryptographic key that uniquely represents the device specified by your registration id. We can then use that key as the ‘[device_key]’ in the python script above to generate a SAS key specific to that device within the group enrollment.

Ok – enough prep, let’s get to it. The next section shows the DPS setup for an Individual Enrollment. Skip to the section beneath it for Group Enrollment.

DPS Individual Enrollment – setup

The setup for an individual device enrollment for symmetric key in DPS is pretty straightforward. Navigate to the “manage enrollments” blade from the left nav underneath your DPS instance and click “Add Individual Enrollment”. On the ‘Add Enrollment’ blade, for Mechanism, choose “Symmetric Key”, then below, enter in your desired registration Id (and an option device id for iot hub if you want it to be different). It should look similar to the below (click on the pic for a bigger version).

Click Save. Once saved, drill back into the device and copy the Primary Key and remember your registration id, we’ll need both later.

That’s it for now. You can skip to the “call DPS REST APIs” section below, or read on if you want to know how to do this with a group enrollment.

DPS Group Enrollment – setup

The setup for an group enrollment for symmetric key is only slightly more complicated than individual. On the portal side, it’s fairly simple. In the Azure portal, under your DPS instance, on the left nav click on ‘manage enrollments’ and then “Add Group Enrollment”. On the Add Enrollment page, give the enrollment a meaningful name and set Attestation Type to Symmetric Key, like the screenshot below.

Once you do that, click Save, and then drill back down into the enrollment and copy the “Primary Key” that got generated. This is the group key referenced above, from which we will derive the individual device keys.

In fact, let’s do that before the next section. Recall the bash command given above for deriving the device key, below is an example using the group key from my ‘dps-test-sym-group1’ group enrollment above and I’ll just ‘dps-test-sym-device01’ as my registration id

You can see from the picture that the script generated a device-specific key (by hashing the registration id with the group key).

Just like with the individual enrollment above, we now have the pieces we need to generate our SAS key and call the DPS registration REST APIs

call DPS REST APIs

Now that we have everything setup, enrolled, and our device-specific keys ready, we can set up to call the APIs. First we need to generate our SAS tokens to authenticate. Plug in the values from your DPS instance into the python script you saved earlier. For the [device key] parameter, be sure and plug in either the individual device key you copied earlier, or for the group enrollment, make sure and use the derived key you just created and not the group enrollment key.

Below is an example of a run with my keys, etc

the very last line is the one we need. In my case, it was (with a couple of characters changed to protect my DPS):

SharedAccessSignature sr=0ne00052505%2Fregistrations%2Fdps-test-sym-device01&skn=registration&sig=FKOnylJndmpPYgJ5CXkw1pw3kiywt%2FcJIi9eu4xJAEY%3D&se=1568718116

So we now have the pieces we need for the API call.

The CURL command for the registration API looks like this (with the variable parts bolded).

curl -L -i -X PUT -H ‘Content-Type: application/json’ -H ‘Content-Encoding: utf-8’ -H ‘Authorization: [sas_token]‘ -d ‘{“registrationId”: “[registration_id]“}’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/register?api-version=2019-03-31

where

[sas_token] is the one we just generated
[dps_scope_id] is the one we grabbed earlier from the azure portal
[registration_id] is the the one we chose for our device.
the –L tells CURL to follow redirects
-i tells CURL to output response headers so we can see them
-X PUT makes this a put command
-H ‘Content-Type: application/json’ and –H ‘Content-Encoding: utf-8’ are required and tells DPS we are sending utf-8 encoded json in the body (change the encoding to whatever matches what you are sending)

Above is an example of my call and the results returned.

Note two things.. One is the operationId. DPS enrollment in an IoT Hub is a (potentially) long running operation, and thus is done asynchronously. So to see the status of your IoT Hub provisioning, we’ll need to poll for status. I’ll get to that in a minute. The second thing is the “status” field, which begins in the ‘assigning’ status.

The next API call we need to make is get the status. You’ll basically do this in a loop until you either get a success or failure status. The valid status values for DPS are:

assigned
– the return value from the status call will indicate what IoT Hub the device was assigned to
assigning
disabled
– the device enrollment record is disabled in DPS, so we can’t assigned
failed
– assignment failed. There will be an errorCode and errorMessage returned in an registrationState record in the returned JSON to indicate what failed.
unassigned – ummm.. no clue.

To make the afore-mentioned status call, you need to copy the operationId from the return status above. The CURL command for that call is (with variables bolded):

curl -L -i -X GET -H ‘Content-Type: application/json’ -H ‘Content-Encoding: utf-8’ -H ‘Authorization: [sas_token]’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/operations/[operation_id]?api-version=2019-03-31

use the same sas_token and registration_id as before and the operation_id you just copied.

A successful call looks like this:

Unfortunately, I’m not a fast enough copy/paste-r to catch it in a status other than ‘assigned’ (DPS is just too fast for me). But you can try this all programmatically or in a script to do it.

Viola

That’s it. Done. You can check the status of your registration in the azure portal and see that the device was assigned.

enjoy, and as always, if you have questions or even suggested topics (remember, it has to be complex, technical, and not well covered in the docs), hit me up in the comments

Azure IoT Device Provisioning Service via REST–part 1

This will be a two-part article on how to provision IoT devices using Microsoft’s Azure IoT Device Provisioning Service, or DPS, via its REST API. DSP is part of our core IoT platform. It gives you an global-scale solution for near zero touch provisioning and configuration of your IoT Devices. We make the device-side provisioning pretty easy with nice integration with our open-source device SDKs.

With that said, one of our core design tenants for our IoT platform is that, while they do make life easier for you in most instances, you do not have to use our SDKs to access any of our services, whether that’s core IoT Hub itself, IoT Edge, or in this case, DPS.

I recently had a customer ask about invoking DPS for device registration from their field devices over the REST API. For various reasons I won’t dive deep in, they didn’t want to use our SDKs and preferred the REST route. The REST APIs for DPS are documented, but.. well.. we’ll just say “not very well” and leave it at that……..

ahem……..

anyway..

So I set out to figure out how to invoke device registration via the REST APIs for my customer and thought I would document the process here.

Ok, enough salad, on to the meat of the post.

First of all, this post assumes you are already familiar with DPS concepts have maybe even played with it a little. If not, review the docs here and come back. I’ll wait……….

Secondly, in case you didn’t notice the “part 1” in the title, because of the length, this will be a two-part post. The first half will show you how to invoke DPS registration over REST for the x.509 certificate attestation use cases, both individual and group enrollments, and part 2 will show for the Symmetric Key attestation use cases.

NOTE- but what about TPM chip-based attestation?, you might ask.. Well, I’m glad you asked! Using a TPM-chip for storing secrets and working with any IoT device is a best practice that we highly recommend! with that said, I’m not covering it for three reasons, 1) the process will be relatively similar to the other scenarios, 2) despite it being our best practice, I don’t personally have any customers today it (shame on us all!) and 3) I don’t have an IoT device right now that has a TPM to test with

One final note – I’m only covering the registration part from the device side. There are also REST APIs for enrolling devices and many other ‘back end’ processes that I’m not covering. This is partially because the device side is the (slightly) harder side, but also because it’s the side that is more likely to need to be invoked over REST for constrained devices, etc.

Prep work (aka – how do I generate test certs?)

The first thing we need f0r x.509 certificate attestation is, you guess it, x.509 certificates. There are several tools available out there to do it, some easy, and some requiring an advanced degree in “x.509 Certificate Studies”. Since I don’t have a degree in x.509 Certificate Studies, I chose the easy route. But to be clear, I just picked this method, but any method that can generate valid x.509 CA and individual certs will work.

The tool I chose is provided by the nice people who write and maintain our Azure IoT SDK for Node.js. This tool is specifically written to work/test with DPS, so it was ideal. Like about 50 other tools, it’s a friendly wrapper around openssl. Instructions for installing the tool are provided in the readme and I won’t repeat them here. Note that you need NodeJs version 9.x or greater installed for it to work.

For my ‘dev’ environment, I used Ubuntu running on Windows Subsystem for Linux (WSL) (man, that’s weird for a 21-year MSFT veteran to say), but any environment that can run node and curl will work.

Also, final note about the cert gen scripts.. these are scripts for creating test certificates… please.. pretty please.. don’t use them for production. If you do, Santa will be very unhappy with you!

In the azure portal, I’ll assume you’ve already set up a DPS instance. At this point, on the DPS overview blade, note your scope id for your instance (upper right hand side of the DPS Overview blade), we’ll need it here in a sec. From here on out, I’ll refer to that as the [dps_scope_id]

The next two sections tell you how to create and setup the certs we’ll need for either the Individual or Group Enrollments. Once the certs are properly setup, the process for making the API calls are the same, so pick the one of the next two sections that apply to you and go (or read them both if you are really thirsty for knowledge!)

x.509 attestation with Individual Enrollments – setup

Let’s start with the easy case, which is x.509 attestation with an Individual Enrollment. The first thing I want to do is to generate some certs.

For my testing, using the create_test_cert tool to create a root certificate, and a device certificate for my individual enrollment, using the following two commands.

node create_test_cert.js root “dps-test-root”

node create_test_cert.js device “dps-test-device-01” “dps-test-root”

The tool creates the certificates with the right CN/subject, but when saving the files, it drops the underscores and camel-cases the name. I have no idea why. Just roll with it. Or feel free to create a root cert and device cert using some other tool. The key is to make the subject/CN of the device cert be the exact same thing you plan to make your registration id in DPS. They HAVE to match, or the whole thing doesn’t work. For the rest of the article, I will refer to the registration id as [registration_id]

so my cert files look like this…. (click on the pic to make it larger)

Back in the Azure portal, under “Manage enrollments” on the left nav, click on “Add Individual Enrollment”. On the “Add Enrollment” blade, leave the default of “X.509” for the Mechanism. On the “Primary Certificate .pem or .cer file”, click on the folder and upload/choose the device certificate you generated earlier (in my case, it’s dpsTestRoot_cert.pem).

The top half of my “Add Enrollment” blade looks like this (everything below it is default)

I chose to leave the IoT Hub Device ID field blank. If you want your device ID in Hub to be something different than your registration id (which is the CN/subject name in your cert), then you can enter a different name here. click SAVE to save your enrollment and we are ready to go.

x.509 attestation with Group Enrollments – setup

Ok, if you’re reading this section, you are either a curious soul, or decided to go with the group enrollment option for x.509 attestation with DPS.

With the umbrella of group enrollment, there are actually two different options for the ‘group’ certificate, depending on whether or not you want to leverage a root CA certificate or an intermediate certificate. For either option, for test purposes, we’ll go ahead and generate our test certificates. The difference will primarily be which one we give to DPS. In either case, once the certificate is in place, we can authenticate and register any device that presents a client certificate signed by the CA certificate that we gave to DPS. For details of the philosophy behind x.509 authentication/attestation for IoT devices, see this article.

For this test, I generated three certificates, a root CA cert, an intermediate CA cert (signed by the root), and an end IoT device certificate, signed by the intermediate CA certificate. I used the following commands to do so.

node create_test_cert.js root “dps-test-root”

this generated my root CA certificate

node create_test_cert.js intermediate “dps-test-intermediate” “dps-test-root”

this generated an intermediate CA certificate signed by the root CA cert I just generated

node ../create_test_cert.js device “dps-test-device-01” “dps-test-intermediate”

this generated a device certficate for a device called “dps-test-device-01” signed by the intermediate certificate above. So now I have a device certificate that ‘chains up’ through the intermediate to the root.

At this point, you have the option of either setting up DPS with the root CA certificate, or the Intermediate Certificate for attesting the identity of your end devices. The setup process for each option is slightly different and described below.

Root CA certificate attestation

For Root CA certificate attestation, you need to upload the root CA certificate that you generated, and then also provide proof of possession of the private key associated with that root CA certificate. That is to keep someone from impersonating the holder of the public side of the root CA cert by making sure they have the corresponding private key as well.

The first step in root CA registration is to navigate to your DPS instance in the portal and click “Certificates” in the left-hand nav menu. Then click “Add”. In the Add Certificate blade, give your certificate a name that means something to you and click on the folder to upload our cert. Once uploaded, click Save.

At that point, you should see your certificate listed in the Certificates blade with a status of “unverified”. This is because we have not yet verified that we have the private key for this cert.

The process for verifying that we have the private key for this cert involves having DPS generate a “verification code” (a cryptographic alphanumeric string” and then we will take that string and, using our root CA certificate, create a certificate with the verification code as the CN/Subject name and then sign that certificate with the private key of the root CA cert. This proves to DPS that we possess the private key. To do this, click on your newly uploaded cert. On the Certificate Details page, click on the “Generate Verification Code” button and it will generate a verification code as shown below.

Copy that code. Back on the box that you are using to generate the certs, run this command to create the verification cert.

create_test_cert.js verification [–ca root cert pem file name] [–key root cert key pem file name] [–nonce nonce]

where –ca is the path to your root CA cert you uploaded, –key is the path to it’s private key, and–nonce is the verification code you copied from the portal, for example, in my case:

node ../create_test_cert.js verification –ca dpsTestRoot_cert.pem –key dpsTestRoot_key.pem –nonce 07F9332E108C7D24283FB6B8A05284E6B873D43686940ACE

This will generate a cert called “verification_cert.pem”. Back on the azure portal on the Certificates Detail page, click on the folder next to the box “Verification Certificate *.pem or *.cer file” and upload this verification cert and click the “Verify” button.

You will see that the status of your cert back on the Certificates blade now reads “Verified” with a green check box. (you may have to refresh the page with the Refresh button to see the status change).

Now click on “Manage enrollments” on the left-nav and click on “Add enrollment group”. Give it a meaningful name for you, make sure that “Certificate Type” is “CA Certificate” and choose the certificate you just verified from the drop-down box, like below.

Click Save

Now you are ready to test your device cert. You can skip the next section and jump to the “DPS registration REST API calls” section

Intermediate CA certificate attestation

If you decided to go the Intermediate CA certificate route, which I think will be the most common, luckily the process is a little easier than with a root CA certificate. In your DPS instance in the portal, under “Manage enrollments”, click on Add Enrollment Group”. Make sure that Attestation Type is set to “Certificate” and give the group a meaningful name. Under “Certificate Type”, choose “Intermediate Certificate” and click on the folder next to “Primary Certificate .pem or .cer file” and upload the Intermediate Certificate we generated earlier. For me, it looks like this..

click Save and you are ready to go to the next section to try to register a device cert signed by your Intermediate Certificate.

The DPS registration REST API calls

Ok, so the moment you’ve all been waiting (very patiently) for…

As mentioned previously, now that the certs have all been created, uploaded, and setup properly in DPS, the process and the API calls from here on out is the same regardless of how you set up your enrollment in DPS.

For my testing, I didn’t want to get bogged down in how to make HTTP calls from various languages/platforms, so I chose the most universal and simple tool I could find, curl, which is available on both windows and linux.

The CURL command for invoking DPS device registration for x.509 individual enrollments with all the important and variable parameters in []’s..

curl -L -i -X PUT –cert ./[device cert].pem –key ./[device-cert-private-key].pem -H ‘Content-Type: application/json’ -H ‘Content-Encoding: utf-8’ -d ‘{“registrationId”: “[registration_id]“}’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/register?api-version=2019-03-31

That looks a little complicated, so let’s break each part down

-L : tells curl to follow HTTP redirects
– i : tells curl to include protocol headers in output. Not strictly necessary, but I like to see them
-X PUT : tells curl that is an HTTP PUT command. Required for this API call since we are sending a message in the body
–cert : this is the device certificate that we, as a TLS client, want to use for client authentication. This parameter, and the next one (key) are the main thing that makes this an x.509-based attestation. This has to be the same cert you registered in DPS
–key : the private key associated with the device certificate provided above. This is necessary for the TLS handshake and to prove we own the cert
-H ‘Content-Type: application/json’ : required to tell DPS we are posting up JSON content and must be ‘application/json’
-H ‘Content-Encoding: utf-8’ : required to tell DPS the encoding we are using for our message body. Set to the proper value for your OS/client (I’ve never used anything other than utf-8 here)
-d ‘{“registrationId”: “[registration_id]”}’ : the –d parameter is the ‘data’ or body of the message we are posting. It must be JSON, in the form of “{registrationId”:”[registration_id”}. Note that for CURL, I wrapped it in single quotes. This nicely makes it where I don’t have to escape the double quotes in the JSON
Finally, the last parameter is the URL you post to. For ‘regular’ (i.e not on-premises) DPS, the global DPS endpoint is global.azure-devices-provisioning.net, so that’s where we post. https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/register?api-version=2019-03-31. Note that we have to replace the [dps_scope_id] with the one you captured earlier and [registration_id] with the one you registered.

you should get a return that looks something like this…

The next API call we need to make is get the status. You’ll basically do this in a loop until you either get a success or failure status. The valid status values for DPS are:

- assigned
  – the return value from the status call will indicate what IoT Hub the device was assigned to
- assigning
- disabled
  – the device enrollment record is disabled in DPS, so we can’t assigned
- failed
  – assignment failed. There will be an errorCode and errorMessage returned in an registrationState record in the returned JSON to indicate what failed.
- unassigned – ummm.. no clue.

To make the afore-mentioned status call, you need to copy the operationId from the return status above. The CURL command for that call is:

curl -L -i -X GET –cert ./dpsTestDevice01_cert.pem –key ./dpsTestDevice01_key.pem -H ‘Content-Type: application/json’ -H ‘Content-Encoding: utf-8’ https://global.azure-devices-provisioning.net/[dps_scope_id]/registrations/[registration_id]/operations/[operation_id]?api-version=2019-03-31

where [dps_scope_id] and [registration_id] are the same as above, and [operation_id] is the one you copied above. The return will look something like this, keeping in mind the registrationState record will change fields based on what the returned status was.

Unfortunately, I’m not a fast enough copy/paste-r to catch it in a status other than ‘assigned’ (DPS is just too fast for me). But you can try this all programmatically or in a script to do it.

Ta-Da!

That’s it. You can navigate back to DPS, drill in on your device, and see the results

Enjoy – and as always, hit me up with any questions in the comments section.

raw AMQP to IoT Hub and IoT Edge

It seems like lately my life has consisted mostly of trying to figure out how to connect “brownfield or legacy systems” (that’s MSFT-speak for “doesn’t use our IoT device SDKs” :-)) to Azure IoT Hub or Azure IoT Edge or both. I’ve previously in other posts shown how to do it with raw MQTT, Mosquitto, and Node-Red.

I was recently asked by a customer for a sample of connecting a raw AMQP client to IoT Edge. So with unbridled optimism, I quickly did a web search to hunt down what surely already existed as a sample. Both google and bing quickly dashed my hope for that (as they so often do). Even StackOverflow, the best hope for all development-kind failed me! So I waded in to figure it out myself.

Setting the stage

Just for simplicity, I used the python uamqp library for this. This is the AMQP library (or at least the C version of it) that we use ourselves underlying IoT Hub and IoT Edge (and service bus, event hub, etc), so it seemed like a natural fit. And it also came with a sample that I could start from and adapt. The code further below and information in this post is based on that sample. The primary two issues with the sample out of the box was that it used a ‘hub-level’ key vs. a device-scoped key for authentication to IoT Hub (don’t do that!) and for some reason it was written to show device-bound (cloud to device) connectivity vs. cloud-bound (device to cloud, aka ‘telemetry’) messaging. So I adapted the sample for my needs, and will show the adaptations below.

While it took me a little time to figure things out, the two most complicated parts where

Figuring out the right AMQP connection string format to connect to IoT Hub/Edge. This is normally handled under the covers with our SDKs, but getting it right without the SDKs took a little research and trial/error
Figuring out how to get the client to trust the certificate chain that edgeHub presents to connecting clients for TLS connections (for more details on how Edge uses certs, see this article by my very favorite author!). This second bullet is only needed if you are connecting to IoT Edge. The right root-ca certs (i.e. Baltimore) are embedded in the uamqp library for IoT Hub itself.

The format for the AMQP connection string is actually already documented here by our engineering team (under the “protocol specifics” section), but it’s not called out very obviously like the entire sub-article we have for MQTT, so I actually missed it for a while. If you use a device-scoped key (which you generally should), the correct format for the AMQP connection string is:


amqps://[device_id]@sas.[short-hub-name]:[sas-token]@[target-endpoint]/[operation]

where:

[device_id] is an iot-hub registered device id for an IoT device
[short-hub-name] is the name of your IoT Hub *without* the .azure-devices-net
- NOTE: the combination of device_id and short-hub-name, which collectively is the ‘username’ in the connection string, must be URL encoded before sent
[sas-token] is a SAS token generated for your device
[target-endpoint] is either the name of your IoT Hub *with* the .azure-devices.net in the case of an IoT Hub direction connection OR it’s the FQDN of your IoT Edge box in the case of connecting to IoT Edge (i.e. mygateway.contoso.local)
[operation] is the desired operation. For example, to send telemetry data, operation is /devices/[device id]/messages/events

Just to show an example of what the connection string looks like with a live device and hub, below is an example of one of mine (with a few random characters in the sas-token changed to protect my hub :-))


amqps://amqptest%40sas.sdbiothub1:SharedAccessSignature+sr%3Dsdbiothub1.azure-devices.net%252Fdevices%252Famqptest%26sig%3DyfStnV4tfi3p7xeUg2DCTSauZowQ90Gplq3hKFzTY10%253D%26se%3D1552015962@mygateway.contoso.local/devices/amqptest/messages/events

where:

amqptest is the device id of my device registered in IoT Hub
sdbiothub1 is the name of my IoT Hub
mygateway.contoso.local is the FQDN of my IoT Edge device (not really, but you don’t need to know the real one…)
/devices/amqptest/messages/events is the ‘operation’ I’m invoking, which in the case of IoT Hub/Edge means to send device-to-cloud telemetry data

The code

ok, enough pre-amble, let’s get to the code

NOTE:  Please note - strangely enough, as of this writing (3/7/2019) the code and post below will NOT actually work today.  During my work and investigation, and working with one of the IoT Edge engineers, we discovered a small bug in edgeHub that prevented the raw AMQP connection string from being parsed correctly. The bug has already been fixed, per this pull request, but the fix won't be publicly available until later this month in the next official release.  But, since I'm internal MSFT and "it's good to be the king", I was able to get a private build of edgeHub to test against.  I'll update this post once the fix is publicly available.  (technically, if you really want it, you can do your own private build of edgeHub, since it's open source

The first step in using the sample is to install the uamqp library, the instructions for which can be found here.

Below is my modified version of the sample that i started with. I tried to annotate any change I made with a preceding comment that starts with #steve, so you can just search for them to understand what I changed, or just use the sample directly


#-------------------------------------------------------------------------

# Copyright (c) Microsoft Corporation. All rights reserved.

# Licensed under the MIT License. See License.txt in the project root for

# license information.

#--------------------------------------------------------------------------



import os

import logging

import sys

from base64 import b64encode, b64decode

from hashlib import sha256

from hmac import HMAC

from time import time

from uuid import uuid4

try:

    from urllib import quote, quote_plus, urlencode #Py2

except Exception:

    from urllib.parse import quote, quote_plus, urlencode



import uamqp

from uamqp import utils, errors



#steve - added to share the SAS token and username broadly

sas_token = ''

auth_username = ''



def get_logger(level):

    uamqp_logger = logging.getLogger("uamqp")

    if not uamqp_logger.handlers:

        handler = logging.StreamHandler(stream=sys.stdout)

        handler.setFormatter(logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s'))

        uamqp_logger.addHandler(handler)

    uamqp_logger.setLevel(level)

    return uamqp_logger



log = get_logger(logging.DEBUG)



def _generate_sas_token(uri, policy, key, expiry=None):



    if not expiry:

        expiry = time() + 3600  # Default to 1 hour.

    encoded_uri = quote_plus(uri)

    ttl = int(expiry)

    sign_key = '%s\n%d' % (encoded_uri, ttl)

    signature = b64encode(HMAC(b64decode(key), sign_key.encode('utf-8'), sha256).digest())

    result = {

        'sr': uri,

        'sig': signature,

        'se': str(ttl)}

    #if policy:

    #    result['skn'] = policy

    return 'SharedAccessSignature ' + urlencode(result)





def _build_iothub_amqp_endpoint_from_target(target, deviceendpoint):

#steve - reference global sas_token and auth_username because we will use it outside this function

    global sas_token

    global auth_username



    hub_name = target['hostname'].split('.')[0]



#steve - the format for a *device scoped* key for amqp is

# [deviceid]@sas.[shortiothubhubname]

# this is the same for both IoT Hub and IoT Edge.  This is a change from the original sample

# which used a 'hub scoped' key

    endpoint = "{}@sas.{}".format(target['device'], hub_name)

#steve - grab the username for use later..  before the URL-encoding below

    auth_username = endpoint



    endpoint = quote_plus(endpoint)

    sas_token = _generate_sas_token(target['hostname'] + deviceendpoint, target['key_name'],

                                    target['access_key'], time() + 36000)



#  steve - the first line below is used for talking to IoThub, the second for IoT Edge

#  basically we are just changing the connection endpoint

#    endpoint = endpoint + ":{}@{}".format(quote_plus(sas_token), target['hostname'])

    endpoint = endpoint + ":{}@{}".format(quote_plus(sas_token), target['edgehostname'])

    return endpoint



def test_iot_hub_send(live_iothub_config):

#steve - reference my globals set earlier

    global sas_token

    global auth_username



    msg_content = b"hello world"

    app_properties = {"test_prop_1": "value", "test_prop_2": "X"}

    msg_props = uamqp.message.MessageProperties()

#steve - honestly dunno what this property does :-), but we aren't going devicebound, so nuked it

#    msg_props.to = '/devices/{}/messages/devicebound'.format(live_iothub_config['device'])

    msg_props.message_id = str(uuid4())

    message = uamqp.Message(msg_content, properties=msg_props, application_properties=app_properties)



#steve - the original sample was set up for cloud-to-device communication.  I changed the 'operation'

# to be device-to-cloud by changing the operation to /devices/[device id]/messages/events

    #operation = '/messages/devicebound'

    deviceendpoint='/devices/{}'.format(live_iothub_config['device'])

    operation = deviceendpoint + '/messages/events'

    endpoint = _build_iothub_amqp_endpoint_from_target(live_iothub_config, deviceendpoint)



    target = 'amqps://' + endpoint + operation

    log.info("Target: {}".format(target))



#steve - this is where the magic happens for Edge.  We need a way to specify the 

# path to the root ca cert used for the TLS connection to IoT Edge.  So we need to 

# manually created the SASLPlain authentication object to be able to specify that

# and then pass it to the SendClient method below.  All of that is not necessary

# just to talk directly to IoT Hub as the root Baltimore cert for IoT Hub itself is buried

# somewhere in the uamqp library

# if you are connecting to IoT Hub directly, you can remove/comment this line

    auth_settings = uamqp.authentication.SASLPlain(live_iothub_config['edgehostname'], auth_username, sas_token, verify=live_iothub_config['edgerootcacert'])



#steve - for iot hub  (simple because we don't have to worry about the edge TLS cert

#    send_client = uamqp.SendClient(target, debug=True)

# for iot edge

    send_client = uamqp.SendClient(target, debug=True, auth=auth_settings)

    send_client.queue_message(message)

    results = send_client.send_all_messages()

    assert not [m for m in results if m == uamqp.constants.MessageState.SendFailed]

    log.info("Message sent.")



if __name__ == '__main__':

    config = {}

#steve - changed from environment variables to hardcoded, just for this sample

#    config['hostname'] = os.environ['IOTHUB_HOSTNAME']

#    config['device'] = os.environ['IOTHUB_DEVICE']

#    config['key_name'] = os.environ['IOTHUB_SAS_POLICY']

#    config['access_key'] = os.environ['IOTHUB_SAS_KEY']

    config['hostname'] = 'your long iothub name'  # e.g. 'sdbiothub1.azure-devices.net'

    config['device'] = 'your device id'  # e.g. 'amqptest'

    config['key_name'] = ''  # leave empty string

    config['access_key'] = 'your primary or secondary device key'   # e.g 'P38y2x3vdWNGu7Fd9Tqq9saPgDry/kZTyaKmpy1XYhg='

#steve - the FQDN of your edge box (i.e. mygateway.local)

# it MUST match the 'hostname' parameter in config.yaml on your edge box

# otherwise TLS certificate validation will fail

    config['edgehostname'] = 'your FQDN for your edge box'  # e.g. 'mygateway.contoso.local'

#steve - path to the 'root ca' certificate used for the IoT Edge TLS cert

    config['edgerootcacert'] = 'the path to your root ca cert for edge'  # e.g. '/home/stevebus/edge/certs/azure-iot-test-only.root.ca.cert.pem'



    test_iot_hub_send(config)

The code is a little hard to read in blog format, so feel free to copy/paste into your favorite python editor to view it.

The key changes are to the line that generates the ‘username’ for the connection string, changing it from a ‘hub level’ key to a device-level key


endpoint = "{}@sas.{}".format(target['device'], hub_name)

and the two lines that allow me to customize the SASLPlain authentication information to add in the path to the IoT Edge root CA cert


auth_settings = uamqp.authentication.SASLPlain(live_iothub_config['edgehostname'], auth_username, sas_token, verify=live_iothub_config['edgerootcacert'])



send_client = uamqp.SendClient(target, debug=True, auth=auth_settings)

When you run the sample, you’ll see a ton of debug output. At the top you should see your connection string dumped out, but most importantly, if it works, you should see a line like this somewhere in the middle of the output


2019-03-07 19:00:15,893 uamqp.client DEBUG Message sent: &lt;MessageSendResult.Ok: 0>, []

This indicates a successful sending of a message to IoT Hub/Edge.

Enjoy, and as always, feel free to ping me via my contact page and/or via comments here

Quick update on MQTT and message routing in IoT Hub and IoT Edge

I’ve made an update to my posts on using a standard MQTT client and using Node-Red to connect to IoT Edge via MQTT.

One thing I missed in the original post, that I needed to figure out today for a customer, is that if you want to route messages in IoT Edge based on the message body, you need to use the contentType and contentEncoding fields to tell edgeHub that the content is, in fact, JSON, and also what encoding it is (i.e. utf-8, utf-16, etc). This is not unique to IoT Edge, but a requirement for IoT Hub itself as well.

The way you do that is by appending ‘properties’ to the end of your MQTT topic to indicated the content type and the content encoding. In MQTT, those are the $.ct and $.ce properties, respectively.. So, regardless of your client, you do that by appending the values $.ct=application/json and $.ce=utf-8, after URL encoding them, to your topic, like this:

devices/[device_id]/messages/events/$.ct=application%2Fjson&$.ce=utf-8

where [device_id] is obviously the device id in iot hub of your sending device. This allows you to do things like this in your IoT Edge routing (pretend we are sending a message like {“messageType”:”alert”, ……})

{
"routeToAlertHandler":"FROM /messages/* WHERE $body.messageType='alert' INTO ........
}

Enjoy! and, as always, if you have any questions, hit me up in the comments section or twitter/etc (links in my ‘about me’ page)

Getting started!

Hi! At the prompting of a few of my compatriots, I’ve decided to (finally) start a blog. This blog will, mostly, focus on Internet of Things solutions with Microsoft Azure and it’s related technologies, but as with the randomness of topics inside my head, may wander from time to time! If nothing else, it gives me a place to collect all my random tips and tricks so I can stop trying to remember exactly where it was that I saved that little nugget of random information that helps me get through the day!

Hope you enjoy it! Check out the “About Me” page for a little background.