Azure Device Provisioning Server over MQTT using x509 Certificates

In a previous post, I showed you how to register a device with Azure’s Device Provisioning Server (DPS) over raw MQTT.  A reader/commenter asked how the process would differ if we used x.509 certificate based authentication vs. the SAS-token based authentication that the article was based on.

Since it’s inevitable that I’ll run across this in a customer situation, I thought I’d tackle it.  Based on the knowledge from the previous article, as well as my article on DPS over the REST APIs, it was pretty straightforward.  The process was nearly identical except for a few fields in the connection information, specifically specifying the iothub root cert, the device cert/key, and leaving off the SAS token.  I’ll cover the details below.

Generating Certs

The steps for generating the device certificates and creating the enrollment in DPS is the same process as outlined in my DPS over REST API article.  Specifically the sections titled “Prep work (aka – how do I generate test certs?)” and “X.509 attestion with individual enrollments- setup”, so I won’t repeat them here…    For the screenshots below, I called my enrollment registration id ‘dpstestdev01’.

The only other thing you need is the IoT Root CA cert.  This is the Baltimore-based root ca cert from which all the IoT Hub and DPS  “server-side” TLS certificates are generated.   The client needs this to validate that it is indeed talking to the genuine microsoft endpoint and not a ‘man in the middle’.   The easiest way to get this cert, is to open this file from the Azure IoT C SDK, copy everything from (and including) line 23 to (and including) line 43, then strip out the quotes at the beginning and end of each line, and strip off the ‘\r\n’ off the ends.  Save the file with a .pem extension.  We will call that the “DPS-root-CA” cert.

Client Setup

You can leverage any MQTT 3.1.1 client to talk to DPS, however, like in previous articles, I’m going to use MQTT.fx, which is an excellent MQTT GUI tool for ‘manually’ doing MQTT.  It allows you to get a really good feel for what’s happening under the covers without writing a bunch of code.

Through a series of screenshots below, I’ll show you my configuration.

The first step is to open Mqtt.Fx, click on the little ‘gear’ button next to the Connect button, and in the bottom right, click on the “+” button to create a new connection.   You can call it anything (I called mine ‘dpscert’ in the screenshots below)

general-settings

This screenshot shows the ‘general’ settings…

  • The type is MQTT Broker
  • The broker address is the global DPS endpoint (global.azure-devices-provisioning.net)
  • The port is the MQTTS (tls) port 8883
  • The client ID is the ‘registration id’ from DPS, specifically in this instance is the CN/Subject name you used for your device cert when you generated it
  • the only other change from the defaults is to explicitly choose MQTT version 3.1.1

user-credentials

This screenshot shows the user credentials.   For DPS, the user-id is of the form of:

{idScope}/registrations/{registration_id}/api-version=2019-03-31

where {idScope} is the idScope of your DPS instance.

Note that, unlike the SAS-Token case, the password is BLANK for x.509 authentication.

ssl-tls-certs

This screenshot is the most important one and the biggest difference from the SAS-Token case.

  • Make sure you explicitly select TLS version 1.2 (we don’t support older versions)
  • in our use case, we are using self-signed certificates, so choose that option
  • For the “CA files”, this the DPS-root-CA cert we captured from github earlier.  (the baltimore root cert)
  • For the Client Certificate file, this is the device certificate we created earlier
  • For the Client Key file, this is the private key for the device cert that we generated earlier.
  • Make sure and check the “PEM formatted” checkbox, as that’s the format our certs are in.

All the other tabs are just left default.

Click Ok to close this dialog.  Click the “Connect” button to connect to DPS.

From this point on, you subscribe and publish exactly like you did in the previous article and/or as specified in the official DPS documentation here.

Enjoy – and as always, let me know if you run into any issue.  Hit me up on Twitter (@BamaSteveB), email (steve.busby ( at ) microsoft.com) or in the comments below.

9 thoughts on “Azure Device Provisioning Server over MQTT using x509 Certificates”

  1. Hi! This is a really clear and detailed explanation on how to use the individual enrollments of the DPS, thank you very much 🙂
    With respect to the group enrollment, may I ask you if you know how to build the parameters which require the [registration_id], that actually does not exist in the group enrollment feature?

    1. Hi Nicola, Thank you for your comment. I’m not 100% sure I understand it, but I *think* you are asking where the registration_id comes from? The short answer is that you should use the same name for registration id that you used as the CN/SubjectName when you generated your device’s certificate. Does that help?

      1. Sorry for the delay. Your answer is very helpful! The MQTT documentation does not talk about group enrollments, so I wasn’t able to understand where the registration_id came from. May I ask you how did you understand it? Maybe I’m missing something…

  2. Hi,
    This is really helpful blogs. I have couple of questions as I am failing to setup the link using x.509.
    1) We created Root certificate, intermediate certificate from it and then device certificate from intermediate.
    2) In IoT DPS console, we uploaded and verified the above created “Root certificate” for Group enrollment.
    3) But in MQTT.fx, for CA files location, we are selecting a completely different Root certificate (a certificate which we took from github code). Why ?

    How does this work ? Because for me if I select the Root certificate which we generated using tool in CA files, MQTT.fx log shows Invalid certificate and if we select the GitHub extracted CA cert, it says unauthorised.

    1. Hey Abhijit,

      Thank you for your comment. I could see how that is confusing, because that particular dialog really mixed two different concepts. The “CA file” is used by MQTT.fx to verify the certificate *returned by DPS*. When the MQTT.fx client opens a TLS connection to DPS, it returns a server-side TLS certificate (*.azure-devices-provisioning.net). The certificate that you are putting in the “CA file” entry is the root certificate at the top of the signing chain for that DPS cert, which is a Digicert Baltimore root cert (that we snagged from github)

      The other two certificate entries on that dialog are the cert and key for your device that you generated for your device

      Despite being in the same dialog box, the two have nothing to do with each other :-). Once you upload and verify the root certificate that you generated yourself using the scripts to DPS and verify it, you are done with that certificate and aren’t using it in the MQTT.fx client

      I’m not sure why using the DPS root cert from github isn’t working for you. My guess is something wrong with the formatting. If you open it in notepad (or nano on linux), it should look like this (no quotes, etc), **including** the BEGIN CERTIFICATE and END CERTIFICATE lines.

      —–BEGIN CERTIFICATE—–
      MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
      RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
      VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
      DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
      ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
      VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
      mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
      IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
      mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
      XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
      dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
      jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
      BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
      DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
      9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
      jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
      Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
      ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
      R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
      —–END CERTIFICATE—–

      Let me know if that helps

  3. Hi,

    I have followed both tutorials for REST and MQTT using the Intermediate CA Certificate Attestation and I get the same error below using Curl or MQTT.fx:

    {“errorCode”:401002,”trackingId”:”a5844a41-7342-42d5-bb8b-c28aca0aa13c”,”message”:”CA certificate not found.”,”timestampUtc”:”2020-11-28T07:57:17.3622479Z”}

    I have a Enrolment group with the Intermediate Certificate setup on the DPS.

    I have My MQTT client setup using the following details:

    username: SCOPE_ID/registrations/dps-test-device-01/api-version=2019-03-31

    I have self signed certificates ticked.

    CA cert – set to Baltimore
    Client cert – set to device client
    Client key – set to device key

    Appreciate you help.

    The error does not occur on the initial connection, it occurs on Publish after I have Subscribed:

    Subscribe – $dps/registrations/res/#
    Publish – $dps/registrations/PUT/iotdps-register/?$rid=2

    {
    “registrationId”: “dps-test-device-01”
    }

    Here is the log:

    2020-11-28 08:09:45,922 INFO — BrokerConnectorController : onConnect
    2020-11-28 08:09:45,923 INFO — ScriptsController : Clear console.
    2020-11-28 08:09:45,932 INFO — MqttFX ClientModel : MqttClient with ID dps-test-device-01 assigned.
    2020-11-28 08:09:46,149 INFO — MqttFX ClientModel : session present: false
    2020-11-28 08:09:52,013 INFO — SubscribeController : onSubscribe
    2020-11-28 08:09:52,035 INFO — MqttFX ClientModel : rebuildMessagesList()
    2020-11-28 08:09:52,035 INFO — MqttFX ClientModel : attempt to addRecentSubscriptionTopic
    2020-11-28 08:09:52,035 INFO — MqttFX ClientModel : addRecentSubscriptionTopic : de.jensd.mqttfx.entities.Topic@472ecf85
    2020-11-28 08:09:52,035 INFO — MqttFX ClientModel : attempt to add PublishTopic
    2020-11-28 08:09:52,039 INFO — MqttFX ClientModel : sucessfully subscribed to topic $dps/registrations/res/# (QoS 0)
    2020-11-28 08:09:55,980 INFO — PublishController : publish
    2020-11-28 08:09:55,984 INFO — MqttFX ClientModel : attempt to add PublishTopic
    2020-11-28 08:09:55,984 INFO — MqttFX ClientModel : sucessfully published message {
    “registrationId”: “dps-test-device-01”
    } to topic $dps/registrations/PUT/iotdps-register/?$rid=2 (QoS 0, Retained: false)
    2020-11-28 08:09:56,334 INFO — MqttFX ClientModel : messageArrived() with topic: $dps/registrations/res/401/?$rid=2
    2020-11-28 08:09:56,335 INFO — MqttFX ClientModel : messageArrived() added: message #1 to topic ‘$dps/registrations/res/401/?$rid=2’
    2020-11-28 08:09:56,345 INFO — MqttFX ClientModel : Broker connection lost: Resetting client.
    2020-11-28 08:09:56,360 INFO — ScriptsController : Clear console.
    2020-11-28 08:09:56,360 INFO — MqttFX ClientModel : rebuildMessagesList()
    2020-11-28 08:09:56,362 INFO — ScriptsController : Clear console.

    1. Hey Marc – so sorry for the delayed reply.. for some reason, WP doesn’t alert me with new comments. I know this was a while ago. Did you figure it out?

Comments are closed.