Connect MQTT client to Azure IoT Edge

NOTE:  Edited on 9/13/2018 to fix bugs in code and make it send data in a loop, just to be more realistic test

NOTE:  Edited on 2/14/2019 (Happy Valentine’s Day!) to add the contentType and contentEncoding values to allow you to route on message bodies.

As you may know, MSFT provides some nice SDKs to connect devices to IoT Hub.  Those SDKs abstract away much of the complexity of connecting, protocol abstraction, device twins, direct methods, etc.

However, there are many reasons, in particular “brownfield” devices, where you might prefer to connect via an open source MQTT library, like the nice Paho MQTT library, directly to IoT Hub.  The IoT product group has put together a very good description and sample code for connecting your MQTT device directly to IoT Hub.

With the general availability of Azure IoT Edge, the natural next question is “Can I connect my MQTT devices to IoT Hub *through* IoT Edge?”.  This lets you take advantage of all the nice local processing and cloud services you can pull to the Edge, but make minimal changes to your MQTT-based IoT devices.

The short answer is YES!  For the (only slightly) longer answer, keep reading.

There are only a few steps needed to make this happen.

  • If you haven’t done it yet, you need to set up IoT Edge as a transparent gateway.   The instructions for that are here (linux) and here (windows)….(don’t read too much into the “transparent” part of that, as you can still add other modules such as custom code, Azure ML, Stream Analytics, etc)
  • The MQTT client will establish a TLS connection to the IoT Edge device.  As such, it needs to trust the server certificate that the edgeHub component of IoT Edge presents to it.  For the MQTT/TLS connection to work, depending on the MQTT client (I use the paho-mqtt library below, just like the IoT team did), you’ll likely need the “root ca” certificate that was used to generate the device ca certificate used in IoT Edge.  If you used our test scripts provided at the links in step 1 above, then the certificate you need is at $CERTDIR/azure-iot-test-only.root.ca.cert.pem  (where $CERTDIR is the same one from step 1).  If you used a “real” certificate authority, like Baltimore, DigiCert, etc, they each generally have a method to get their public signing certs.   (we happen to have a couple of them here).   You will need to have them in a file you can reference on the file system of the MQTT box.
  • Your MQTT client needs to be able to resolve the hostname of your IoT Edge box (i.e. you should be able to ping it by name).  If you used a “real” FQDN in DNS for your IoT Edge box, then you are good.  If you didn’t, you may have to add a “hosts” file entry (/etc/hosts on Linux or \windows\system32\drivers\etc\hosts on Windows) to resolve the name.   This name should match the “hostname” parameter from the IoT Edge’s config.yaml file.
  • Finally, we need to make a few minor modifications to the sample code provided by the IoT product group, as seen below…   The biggest change is that, while we still need to authenticate the device to IoT Hub with a valid SAS token for the IoT Hub (and thus the credentials stay the same as the direct case), the actual connection point will be edgeHub.

The code changes to the python script are shown below.  I’ve commented each change with a comment starting with #EDGE. Also, if you copy and paste the code, my blog editor doesn’t respect spaces, so you need to indent the lines under the on_xxxx function definitions, and the code under the while(True) statement

from paho.mqtt import client as mqtt
import ssl

import time

path_to_root_cert = '[path to root CA cert]' # e.g. './azure-iot-test-only.root.ca.cert.pem'
device_id = "[iot device id]" # e.g. "myIoTDevice"
sas_token = "[generated SAS token]" # e.g. SharedAccessSignature sr=exampleIotHub.azure-devices.net%2Fdevices%2FmyIoTDevice&sig=8%2Fo6sdsFE%2BplYLQJrxIo5Usx1iVV0gnySaVhkh7aNOk%3D&se=1563635795"
iot_hub_name = "[iothub short name]" #e.g. exampleIoTHub

#EDGE - the FQDN of the device.. for example: myedgedevice.local
edge_device_name = "[edge device name]"

def on_connect(client, userdata, flags, rc):
print ("Device connected with result code: " + str(rc))
def on_disconnect(client, userdata, rc):
print ("Device disconnected with result code: " + str(rc))
def on_publish(client, userdata, mid):
print ("Device sent message")

client = mqtt.Client(client_id=device_id, protocol=mqtt.MQTTv311)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_publish = on_publish

#EDGE - we need to add the + "/api-version=2016-11-14" (api version) to the end of the username. Technically, you can do this if you are
# talking directly to IoTHub as well, but it's optional. Edge seems to require it.
client.username_pw_set(username=iot_hub_name+".azure-devices.net/" + device_id + "/api-version=2016-11-14", password=sas_token)

client.tls_set(ca_certs=path_to_root_cert, certfile=None, keyfile=None, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1, ciphers=None)
client.tls_insecure_set(False)

# start paho's background processing

client.loop_start()

#EDGE - although we are still authenticating to IoTHub with the IoTHub based SAS token, we are actually connecting
# to IoT Edge device's MQTT endpoint

client.connect(edge_device_name, port=8883)

i=0

while(True):

payload = "{"message_id": %d}" % (i)

client.publish("devices/" + device_id + "/messages/events/", payload, qos=1)

time.sleep(5)

i=i+1

NOTE:  *if* your payload is JSON, and *if* you think you might want to do routing of the messages based on the *body* of the message, you need to send a content type of ‘application/json’ and a content encoding of ‘utf-8’  (or 16, etc).  to do that, we need to append that information (url-encoded) to the MQTT topic, which gets passed in as a system property.  So, in that case, the ‘publish’ call would look like this (note the ‘/$.ct=application%2Fjson&$.ce=utf-8’ appended to the MQTT topic)

client.publish(“devices/” + device_id + “/messages/events/$.ct=application%2Fjson&$.ce=utf-8”, payload, qos=1)

That’s it. At this point you should be able to run your script and see the successful connection reflected in your edgeHub logs (docker logs -f edgeHub) and, assuming you have the default route set (“FROM /* INTO $upstream”), see the data flowing into IoTHub. (note that the sample app only sends one message and then stops… CTRL-C to exit and run again).

Enjoy!

PIP install for Python SDK now works on Raspberry Pi!

The Azure IoT product group has been busy making improvements to our Python SDK for Azure IoTHub.  The biggest improvement is that they’ve put a fix in for the infamous “libboost” issue that prevented the PIP package from working on the Raspberry Pi.   Previously, if you wanted to use the Python SDK on a Raspberry Pi, you had to manually build from the source, a time and resource intensive process.

If you aren’t familiar with it, Boost allows you to invoke C code from Python (the Python SDK is built on the C SDK).

Base SDK install/usage

For now, they’ve updated the SDK to link against the current version of libboost that works on the latest version or Raspbian stretch (1.62.0).  Future improvements may remove this requirement, but for now, you need to make sure that libboost is installed on the Pi (it’s not, by default).  So use the following two statements to get the SDK installed and setup:

sudo apt-get install libboost-python1.62.0

pip install azure-iothub-device-client
After that, in your code, you can ‘import iothub_client’ and go to town!

IoT Edge/Docker

As a bonus (for writing IoT Edge modules for the RPI), i did a quick test to confirm that all of this works in a docker container too.  There will be official guidance and tooling from the IoT product group in the future, but for now, a simple Dockerfile like this works for creating a docker image that contains a Python app that uses our Python SDK…

FROM python:2.7.15-stretch

WORKDIR /usr/src/app

RUN apt-get update && apt-get install libboost-python1.62.0

COPY requirements.txt ./

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "-u", "./app.py" ]

the requirements.txt file, just contains one line:   “azure-iothub-device-client”

next up, I’ll be testing with a full blown IoT Edge module on the raspberry pi, and I’ll post the results soon

 

Azure IoTHub Auto-scaling

For my inaugural post, I figured, just to have it in one place, I’d revisit something I posted a few months ago on the Azure.com IoT blog.  In fact, doing this is one of the things that prompted me to start this one…   Anyway, a frequent requested feature for Azure IoT Hub is the ability to auto-scale the IoT Hub when you approach your message limit for the day.   It’s a feature that is definitely on our roadmap, but IoT Hub doesn’t natively support it today.   This blog post on azure.com and accompanying sample show you an example solution of how you can do it today.

Getting started!

Hi!    At the prompting of a few of my compatriots, I’ve decided to (finally) start a blog.  This blog will, mostly,  focus on Internet of Things solutions with Microsoft Azure and it’s related technologies, but as with the randomness of topics inside my head, may wander from time to time!  If nothing else, it gives me a place to collect all my random tips and tricks so I can stop trying to remember exactly where it was that I saved that little nugget of random information that helps me get through the day!

Hope you enjoy it!   Check out the “About Me” page for a little background.