Mosquitto MQTT broker to IoT Hub/IoT Edge

 

EDIT:  edited on 8/30 to change tls version to tls 1.2.  Seems that TLS 1.0 doesn’t work any more.  Thanks to Asish Sinha for the heads up.  Also updated the api-version to the latest

 

Earlier, I had a post on connecting an MQTT client to IoT Edge.

It seems like lately my team and I have had a lot of customers with brownfield equipment that can speak MQTT, but are either too old or too low powered (the devices, not the customers Smile)  to do MQTT over TLS.  It is also often the case that you have no real control over the MQTT topic(s) that the device sends events/messages over.  Additionally, many devices even imply “intelligence” or “data” into the topic structure, meaning the topic hierarchy itself conveys information vs. only having important information in the message payload.

Both a TLS connection, and sending data on a very specific topic, are current requirements to talk to either IoT Edge or IoT Hub itself over MQTT.   So, how do we overcome this impedance mismatch between what IoT Hub/Edge requires, and the equipment can do?   One way is to use a middle layer to do the translations.  A popular choice is the open source MQTT broker mosquitto from the Eclipse Foundation.   Mosquitto has a built-in option to set up an MQTT “bridge”, in which the broker will accept incoming messages over MQTT and then forward them as an MQTT client to another MQTT server.  The good news is, Mosquitto can listen to the unencrypted MQTT traffic (port 1883 by default), and then forward it along over a TLS-protected MQTTS connection (port 8883) via this bridge. 

That takes care of our MQTT vs. MQTTs issue.  But what about the any topic vs. a specific topic problem.  Unfortunately, IoT Hub and IoT Edge both only accept telemetry/event data on a specific MQTT topic:  devices/[device-id]/messages/events where [device-id] is the ID of the connected device.  That one is a little trickier, and will be addressed later in this post after we cover the basics of setting up the bridge.

A couple of notes/caveats before we get started:

  • I am NOT a mosquitto expert.  I’ve learned just enough to get this working Smile
  • This is certainly not the only way to solve this problem.  But is one way that seems to work pretty well.
Mosquitto Bridge Setup for IoT Hub/Edge

Before we can configure our Mosquitto MQTT bridge, there are a few pre-requisites to take care of 

  • If you don’t already have one, create an IoT Hub and create a device (only follow that one section) that will represent our Mosquitto broker. The messages in IoT Hub/Edge will appear as if they come from the broker as the IoT device.
  • If you are talking directly to IoT Hub, you can skip this step.  If you are wanting to route your messages through IoT Edge, you need to setup an IoT Edge device as a gateway.
  • Gather the TLS server-side root certificate.  In order for mosquitto to establish a TLS connection to either IoT Hub or IoT Edge, it needs to trust the server-side TLS certificate that will be presented to the broker when it tries to open the connection to IoT Hub/Edge.  Gathering the CA cert from which the TLS server-side cert was generated, the process differs slightly based on whether you are connecting to IoT Hub or IoT Edge.  Either way, save the cert to a file on the mosquitto server, we’ll use it later.

For IoT Hub, the TLS certificate chains up to the public DigiCert Baltimore Root certificate. You can create this file by copying the certificate information from certs.c in the Azure IoT SDK for C. Include the lines —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–, remove the ” marks at the beginning and end of every line, and remove the \r\n characters at the end of every line.  Name the file with a .pem extension.

For IoT Edge, use whatever root certificate you used to create the IoT Edge Device CA Certificate.  If you used our convenience scripts to set up IoT Edge, that will be the azure-iot-test-only.root.ca.cert.pem found in the ‘certs’ folder where you ran the scripts

Now that we have our pre-req’s finished, we can do our Mosquitto  bridge setup.  This is done via the Mosquitto configuration file.   There may be other things in that file, however, below is an example configuration entry.



# Bridge configuration

connection azureiot-bridge

log_type debug

address [edge or hub fqdn]:8883

remote_username [iothub-shortname].azure-devices.net/[device-id]/api-version=2019-03-31

remote_password [sas-token]

remote_clientid [device-id]

bridge_cafile [iot hub or edge root ca cert]

try_private false

cleansession true

start_type automatic

bridge_insecure false

bridge_protocol_version mqttv311

bridge_tls_version tlsv1.2

notifications false

notification_topic events/



topic devices/[device-id]/messages/events/# out 1

The parts in bold need to be replaced with your values, where

  • [iot hub or edge FQDN] is the DNS name of either your IoT Hub (including the .azure-devices.net) or your IoT Edge device  (i.e. whatever name was used as the ‘hostname’ in config.yaml on IoT Edge)
  • [iothub-shortname] is the name of your IoT Hub  (e.g. ‘myiothub’) without the .azure-devices.net
  • [device-id] is the name of the IoT device created in IoT Hub to represent this broker
  • [sas-token] is a SAS token generated for that device-id in that hub
  • [iot hub or edge root ca cert] is the full path to the root certificate file you created earlier
  • All values are case sensitive.

The very last line (that starts with the word ‘topic’) subscribes the bridge to all messages that are sent with the topic structure of ‘devices/[device-id]/messages/events/#’ (the # is a wildcard to include any sub-topics). When a message that fits that topic structure gets published, the bridge will get it and pass it along to the IoT Hub/Edge.

restart your Mosquitto broker using the updated configuration file.  You should see debug output indicating that it has connected the bridge (and if you are using IoT Edge, you should see debug output in the edgeHub logs showing the connection from the broker)

If you want to test the connection, you can send a test message using the mosquitto_pub command, using the following command (replacing [device-id] with your device id you created above):



mosquitto_pub -t devices/[device-id]/messages/events/ -m "hello world!"

The trailing slash is important and required. You should see the message above be forwarded by the MQTT bridge to either IoT Hub or IoT Edge.

If you are fortunate enough to have full control over your MQTT topic structure from your devices, and there is no intelligence in your topic structure, you’re done.  Congrat’s and have fun!  You can just point your MQTT clients at the broker address (making sure you update the MQTT topic to point to devices/[device-id]/messages/events/) and rock and roll.

However, for the use cases where you don’t have MQTT topic control, or there is intelligence in your topic hierarchy, keep reading.

MQTT Topic Translation

Unfortunately, this is where things get a little less “clean”.  The mosquitto MQTT bridge has no ability to “rewrite” or completely change the topic structure of the messages it receives.  One way to do it is to write a simple client that subscribes to all potential topics from which the MQTT devices might send data, and then resend the payload after translating the MQTT topic into the IoT Hub/Edge required topic structure. 

Below is a simple python script that I wrote to do the translation as an example.  This sample subscribes to all topics (#  – the wildcard) and, if the message doesn’t already use the IoT Hub/Edge topic structure, it simply resends the message payload using the hub/edge topic.



import paho.mqtt.client as mqtt

import time



#replace [device-id] with your device you created in IoT Hub.

iothubmqtttopic = "devices/[device-id]/messages/events/"



# this sample just resends the incoming message (on any topic) and just

# resends it on the iothub topic structure.  you could, of course, do any

# kind of sophisticated processing here you wanted...

def on_message(client, userdata, message):

     global iothubmqtttopic

     if(message.topic != iothubmqtttopic):

         messageStr = str(message.payload.decode("utf-8"))

         print("message received " ,messageStr)

         print("message topic=",message.topic)

         client.publish(iothubmqtttopic, messageStr)



# replace <broker address> with the FQDN or IP address of your MQTT broker

broker_address="[broker address]"



print("creating new instance")

client = mqtt.Client("iottopicxlate") #create new instance

client.on_message=on_message #attach function to callback

print("connecting to broker")

client.connect(broker_address) #connect to broker



print("Subscribing to all topics")

client.subscribe("#")



client.loop_forever() #stop the loop

Of course, this is one extremely simple example, that just passes along the same message payload and swaps out the message topic.  You can, of course, add any kind of sophisticated logic you need.  For example, you could parse the topic hierarchy, pull out any ‘intelligence’ in it, and add that to the message payload before sending.

if you want to test this, copy this python script to a file, edit it to add your device id and URI of your mosquitto broker, and run it.  You can then try….



mosquitto_pub –t /any/topic/structure/you/want –m "hello world"

You should see the python script receive the file, do the translation, and republish the message.  Then the mosquitto bridge will forward the new message along to IoT Hub/Edge.

16 thoughts on “Mosquitto MQTT broker to IoT Hub/IoT Edge”

  1. Hi,

    We are trying to address a similar problem wherein we will have an opaque IoT Edge gateway which will receive data from downstream devices over MQTT and this data will be sent to IoT Hub. I was thinking of using MQTT adapter as implemented in Azure Protocol Gateway which uses Dotnetty. Like take bits and pieces from the protocol gateway code to have a MQTT listener. What are your thoughts on using dotnetty and mqtt adapter from protocol gateway.

    1. hey Boomjosh – thanks for the comment. That’s not a bad idea at all. I will say that we (MSFT) haven’t really invested much lately in the protocol gateway, so that code is pretty dated. However, I think it would represent some good solid code to start from for an MQTT listener.

      My goal with this post was mostly to give an “as little code as possible” approach, with the only code you need to write being the script/app to do the topic translation. That way you can re-use all the work that has gone into both the Mosquitto listener as well as the Mosquitto bridge. As I mentioned, I did the approach above on behalf of a customer and have several customers doing that approach that just wanted to avoid as much coding as possible.

      I’ve been thinking about taking all of the above and ‘containerizing’ it to an IoT Edge module, but the day job has gotten in the way. 🙂

      But if you aren’t afraid of writing/stealing code, then certainly nothing wrong with the dotnetty/protocol gateway code.

      I will also say, depending on your implementation timing, being able to ingest raw “real” MQTT data (i.e. no TLS, arbitrary topics) into IoT Edge is a challenge we are well aware of internally and I’ve been working with a couple of different engineering teams on releasing an MSFT-sanctioned approach. I don’t have a date to share on that yet, but I’ll be posting again here on the blog when we have something to show for that work.

  2. Hello,

    I’ve followed same steps (mentioned in article), but i’m unable to see any message on my IOT Hub.
    While publishing the message i’m getting “Unknown error”.
    Please help!

  3. Hello Steve, thanks for the article, but looks like using tlsv1 throws the below error and the remote_username does not need the API-version anymore. It worked with tlsv1.2

    From: $ sudo tail -n 200 /var/log/mosquitto/mosquitto.log

    1567163115: mosquitto version 1.6.4 starting
    1567163115: Config loaded from /etc/mosquitto/mosquitto.conf.
    1567163115: Opening ipv4 listen socket on port 1883.
    1567163115: Opening ipv6 listen socket on port 1883.
    1567163115: Bridge local.TestDevice doing local SUBSCRIBE on topic devices/TestDevice/messages/events/#
    1567163115: Connecting bridge (step 1) azureiot-bridge (IOTHubMorocco.azure-devices.net:8883)
    1567163116: Connecting bridge (step 2) azureiot-bridge (IOTHubMorocco.azure-devices.net:8883)
    1567163116: Error: Protocol tlsv1 not supported.
    1567163121: Bridge local.TestDevice doing local SUBSCRIBE on topic devices/TestDevice/messages/events/#
    1567163121: Connecting bridge (step 1) azureiot-bridge (IOTHubMorocco.azure-devices.net:8883)
    1567163121: Connecting bridge (step 2) azureiot-bridge (IOTHubMorocco.azure-devices.net:8883)
    1567163181: Socket error on client local.TestDevice, disconnecting.
    1567163500: mosquitto version 1.6.4 starting
    1567163500: Config loaded from /etc/mosquitto/mosquitto.conf.
    1567163500: Opening ipv4 listen socket on port 1883.
    1567163500: Error: Address already in use

    Made the following changes in the mosquitto.conf file in /etc/mosquito/mosquitto.conf and then publishing messages worked to IOT hub

    # Bridge configuration
    connection azureiot-bridge
    log_type all
    address [IOT hub shortname].azure-devices.net:8883
    remote_username [IOT hub shortname].azure-devices.net/[device-id]
    remote_password [sas-token]
    remote_clientid [device-id]
    bridge_cafile /home/MQTTbroker/cert1.pem
    try_private false
    cleansession true
    start_type automatic
    bridge_insecure false
    bridge_protocol_version mqttv311
    bridge_tls_version tlsv1.2
    notifications false
    notification_topic events/

    topic devices/[device-id]/messages/events/# out 1

    1. Thanks Asish.. good to know. I’ll update the sample with the tlsv1.2 setting. I know in many Azure services in general, we are removing TLS 1.0 as an accepted protocol, but I had been told that it didn’t apply to IoT Hub. Maybe that’s wrong. Also, regarding the api-version.. I believe that it has always worked without that for IoT Hub, but it was required for IoT Edge. And since it worked for both, I just left it in there. I’ll check and see if it’s still required for IoT Edge. I’d love to get rid of it 🙂

    2. how did you get to manage with
      1567163181: Socket error on client local.TestDevice, disconnecting.
      this error??

      because I’m getting
      1591598117: Connection Refused: not authorised
      1591598117: Socket error on client local.MQTTdevice, disconnecting.
      and don’t know why..
      any help..?

      1. You’ll need to share more details about your setup. Without posting any ‘secrets’, can you describe your setup, how you generated your certificates, your config.yaml parameters, your mosquitto conf, etc. Going to need more information to troubleshoot

  5. Pingback: Industrial Internet of Things and Cloud – ideas to build Robustness & recover data Lost in Transit – Technical Blog on Architecture

  6. Hello Steve,
    Thanks for the great article, it helps us a lot.
    Everything is working as expected for SAS device.
    In our use case, we are having a device created using X509 certificates in Azure IOT hub. Now how to use the device certificates in bridge configuration file to make the connection through mqtt client.
    Thanks in advance.

  7. Hi,

    I have the requirement where I am receiving the telemetry data from various devices on local network using mqtt protocol but to send it over to Azure IOTHub I need to use different protocol. I look at the IOTHub and it supports AMQP. So looking at the example where you are translating the messages, can we use some amqp client to send it to IOTHub.

    Tahnks in advance.

  8. Here is my Mosquitto broker Config file

    # Bridge configuration
    connection azureiot-bridge
    log_type all
    address UrbanAustinIO.azure-devices.net:8883
    remote_username UrbanAustinIO.azure-devices.net/GFG-MQTT
    remote_password [sharedsignaturestring]
    remote_clientid GFG-MQTT
    bridge_cafile c:/Users/au911271/Desktop/TLSCertificate.pem
    try_private false
    cleansession true
    start_type automatic
    bridge_insecure false
    bridge_protocol_version mqttv311
    bridge_tls_version tlsv1.2
    notifications false
    notification_topic events/

    topic devices/GFG-MQTT/messages/events/# out 1

    I get the following error:
    1613361209: Connecting bridge azureiot-bridge (UrbanAustinIO.azure-devices.net:8883)
    1613361209: Bridge GFG-MQTT sending CONNECT
    1613361209: Received CONNACK on connection local.GFG-MQTT.
    1613361209: Connection Refused: not authorised
    1613361209: Client local.GFG-MQTT closed its connection.
    1613361210: mosquitto version 2.0.7 terminating

    Any help will be highly appreciated

    1. Hey John,

      did you figure this out? The stuff I see above looks ok. How did you generate your ShareAccessSignature string? And what does it look like? (change a few random characters in the middle of it to make it invalid….)

  9. Hi Steve and congrats for the material !

    I tested mosquitto againt my azur hub Iot with :
    mosquitto_pub -d -h xxxx.azure-devices.net -p 8883 -i yyyy-u “xxx.azure-devices.net/yyyy/?api-version=2018-06-30” -P “…..” -t “devices/yyyy/messages/events/” -m “12” -V mqttv311 –cafile ./rootCA.pem -q 1

    and it works fine.

    then, trying to configure as a bridge, so with the same parameters in the conf file, Connect is sent but no ACK

    Strange to notice that when inspecting wireshark TCP log, I can see TLS handshake starting with Client Hello when using mosquitto_pub CLI command

    But with the bridge, nothing is sent to start TLS handshake… so it can not proceed futher .

    My log is :
    2022-01-18T17:51:24: mosquitto version 2.0.14 starting
    2022-01-18T17:51:24: Config loaded from .\mosquitto.conf.
    2022-01-18T17:51:24: Starting in local only mode. Connections will only be possible from clients running on this machine.
    2022-01-18T17:51:24: Create a configuration file which defines a listener to allow remote access.
    2022-01-18T17:51:24: For more details see https://mosquitto.org/documentation/authentication-methods/
    2022-01-18T17:51:24: Opening ipv4 listen socket on port 1883.
    2022-01-18T17:51:24: Opening ipv6 listen socket on port 1883.
    2022-01-18T17:51:24: Bridge local.yyyy doing local SUBSCRIBE on topic devices/yyyy/messages/events/#
    2022-01-18T17:51:24: Connecting bridge bridge (xxxxx.azure-devices.net:8883)
    2022-01-18T17:51:24: Bridge yyyyy sending CONNECT
    2022-01-18T17:51:24: mosquitto version 2.0.14 running
    2022-01-18T17:51:24: Client local.yyyyy closed its connection.

    If you have any clue on this ?

Comments are closed.