MQTT with TLS Issue

I don’t like asking for help since many questions are duplicates and @brocaar is a legend that has done way more than enough to make this an amazing solution but this is a thing I’ve been actively trying to do for more than 60 hours cumulatively and I’m becoming desperate so any help is highly appreciated.

At some point I gave up on TLS and wanted to do a sanity check so I spun up an MQTT server that doesn’t use TLS and confirmed that this entire setup of mine works perfectly without TLS, therefore confirming that the issues that I’m having are in regards to TLS.

I’m willing to regenerate my CA certificate and all the server certificates I used for my entire lab (2-3 day long process considering I’ll need to reconfigure all of my IoT devices, servers, services, and client devices) if that’s something that’s gonna help me make Chirpstack work.

Overview of my setup

  • The whole lab is local only with all the devices being on my own network, so all certificates are self signed.
  • MultiTech Conduit gateway with Chirpstack Gateway Bridge running on it.
  • Mosquitto MQTT Broker running on my server, with my own self signed CA certificate and broker certificate. This server is only accessible from the local network. It doesn’t use client side certificates. It uses usernames and passwords as well as ACLs as everything is encrypted with TLS. This setup has been working great for 8+ years on more than 100 different devices (off-the-shelf as well as DIY).
  • Chirpstack running in Docker (image: chirpstack/chirpstack:latest) along with Redis and Postgres.
  • My CA certificate is generated as follows:
$ openssl genrsa -des3 -out rootCA.key 4096
$ openssl req -x509 -new -sha256 -key rootCA.key -days 3650 -out rootCA.crt
  • The MQTT Broker TLS certificate is generated as follows:
 $ openssl genrsa -out mosquitto.key 2048
$ openssl req -new -out mosquitto.csr -key mosquitto.key -sha256 
$ openssl x509 -req -in mosquitto.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out mosquitto.crt -days 1800 -sha256
  • This is my chirpstack.toml MQTT config:
[integration.mqtt]
    event_topic="lorawan/application/{{application_id}}/device/{{dev_eui}}/event/{{event}}"
    command_topic="lorawan/application/{{application_id}}/device/{{dev_eui}}/command/{{command}}"
    json=true
    server="ssl://$MQTT_BROKER_HOST:8883/"
    username="redacted"
    password="redacted"
    qos=0
    clean_session=false
    client_id="chirpstack"
    keep_alive_interval="30s"
    ca_cert="/etc/chirpstack/certs/rootCA.crt"
    tls_cert=""
    tls_key=""
  • This is my region MQTT config:
  [regions.gateway]
    force_gws_private=false
    [regions.gateway.backend]
      enabled="mqtt"
      [regions.gateway.backend.mqtt]
        event_topic="lorawan/gateway/+/event/+"
        command_topic="lorawan/gateway/{{ gateway_id }}/command/{{ command }}"
        server="ssl://$MQTT_BROKER_HOST:8883"
        username="redacted"
        password="redacted"
        qos=0
        clean_session=false
        client_id="redacted"
        keep_alive_interval="30s"
        ca_cert="/etc/chirpstack/certs/rootCA.crt"
        tls_cert=""
        tls_key=""
  • This is my Mosquitto MQTT Broker Configuration file:
persistence false
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log

log_dest topic
log_type error
log_type warning
log_type notice
#log_type information

connection_messages true
log_timestamp true
allow_anonymous false
password_file /mosquitto/config/mqtt_passwords
acl_file /mosquitto/config/mqtt_acl

listener 8883
cafile /mosquitto/config/certs/rootCA.crt
certfile /mosquitto/config/certs/mosquitto.crt
keyfile /mosquitto/config/certs/mosquitto.key
tls_version tlsv1.3 #I've tried tlsv1.1, tls1.2 and tls1.3
  • I found out my CA was TLS v1, so I regenerated the CA certificate to be v3 but this didn’t help at all, as I’m getting the same issue as shown below.

The Issue

Whatever I do, I get the following error in my Docker (as well as on the gateway itself, but I assume that when the issue is resolved on docker, it’s going to be easy to get the gateway working as well):

ERROR chirpstack::gateway::backend::mqtt: MQTT error error=TLS: I/O: invalid peer certificate: Other(OtherError(UnsupportedCertVersion))  
ERROR chirpstack::integration::mqtt: MQTT error error=TLS: I/O: invalid peer certificate: Other(OtherError(UnsupportedCertVersion))

Maybe you’ve already done this but run

openssl x509 -in <certificate>.crt -text -noout | grep "Version"

On each of your certificates (mosquitto chain and ca). Are they all v3 like your CA? The first step to help debug an unsupportedcertversion error would be to know the versions of all of the certs.

Also try this youself:

openssl s_client -connect <MQTT_BROKER_HOST>:8883 -CAfile /etc/chirpstack/certs/rootCA.crt

To confirm the issue is with the certs and not with Chirpstacks handling of them.

Thanks for the suggestion, I ran the commands and all the certificates in the chain were v3 already.

TL;DR: Chirpstack in Docker now works, but Chirpstack Gateway Bridge on the MultiTech Conduit gateway doesn’t.

Solution for Chirpstack in Docker

Fortunately, I have managed to get Chirpstack in docker to work by regenerating the root CA and then generating the mosquitto certificate in a different way.
Here is exactly what I did for others who may be struggling with this as well:

  • Generated new root CA using the same method as shown in my first post.
  • Created a mosquitto.csr.cnf file with the following content:
    [req]
    default_bits = 2048
    prompt = no
    default_md = sha256
    distinguished_name = dn
    
    [dn]
    C=US # Example country code for mosquitto certificate
    O=Mosquitto MQTT Broker # Example name of certificate
    CN=myserver.lan # Example hostname of server where mosquitto is hosted
    
  • Created a mosquitto_v3.ext with the following inside:
    authorityKeyIdentifier=keyid,issuer
    basicConstraints=CA:FALSE
    keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
    subjectAltName = @alt_names
    extendedKeyUsage=serverAuth
    
    [alt_names]
    DNS.1 = myserver.lan # Example hostname with .lan
    DNS.2 = myserver # Example hostname of server where Mosquitto is hosted
    IP.1 = 192.168.1.15 # Example IP of Mosquitto Broker
    
  • Ran the following command to generate a private key and certificate signing request:
    openssl req -new -sha256 -nodes -out mosquitto.csr -newkey rsa:2048 -keyout mosquitto.key -config <( cat mosquitto.csr.cnf )
    
  • Ran the following command to generate the certificate and sign it with my CA:
    openssl x509 -req -in mosquitto.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out mosquitto.crt -days 825 -sha256 -extfile mosquitto_v3.ext
    
  • Transferred the mosquitto.key and mosquitto.crt to Mosquitto MQTT Broker configuration folder of course.
  • Copied the rootCA.crt to Chirpstack (/etc/chirpstack/certs/rootCA.crt, which is mapped in a docker volume)

Now, everything is working on the docker side of things and this is what I get in the log files of Chirpstack:

INFO chirpstack::integration::mqtt: Initializing MQTT integration
INFO chirpstack::integration::mqtt: Configuring client with TLS certificate, ca_cert: /etc/chirpstack/certs/rootCA.crt, tls_cert: , tls_key: 
INFO chirpstack::integration::mqtt: Connecting to MQTT broker server_uri=ssl://192.168.1.15:8883/ client_id=chirpstack clean_session=false
INFO chirpstack::gateway::backend: Setting up gateway backends for the different regions
INFO chirpstack::integration::mqtt: Starting MQTT event loop
INFO chirpstack::gateway::backend::mqtt: Configuring client with TLS certificate, ca_cert: /etc/chirpstack/certs/rootCA.crt, tls_cert: , tls_key: 
INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=eu868 server_uri=ssl://192.168.1.15:8883 clean_session=false client_id=chirpstack-region-gw
INFO chirpstack::integration::mqtt: Subscribing to command topic command_topic=lorawan/application/+/device/+/command/+
INFO chirpstack::gateway::backend::mqtt: Subscribing to gateway event topic region_id=eu868 event_topic=$share/chirpstack/lorawan/gateway/+/event/

Issue with Chirpstack Gateway Bridge on MultiTech Conduit LoRaWAN Gateway

I uploaded the new rootCA.crt to the MultiTech Conduit Gateway along with the same credentials that worked without TLS (redacted):

This gateway is pretty old but it runs the newest MultiTech mPower Edge Intelligence Application Enablement Platform (firmware version 6.3.4) with the official Chirpstack Gateway Bridge add-on installed, which is exactly what I’m filling out in the screenshot.

When I save and apply the configuration and check the log files of my Mosquitto MQTT Broker, I keep getting the following error:

1731778295: New connection from 192.168.1.15:36494 on port 8883.
1731778295: OpenSSL Error[0]: error:0A000412:SSL routines::ssl/tls alert bad certificate
1731778295: Client <unknown> disconnected: Protocol error.

I have:

  • Triple checked if the root CA is correctly copied.
  • Figured out that the frontend actually saves everything (including the CA that I pasted in the text field) in /var/run/lora/1 on the gateway itself, so I checked it there as well. Interestingly, the official Chirpstack package saves the certificate I pasted as “ca_cert.txt”, but it has the exact same contents as my rootCA.crt. In the chirpstack-gateway-bridge.toml in the same folder, the file is correctly referenced with ca_cert = "ca_cert = "/var/run/lora/1/ca_cert.txt"
  • Added my rootCA.crt to the Administration > X.509 CA Certificates on the gateway web interface.

One note: I can’t use the Chirpstack MQTT Forwarder on this gateway as it doesn’t work due to the LoRaWAN Card being used in this gateway being too old (version 1.0.0). I learned this the hard way after many debugging hours and somebody on the forums mentioned that only this Official Chirpstack Add-On works with this card.

Anyways, I have no idea what I’m doing wrong, and again, any help is greatly appreciated.

Liam, thank you so much. This is one of those cases where you can’t even believe your own stupidity.
I’ve wasted so much time and effort only to run your command again:
openssl s_client -connect <MQTT_BROKER_HOST>:8883 -CAfile /etc/chirpstack/certs/rootCA.crt
but this time from the gateway itself, just to check what’s up, and I found a single line in that mass of text that said:

SSL handshake has read 3208 bytes and written 373 bytes
Verification error: certificate is not yet valid

It was the time and date. The gateway’s date was defaulting to 2021.
I set the time properly and now everything is working.
I’m going offline to drown 2 beers right now. Thanks again.

3 Likes