Mqtt integration and TLS

Good morning,
in the procedure below I put some comments and questions ( highlighted with ******) about my configuration to understand why something is wrong in generating certificates, thanks to your kindness and help. I hope this post can help someone else.

Mosquitto TLS configuration

Requirements

Before proceeding, please make sure that you have installed the cfssl utility. *****Done
You should also already have a working ChirpStack environment. ******Done

If using Debian or Ubuntu ( *******my version 22 04 3), this package can be installed using:

sudo apt-get install golang-cfssl ******Done

Generating CA
The CA certificate (and key) has two purposes:

It is used to sign certificates
It is used to validate certificates (checking it has been signed by the CA)

Save the following files to disk: *******Done ( I created a directory in my home)

ca-csr.json: *******Done

{
“CN”: “ChirpStack CA”,
“key”: {
“algo”: “rsa”,
“size”: 4096
}
}
ca-config.json: *******Done

{
“signing”: {
“default”: {
“expiry”: “8760h”
},
“profiles”: {
“server”: {
“expiry”: “8760h”,
“usages”: [
“signing”,
“key encipherment”,
“server auth”
]
}
}
}
}
Then execute the following command to generate the CA certificate and key:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca *******Done (certificate and key created)

Generate MQTT server-certificate
The MQTT server-sertificate is used to establish a secure TLS connection between the MQTT client (gateway or integration) and the MQTT broker.

Save the following file to disk and change example.com into the hostname that will be used by clients that will connect to the MQTT broker: ********Done

mqtt-server.json:
********Done, my question: I replaced example.com both on CN and hosts with this syntax: mydomain.duckdns.org, is that correct? That domain is actually the server where my gateways connect correctly.

{
“CN”: “example.com”,
“hosts”: [
“example.com”
],
“key”: {
“algo”: “rsa”,
“size”: 4096
}
}
Then execute the following command to generate the MQTT server-certificate:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile server mqtt-server.json | cfssljson -bare mqtt-server ********Done

Configure ChirpStack
You must configure ChirpStack with the CA certificate and CA key which you have generated in the previous step. ChirpStack will use the CA certificate (+ key) to sign the gateway or application MQTT integration client-certificates when generated using the web-interface.

Create a directory for the certificate files and copy these files using the following command:

mkdir -p /etc/chirpstack/certs *******Done
cp ca.pem /etc/chirpstack/certs *******Done
cp ca-key.pem /etc/chirpstack/certs *******Done

Depending your setup, you might need to modify the ownership and / or permissions of the created directory and files. *******Question: I dont’ know well this, I know that to access to chirpstack directory I can only as a root. Is that ok?
*******Attached my section below on chirpstack.toml, is that ok?

Update the following configuration sections in the chirpstack.toml configuration file:

[gateway] section: ********Done

client_cert_lifetime=“12months”
ca_cert=“/etc/chirpstack/certs/ca.pem”
ca_key=“/etc/chirpstack/certs/ca-key.pem”

[integration.mqtt.client] section: ********Done

client_cert_lifetime=“12months”
ca_cert=“/etc/chirpstack/certs/ca.pem” ****Done
ca_key=“/etc/chirpstack/certs/ca-key.pem” *****Done

Make sure to restart ChirpStack. ****Done

Also verify the logs for possible errors.****** No errors

Configure Mosquitto
Create a directory for the certificate files using the following command:

mkdir -p /etc/mosquitto/certs *******Done
cp ca.pem /etc/mosquitto/certs *******Done
cp mqtt-server.pem /etc/mosquitto/certs *******Done
cp mqtt-server-key.pem /etc/mosquitto/certs ******Done

Depending your setup, you might need to modify the ownership and / or permissions of the created directory and files. ****** Question as above, not sure

To restrict MQTT clients (gateway and integrations) to their own topics, create the following ACL file: ******Done. ******Question: is it ok the only name acl for this file with no extension?

/etc/mosquitto/acl:

pattern readwrite +/gateway/%u/#
pattern readwrite application/%u/#application-server

Note that the %u will be automatically replaced by the CN field of the client-certificate. The + prefix in +/gateway/%u/# is used for the region prefix.

The following configuration file will configure two listeners:

One listener on port 1883 (no TLS), which is accessible by any MQTT on the same machine (localhost only).
One listener on port 8883 (TLS), which is accessible on any network interface. Client must use a client-certificate in order to connect to this listener.

/etc/mosquitto/conf.d/listeners.conf: *******Done.I did only copy/paste the text below, is it ok?

per_listener_settings true

listener 1883 127.0.0.1
allow_anonymous true

listener 8883 0.0.0.0
cafile /etc/mosquitto/certs/ca.pem
certfile /etc/mosquitto/certs/mqtt-server.pem
keyfile /etc/mosquitto/certs/mqtt-server-key.pem
allow_anonymous false
require_certificate true
use_identity_as_username true
acl_file /etc/mosquitto/acl
Make sure to restart ChirpStack. Also verify the logs for possible errors.

Verify localhost (no TLS)
From the same machine on which Mosquitto is running, you should be able to connect without providing any credentials, using the following command:

mosquitto_sub -h localhost -t “#” -v -d
*******Done, I see data updated, even when sensors send update

********Now below is where I have errors:

Any host (TLS)
From any other machine, you should be able to connect to the MQTT broker using the following command:

mosquitto_sub -h example.com -p 8883 --cafile ca.pem --cert cert.pem --key key.pem -t “#” -v -d

***The command above I undestand that I need to obtain application MQTT integration certificate, and when I do this I get in the top right the error attached

Please note:

The ca.pem, cert.pem and key.pem must be obtained from the ChirpStack web-interface (gateway certificate or application MQTT integration certificate).

Verify that your firewall rules allow incoming connections to the MQTT broker. ******Done

In case you see TLS related errors, please verify that the hostname (of the -h flag) matches the MQTT server-certificate. *******Question: how to do that?

Validation of the server-certificate can be disabled using the --insecure flag.

*******Final question, maybe silly.
Can I make MQTT integration work without generating certificates, only by subscribe to a topic with user and passwors access only?

Thank you for your help
Paolo

1 Like

If you work it out let me know, i’ve been trying for roughly 3 years on and off to make this work lol.

Hi,

my 2 cents…

cool, i might of gotten a bit further with this now… i dont think i tried file permission 664 but all around it from recommendations everywhere else… i can now get the certs via docker, now to test how i connect :smiley:

does this look right rick?
do you know what the pattern is for the devices?
pattern readwrite application/%u/# is that application/example.com/#? as it says the CN name in the docs.

at least i’m closer with a connection refused, though i got no idea why lol

ok need to sort out whats going on here…

openssl s_client -showcerts -connect example.com:8883
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 310 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Need more infos on the residence of the certs, could you paste the result of

  1. ls /etc/mosquitto/certs -l

  2. ls /etc/chirpstack/certs -l

i got past that bit… looks like its throwing an error on self signed even with --insecure flag. or maybe its because of this No client certificate CA names sent.

❯ openssl s_client -showcerts -connect example.com:8883
CONNECTED(00000005)
depth=0 CN = example.com
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = example.com
verify return:1
---
Certificate chain
 0 s:CN = example.com
   i:CN = example.com
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA512
   v:NotBefore: Sep 16 01:39:00 2023 GMT; NotAfter: Sep 15 01:39:00 2024 GMT
-----BEGIN CERTIFICATE-----
... truncated
-----END CERTIFICATE-----
 1 s:CN = example.com
   i:CN = example.com
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA512
   v:NotBefore: Sep 16 01:38:00 2023 GMT; NotAfter: Sep 14 01:38:00 2028 GMT
-----BEGIN CERTIFICATE-----
... truncated
-----END CERTIFICATE-----
---
Server certificate
subject=CN = example.com
issuer=CN = example.com
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA512:RSA+SHA512:ECDSA+SHA512:RSA-PSS+SHA384:RSA+SHA384:ECDSA+SHA384:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA256
Shared Requested Signature Algorithms: RSA-PSS+SHA512:RSA+SHA512:ECDSA+SHA512:RSA-PSS+SHA384:RSA+SHA384:ECDSA+SHA384:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA256
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3530 bytes and written 430 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---
8020A4FC01000000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:ssl/record/rec_layer_s3.c:1586:SSL alert number 116

Assuming that your certificates are under “/etc/mosquitto/certs” you should try:

openssl s_client -showcerts -connect example.com:8883 -CAfile /etc/mosquitto/certs/ca.pem -cert /etc/mosquitto/certs/mqtt-server.pem

I can see the connection being killed in the docker logs, i think its got to do with the CA name i used which was subdomain.example.com maybe it needs to be named after the docker host name mosquitto.

… hum help me to help you, have some difficulties to follow…

what was the command executed to get this Client unknown into this screenshot ?

I follow this from the website, Mosquitto TLS configuration - ChirpStack open-source LoRaWAN® Network Server documentation
Then generate the certificates and link them into the appropriate containers locations for docker.

go to mqtt integration and click generate certificate, create the 3 files names as said by the integrations suggestions and save to directory.

ca.crt
cert.crt
cert.key

mosquitto_sub -h example.com -p 8883 --cafile ca.crt --cert cert.crt --key cert.key -t “#” -v -d
or
mosquitto_sub -h example.com -p 8883 --cafile ca.crt --cert cert.crt --key cert.key -t “#” -v -d --insecure

when i try to connect i get that error. I’m seeing if i can dig into this a bit further at the moment. i think its most likely got to do with an invalid hostname or i’ve linked the wrong cert somewhere so its incorrect maybe…

generated them all from scratch and grabbed the new certs and still that error lol. too hard basket for now. I’m almost sure its something docker related thats incorrectly set by me causing the error.

Ok back to square one… please paste the output of these commands:

  1. on your chirpstack server

a) ls /etc/chirpstack/certs -l

  1. on your mosquito server

a) ls /etc/mosquitto/certs -l
b) cat /etc/mosquitto/conf.d/listeners.conf

chirpstack
a)

cory@dred:~/chirpstack-docker/configuration/certs$ docker exec -it chirpstack-docker-chirpstack-1 /bin/sh
~ $ ls /etc/chirpstack/certs -l
total 8
-rw-rw-r--    1 root     root          3243 Sep 18 05:16 ca-key.pem
-rw-rw-r--    1 root     root          1793 Sep 18 05:16 ca.pem

Mosquitto
a)

cory@dred:~/chirpstack-docker/configuration/certs$ docker exec -it mosquitto /bin/sh
/ # ls /etc/mosquitto/certs -l
total 12
-rw-rw-r--    1 root     root          1793 Sep 18 05:16 ca.pem
-rw-rw-r--    1 root     root          3243 Sep 18 05:23 mqtt-server-key.pem
-rw-rw-r--    1 root     root          1948 Sep 18 05:23 mqtt-server.pem

b) this is what was in listeners.conf, but with this file it wouldnt even register a connection without the contents being put into mosquitto.conf (so i may have a problem here as well).

listeners.conf

per_listener_settings true

listener 1883
allow_anonymous true

listener 8883 0.0.0.0
cafile /etc/mosquitto/certs/ca.pem
certfile /etc/mosquitto/certs/mqtt-server.pem
keyfile /etc/mosquitto/certs/mqtt-server-key.pem
allow_anonymous false
require_certificate true
use_identity_as_username true
acl_file /etc/mosquitto/acl

mosquitto.conf

listener 1883
allow_anonymous true

for the connection to register hearing something even trying to connect from 8883 i needed to put the settings from listeners.conf into mosquitto.conf so even though the file was in there it doesnt seem to be passing it for any broker settings.

what do your listeners.conf and mosquitto.conf files look like?

Hi,

After some testing…

Let me just say that I’m not a fan of docker, and all my stacks are running under VMWARE. Concerning MOSQUITTO, I migrated to EMQX for the web interface, much more easier to manage as far as I’m concern.

Anyway, I build vm linux machine and installed docker with the latest version 4 of chirpstack on it.

Here are the mods that I applied to the DOCKER/files to complete the setup, that you will find into this pdf.

I completed some preliminary test and it looks ok for now. I was able to connect to MQTT without username and password.

Take a look of this and give me some feedback after your test.

Regards.

Thanks for taking the time to be so thorough. Everything was/is being done by your example to the letter. Except I need to clarify what you mean by:

“MANDATORY: the hostname that you used to generate the certificates, a DNS record must be created in your environment to complete this procedure”

I dont think i understand what you mean by DNS record must be created in your environment to complete this procedure can you explain what you mean by this? this maybe where it is failing internally.

It depends of your infrastructure, usually you can define it into your router where the DNS server is running.

I’m using Pihole, so I had to create a record like domain example.com ip 192.168.2.100.

Hope it’s clear.

I run this in a vps cloud server with a cname record pointing to it with multiple subdomains behind a nginx reverse proxy - so it sounds like it shouldn’t be that but who knows it seems like the only point of difference assuming you got it to work with docker yourself.

The only other things i can think of is to try run the stack on host network rather then its own docker network.

Conclusion, looks like a network issue.

Hi ccall48

I think those final tests confirm the DNS record requirement… as far as I’m concern.

TEST1 - with DNS record mqtt2498.example.com = 192.168.2.100

mosquitto_sub -h mqtt2498.example.com -p 8883 --cafile ca.pem --cert client.pem --key client-key.pem  -t "#" -v -d

2023-09-21 21:26:09 1695345969: New connection from 172.18.0.1:43338 on port 8883.
2023-09-21 21:26:09 1695345969: New client connected from 172.18.0.1:43338 as auto-71D7C9AE-82C1-56F8-1B3E-AE0EB6388236 (p2, c1, k60, u'client.example.com').
2023-09-21 21:26:23 1695345983: Client auto-71D7C9AE-82C1-56F8-1B3E-AE0EB6388236 disconnected.

TEST2 - deleted DNS record mqtt2498.example.com = 192.168.2.100

mosquitto_sub -h mqtt2498.example.com -p 8883 --cafile ca.pem --cert client.pem --key client-key.pem  -t "#" -v -d
Unable to connect (lookup error.)

TEST3 - with IP of the MQTT server without DNS record

mosquitto_sub -h 192.168.2.100 -p 8883 --cafile ca.pem --cert client.pem --key client-key.pem  -t "#" -v -d

2023-09-21 21:27:28 1695346048: New connection from 172.18.0.1:40476 on port 8883.
2023-09-21 21:27:28 1695346048: OpenSSL Error[0]: error:140393F2:SSL routines:ACCEPT_SR_CERT_VRFY:sslv3 alert unexpected message
2023-09-21 21:27:28 1695346048: Client <unknown> disconnected: Protocol error.

TEST4 - recreated DNS record mqtt2498.example.com = 192.168.2.100

mosquitto_sub -h mqtt2498.example.com -p 8883 --cafile ca.pem --cert client.pem --key client-key.pem  -t "#" -v -d

2023-09-21 21:28:11 1695346091: New connection from 172.18.0.1:51388 on port 8883.
2023-09-21 21:28:11 1695346091: New client connected from 172.18.0.1:51388 as auto-BBB37C37-CB08-C29F-35A5-EA89DD58E1EE (p2, c1, k60, u'client.example.com').
2023-09-21 21:28:16 1695346096: Client auto-BBB37C37-CB08-C29F-35A5-EA89DD58E1EE disconnected.

Regards.

Just letting you know i ended up playing with this and got it working a few days ago.

I took a slightly different approach, but not unconventional I don’t think as you can add/use many brokers if needed - i ended up adding a second broker just to handle the mqtt integration on 8883 via tls.

main thing was that certs and config files had to be owned by user/group 1883 with 644 permissions. with the exception being the acl file which accepted 644 but with a legacy warning to set it as 700 which i did as well.

thanks again for the time you put into helping me out :wink: