To log messages from all applications and gateways

What is the suggested way to to log and view all messages from devices that are registered to different applications?

I’m looking for similar functionality that Actility’s Wireless Logger module in Thingpark gives, ie. to see latest (and history of) message reveiced by all the devices.

It seems that web ui can visualise messages, but only per gateway. But what if we want to see all lora uplinks/downlinks regardless of the gateway?

You could achieve this by subscribing to the application/# MQTT topic. This would mean subscribe to all MQTT topics starting with application/, thus it would match all applications and devices.

1 Like

Tried this, but seems that nothing shows. Am I missing something? In web UI, I see application’s device events coming in.

Another thing, are the tx/rxinfo published to the topic, because what I actually need to persist, is SNR, RSSI, SF and deveui information?

What I would suggest is enabling the postgresql integration, this way Chipstack will forward all device events to a sql database that will store them indefinitely, with all the data and metadata. All thats required is to set up another PSQL database (or just another table with the same database/user in your postgres container) and then add the following to your [integration] section of your Chirpstack.toml.

[integration]
  enabled=["mqtt", "postgresql"]

  [integration.mqtt]
    server="tcp://$MQTT_BROKER_HOST:1883/"
    json=true

  [integration.postgresql]
    dsn="postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>/<DATABASE>?sslmode=disable"

As a side note it seems strange that you are not seeing the MQTT messages in your broker, have you already changed the [integration.mqtt] section above? Or perhaps you have an atypical TLS setup (ACL and mosquitto.conf) on your MQTT broker such that the mqtt integration isn’t posting the events to your MQTT broker. As your Chirpstack setup is still working you should at the very least be able to see the raw still encrypted messages coming from the gateway bridge if you subscribe to the topic “#” rather than “application/#”.

When I subscribe inside mosquitto container (or from outside, using 1883) I see events published as expected :

From outside using 8883 port, nothing, except when I sub to the topic #, I see something, though, but not application uplinks:

I have configured stack like this:

chirpstack.toml:
[integration]
enabled=[“mqtt”]

[integration.mqtt]
server=“tcp://$MQTT_BROKER_HOST:1883/”
json=true

acl:
pattern readwrite +/gateway/%u/#
pattern readwrite application/%u/#

mosquitto.conf:
per_listener_settings true

listener 1883 0.0.0.0
allow_anonymous true

listener 8883 0.0.0.0
max_connections -1
cafile /mosquitto/certs/ca.pem
certfile /mosquitto/certs/mqtt-server.pem
keyfile /mosquitto/certs/mqtt-server-key.pem
allow_anonymous false
require_certificate true
use_identity_as_username true
acl_file /mosquitto/acl

This is all expected behaviour, you must just be using the gateway certificates to connect instead of the applications certificates.

When you set up the ACL using:

pattern readwrite +/gateway/%u/#
pattern readwrite application/%u/#

and in the mosquitto.conf use:

use_identity_as_username true

The %u in the ACL gets replaced by the MQTT client’s (in this scenario thats you using mosquitto_sub) username, restricting a client to only view the topics with their username in them.

use_identity_as_username means it uses the CN on the certificates as your username. Presumably you are using the gateway’s certificates to connect with mosquitto_sub, thus it is only showing you the events following the pattern eu868/gateway/%u/# which are messages directly to/from the MQTT forwarder, rather than application/%u/#, which contains the application events after they have been processed and decrypted by chirpstack.

What you should do is go into the UI, under the application you want to monitor and retrieve the certificate from the “mqtt integration” button there, that will have the proper CN to view that applications MQTT events.

The reason you can view all events from inside the container or using 1883 is because of the other listener set up:

listener 1883 0.0.0.0
allow_anonymous true

Which allows any other connection on the same machine to connect without TLS and without the ACL restrictions. If you tried to connect to the mosquitto broker over port 1883 from another machine it would fail.

Thanks a lot Liam_Philipp for clarifying this!

But isn’t it so that the lines

listener 1883 0.0.0.0
allow_anonymous true

will allow connections from any interface in the internet?

If I change the setting to as per the instructions:

listener 1883 127.0.0.1

and try to start the stack, I get lots of access denied errors, as other services cant access mosquitto.

Do I need to setup a network in the docker-compose file, and how can I create a mosquitto listerner on port 1883 that allows chirpstack service to connect?

Sorry you are right, as per the instructions you should use:

listener 1883 127.0.0.1
allow_anonymous true

Which allows connections from the same device to be unencrypted. If you also have the allow_anonymous below the listener it should let the other containers connect without issue. If not please share your full mosquitto.conf.

This is the mosquitto.conf

Are all of your components on the same machine? If so I don’t see how that configuration difference could cause those issues. It would be a good idea to set all of your services onto the same docker network although if you haven’t defined a network yet they should all default to the same one.

Could you share the errors you are receiving?

Here’s the startup log showing some connection refused errors:

azure@machine:~/workspace/chirpstack-docker$ sudo docker-compose up
Starting chirpstack-docker_postgres_1 … done
Starting chirpstack-docker_mosquitto_1 … done
Starting chirpstack-docker_redis_1 … done
Starting chirpstack-docker_chirpstack-gateway-bridge-basicstation_1 … done
Starting chirpstack-docker_chirpstack-gateway-bridge_1 … done
Starting chirpstack-docker_chirpstack_1 … done
Starting chirpstack-docker_chirpstack-rest-api_1 … done
Attaching to chirpstack-docker_postgres_1, chirpstack-docker_mosquitto_1, chirpstack-docker_redis_1, chirpstack-docker_chirpstack-gateway-bridge_1, chirpstack-docker_chirpstack-gateway-bridge-basicstation_1, chirpstack-docker_chirpstack_1, chirpstack-docker_chirpstack-rest-api_1
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.811243642Z” level=info msg=“starting ChirpStack Gateway Bridge” docs=“Introduction - ChirpStack open-source LoRaWAN<sup>®</sup> Network Server” version=
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.495188476Z” level=info msg=“starting ChirpStack Gateway Bridge” docs=“Introduction - ChirpStack open-source LoRaWAN<sup>®</sup> Network Server” version=
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.495258477Z” level=info msg=“backend/semtechudp: starting gateway udp listener” addr=“0.0.0.0:1700”
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497620818Z” level=error msg=“[client] dial tcp 172.27.0.3:1883: connect: connection refused” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497638318Z” level=warning msg=“[client] failed to connect to broker, trying next” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497647118Z” level=error msg=“[client] Failed to connect to a broker” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497662019Z” level=error msg=“integration/mqtt: connection error” error=“network Error : dial tcp 172.27.0.3:1883: connect: connection refused”
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.813350678Z” level=error msg=“[client] dial tcp 172.27.0.3:1883: connect: connection refused” module=mqtt
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.813395879Z” level=warning msg=“[client] failed to connect to broker, trying next” module=mqtt
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.813405979Z” level=error msg=“[client] Failed to connect to a broker” module=mqtt
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.813420979Z” level=error msg=“integration/mqtt: connection error” error=“network Error : dial tcp 172.27.0.3:1883: connect: connection refused”
chirpstack-rest-api_1 | Starting ChirpStack REST API server
chirpstack_1 | 2024-06-04T09:03:00.998703Z INFO chirpstack::cmd::root: Starting ChirpStack LoRaWAN Network Server version=“4.7.0” docs=“https://www.chirpstack.io/
chirpstack_1 | 2024-06-04T09:03:00.998758Z INFO chirpstack::storage: Setting up PostgreSQL connection pool
mosquitto_1 | 1717491776: mosquitto version 2.0.18 starting
mosquitto_1 | 1717491776: Config loaded from /mosquitto/config/mosquitto.conf.
postgres_1 |
postgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1 |
mosquitto_1 | 1717491776: Warning: File /mosquitto/acl has world readable permissions. Future versions will refuse to load this file.
mosquitto_1 | To fix this, use chmod 0700 /mosquitto/acl.
mosquitto_1 | 1717491776: Opening ipv4 listen socket on port 1883.
mosquitto_1 | 1717491776: Opening ipv4 listen socket on port 8883.
mosquitto_1 | 1717491776: mosquitto version 2.0.18 running
redis_1 | 1:C 04 Jun 2024 09:02:57.924 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see vm.max_map_count growing steadily when vm.overcommit_memory is 2 · Issue #1328 · jemalloc/jemalloc · GitHub. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
redis_1 | 1:C 04 Jun 2024 09:02:57.924 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 04 Jun 2024 09:02:57.924 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 04 Jun 2024 09:02:57.924 * Configuration loaded
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * monotonic clock: POSIX clock_gettime
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * Running mode=standalone, port=6379.
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * Server initialized
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * Loading RDB produced by version 7.2.4
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * RDB age 30 seconds
redis_1 | 1:M 04 Jun 2024 09:02:57.925 * RDB memory usage when created 1.49 Mb
redis_1 | 1:M 04 Jun 2024 09:02:57.927 * Done loading RDB, keys loaded: 1045, keys expired: 0.
redis_1 | 1:M 04 Jun 2024 09:02:57.927 * DB loaded from disk: 0.002 seconds
redis_1 | 1:M 04 Jun 2024 09:02:57.927 * Ready to accept connections tcp
mosquitto_1 | 1717491777: New connection from x.x.x.x:40064 on port 8883.
chirpstack_1 | 2024-06-04T09:03:00.998925Z INFO chirpstack::storage: Applying schema migrations
chirpstack_1 | 2024-06-04T09:03:01.044351Z INFO chirpstack::storage: Setting up Redis client
chirpstack_1 | 2024-06-04T09:03:01.044607Z INFO chirpstack::region: Setting up regions
mosquitto_1 | 1717491778: New client connected from xxx.xxx.xxx.xxx:xxxxx as 7076ff0056080d16 (p5, c0, k30, u’7076ff0056080d16’).
chirpstack_1 | 2024-06-04T09:03:01.044663Z INFO setup{common_name=EU868 region_id=eu868}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044757Z INFO setup{common_name=KR920 region_id=kr920}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044783Z INFO setup{common_name=US915 region_id=us915_1}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044832Z INFO setup{common_name=AS923_4 region_id=as923_4}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044859Z INFO setup{common_name=IN865 region_id=in865}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044903Z INFO setup{common_name=EU433 region_id=eu433}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044925Z INFO setup{common_name=AS923_3 region_id=as923_3}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.044953Z INFO setup{common_name=AU915 region_id=au915_0}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045018Z INFO setup{common_name=US915 region_id=us915_0}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045104Z INFO setup{common_name=RU864 region_id=ru864}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045162Z INFO setup{common_name=AS923_2 region_id=as923_2}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045204Z INFO setup{common_name=CN470 region_id=cn470_10}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045262Z INFO setup{common_name=AS923 region_id=as923}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045290Z INFO setup{common_name=CN779 region_id=cn779}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045315Z INFO setup{common_name=ISM2400 region_id=ism2400}: chirpstack::region: Configuring region
chirpstack_1 | 2024-06-04T09:03:01.045387Z INFO chirpstack::backend::joinserver: Setting up Join Server clients
chirpstack_1 | 2024-06-04T09:03:01.045393Z INFO chirpstack::backend::roaming: Setting up roaming clients
chirpstack_1 | 2024-06-04T09:03:01.045397Z INFO chirpstack::adr: Setting up adr algorithms
chirpstack_1 | 2024-06-04T09:03:01.045414Z INFO chirpstack::integration: Setting up global integrations
chirpstack_1 | 2024-06-04T09:03:01.045418Z INFO chirpstack::integration::redis: Initializing Redis integration
chirpstack_1 | 2024-06-04T09:03:01.045422Z INFO chirpstack::integration::mqtt: Initializing MQTT integration
chirpstack_1 | 2024-06-04T09:03:01.056321Z INFO chirpstack::integration::mqtt: Connecting to MQTT broker server_uri=tcp://mosquitto:1883/ client_id=a8f55a67331bceb0 clean_session=false
chirpstack_1 | 2024-06-04T09:03:01.056416Z INFO chirpstack::integration::mqtt: Starting MQTT event loop
postgres_1 | 2024-06-04 09:02:56.373 UTC [1] LOG: starting PostgreSQL 14.11 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
postgres_1 | 2024-06-04 09:02:56.373 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
postgres_1 | 2024-06-04 09:02:56.373 UTC [1] LOG: listening on IPv6 address “::”, port 5432
postgres_1 | 2024-06-04 09:02:56.718 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
postgres_1 | 2024-06-04 09:02:57.157 UTC [23] LOG: database system was shut down at 2024-06-04 09:02:27 UTC
postgres_1 | 2024-06-04 09:02:57.404 UTC [1] LOG: database system is ready to accept connections
chirpstack_1 | 2024-06-04T09:03:01.056611Z INFO chirpstack::gateway::backend: Setting up gateway backends for the different regions
chirpstack_1 | 2024-06-04T09:03:01.056619Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=eu868 region_common_name=EU868
chirpstack_1 | 2024-06-04T09:03:01.056958Z ERROR chirpstack::integration::mqtt: MQTT error error=I/O: Connection refused (os error 111)
chirpstack_1 | 2024-06-04T09:03:01.064460Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=eu868 server_uri=tcp://mosquitto:1883 clean_session=false client_id=d5cef21868016f36
chirpstack_1 | 2024-06-04T09:03:01.064772Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
chirpstack_1 | 2024-06-04T09:03:01.065043Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=kr920 region_common_name=KR920
chirpstack_1 | 2024-06-04T09:03:01.065309Z ERROR chirpstack::gateway::backend::mqtt: MQTT error error=I/O: Connection refused (os error 111)
chirpstack_1 | 2024-06-04T09:03:01.072113Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=kr920 server_uri=tcp://mosquitto:1883 clean_session=false client_id=2342a14f78599655
chirpstack_1 | 2024-06-04T09:03:01.072178Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=us915_1 region_common_name=US915
chirpstack_1 | 2024-06-04T09:03:01.072316Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
chirpstack_1 | 2024-06-04T09:03:01.072808Z ERROR chirpstack::gateway::backend::mqtt: MQTT error error=I/O: Connection refused (os error 111)
chirpstack_1 | 2024-06-04T09:03:01.078567Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=us915_1 server_uri=tcp://mosquitto:1883 clean_session=false client_id=cce8572fafe27687
chirpstack_1 | 2024-06-04T09:03:01.078629Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=as923_4 region_common_name=AS923_4
chirpstack_1 | 2024-06-04T09:03:01.081470Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
chirpstack_1 | 2024-06-04T09:03:01.086140Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=as923_4 server_uri=tcp://mosquitto:1883 clean_session=false client_id=8474bf5e33f8af8
chirpstack_1 | 2024-06-04T09:03:01.086213Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
chirpstack_1 | 2024-06-04T09:03:01.086214Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=in865 region_common_name=IN865
chirpstack_1 | 2024-06-04T09:03:01.091862Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=in865 server_uri=tcp://mosquitto:1883 clean_session=false client_id=195fd88821b544aa
chirpstack_1 | 2024-06-04T09:03:01.096611Z ERROR chirpstack::gateway::backend::mqtt: MQTT error error=I/O: Connection refused (os error 111)
chirpstack_1 | 2024-06-04T09:03:01.096712Z ERROR chirpstack::gateway::backend::mqtt: MQTT error error=I/O: Connection refused (os error 111)

Hey @vmt,

This is strange behaviour and I’m definitely getting out of my comfort zone but I have an idea of what the issue might be.

The lines:

msg=“[client] dial tcp 172.27.0.3:1883: connect: connection refused” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497638318Z” level=warning msg=“[client] failed to connect to broker, trying next” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497647118Z” level=error msg=“[client] Failed to connect to a broker” module=mqtt
chirpstack-gateway-bridge_1 | time=“2024-06-04T09:03:00.497662019Z” level=error msg=“integration/mqtt: connection error” error=“network Error : dial tcp 172.27.0.3:1883: connect: connection refused”
chirpstack-gateway-bridge-basicstation_1 | time=“2024-06-04T09:03:00.813350678Z” level=error msg=“[client] dial tcp 172.27.0.3:1883: connect: connection refused” module=mqtt

Suggest that your MQTT broker is not located at 172.0.0.x but 172.27.0.x

I bet all of your components are going on that same 172.27.0.x subnet, and hence the connections from 172.0.0.x are failing.

Try changing the mosquitto listener instead to:

listener 1883 127.27.0.1
allow_anonymous true

and see if that fixes it.

Did you make any changes to your docker setup such that the containers would be getting those IPs?

PS. the “preformatted text” option is helpful for code and log files

Using the setting below did not help.

listener 1883 127.27.0.1
allow_anonymous true
time="2024-06-05T05:38:30.404718415Z" level=error msg="integration/mqtt: connection error" error="network Error : dial tcp 172.27.0.4:1883: connect: connection refused"

Only modifications are: I exposed the port 8883 so that the mqtt forwarder in Kerlink can connect to chirpstack

mosquitto:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - 1883:1883
      - 8883:8883
    volumes:
      - ./configuration/mosquitto/config/:/mosquitto/config/
      - ./configuration/mosquitto/acl:/mosquitto/acl
      - ./configuration/mosquitto/certs:/mosquitto/certs

This seems to work, although I’m not sure if this is recommended: in mosquitto.conf, binding both listeners to default wildcard like:

listener 1883 0.0.0.0
allow_anonymous true

listener 8883 0.0.0.0
max_connections -1

and exposing only port 1883

mosquitto:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - 8883:8883
    volumes:

allows stack containers to connect anonymously, but from outside 1883 is not open.

I believe that should be fine. As long as when you try to mosquitto_sub to the broker from outside the machine on 1883 it fails. I have a similar setup, where I just use:

listener 1883
configs...

listener 8883
configs...

And then using a Traefik reverse-proxy I only route external traffic to the 8883 port. For most purposes it should be fine to leave the listeners as wildcards and then only expose the 8883 port to WAN.

Thanks @Liam_Philipp for the advice, this is now sorted!

But the original question about getting logs from all devices under different apps:

You could achieve this by subscribing to the application/# MQTT topic. This would mean subscribe to all MQTT topics starting with application/ , thus it would match all applications and devices.

So I made an an mqtt integration under one application

Then I subscibe with app’s generated certificates to the topic application/#, but I only see device log data from devices under that specific application. I suppose that is the way it is intended to work?

Yes the “mqtt integration” through the UI just gives you the certificates signed for that specific application. If you needed to subscribe to all the MQTT events regardless of application there are a few methods I know of:

  1. Use mosquitto_sub on the local machine through port 1883, since the listener is not bound by the ACL rules you can see all the events.

  2. If you need subscribe from a remote machine, a slightly more in depth but certainly possible option would be to generate a certificate that is signed by the MQTT broker’s CA with some admin type username and then in your ACL allow access for that user to all topics.

  3. Last but not least (infact I’d say the best option) is if you set up an external MQTT broker on the server you wish to monitor the events from, you can simply set the “server=” line in the [integration.mqtt] section of your chirpstack.toml to the addres of your remote MQTT broker, and chirpstack will forward all “integration” mqtt events to that broker, regardless of application. There is even the option to set up certificates for that as well if you require TLS.

1 Like

I suppose you could also just delete your ACL file and the reference to it in your mosquitto.conf, that way any certificate signed by your CA could view any applications devices. This would reduce security, as in the event one of your gateways was compromised, it could be used to view/post on any application on the server. A considerable risk if you have multiple tenants with their own gateways but trivial in some smaller applications, depends on you.

1 Like

Thanks, @Liam_Philipp, I’ll consider these, or the Postgres integration!

1 Like