Bridge no longer connects to mqtt server

It appears my bridge is no longer connecting to the MQTT server, and I don’t understand why. Everything is running in a container and has been fine for ages. As of 2024-01-22T13:33:22, it appears things stopped working. I am not positive if I did an update at that time, as it would be about 8:30am in the morning, and that would be somewhat unusual, although I honestly can’t remember that far back :slight_smile:

As part of the effort to resolve this issue, I did do a docker-compose pull to update everything, and it made no difference. Here’s the relevant section from the the docker logs loraserver-docker_chirpstack-gateway-bridge_1 output.

time="2024-01-22T13:33:29.889372382Z" level=info msg="integration/mqtt: publishing event" downlink_id=53bf7c1d-3eff-481d-ada2-0fa24d3dfaf8 event=ack qos=0 topic=gateway/e45f01fffe17c7bc/event/ack
time="2024-01-22T13:33:39.826852443Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=dcd51d2e-a019-4555-a40f-e4b24dfad1db topic=gateway/e45f01fffe17c7bc/event/stats
time="2024-01-22T13:34:09.83452081Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=be705226-71cc-479a-bab9-5c8eac6a03c3 topic=gateway/e45f01fffe17c7bc/event/stats
time="2024-01-22T13:34:22.178786317Z" level=error msg="[client]   Connect comms goroutine - error triggered EOF" module=mqtt
time="2024-01-22T13:34:22.180133378Z" level=info msg="signal received" signal=terminated
time="2024-01-22T13:34:22.231048364Z" level=warning msg="shutting down server"
time="2024-01-22T13:34:22.23168051Z" level=error msg="mqtt: connection error" error=EOF
time="2024-01-22T13:34:22.596003786Z" level=info msg="integration/mqtt: publishing state" gateway_id=e45f01fffe17c7bc qos=0 state=conn topic=gateway/e45f01fffe17c7bc/state/conn
time="2024-01-22T13:34:22.755929697Z" level=error msg="[client]   dial tcp 192.168.32.7:11883: connect: connection refused" module=mqtt
time="2024-01-22T13:34:22.789131597Z" level=warning msg="[client]   failed to connect to broker, trying next" module=mqtt
time="2024-01-22T13:35:57.761993149Z" level=info msg="starting ChirpStack Gateway Bridge" docs="https://www.chirpstack.io/gateway-bridge/" version=
time="2024-01-22T13:35:57.762085311Z" level=info msg="backend/semtechudp: starting gateway udp listener" addr="0.0.0.0:11700"
time="2024-01-22T13:36:01.554256507Z" level=error msg="[client]   dial tcp 192.168.32.8:11883: connect: no route to host" module=mqtt
time="2024-01-22T13:36:01.554316892Z" level=warning msg="[client]   failed to connect to broker, trying next" module=mqtt
time="2024-01-22T13:36:01.554349225Z" level=error msg="[client]   Failed to connect to a broker" module=mqtt
time="2024-01-22T13:36:01.554391564Z" level=error msg="integration/mqtt: connection error" error="network Error : dial tcp 192.168.32.8:11883: connect: no route to host"
time="2024-01-22T13:36:03.555722517Z" level=warning msg="[client]   status is already disconnected" module=mqtt
time="2024-01-22T13:36:04.626330424Z" level=error msg="[client]   dial tcp 192.168.32.8:11883: connect: no route to host" module=mqtt
time="2024-01-22T13:36:04.626398743Z" level=warning msg="[client]   failed to connect to broker, trying next" module=mqtt
time="2024-01-22T13:36:04.626437497Z" level=error msg="[client]   Failed to connect to a broker" module=mqtt
time="2024-01-22T13:36:04.626494216Z" level=error msg="integration/mqtt: connection error" error="network Error : dial tcp 192.168.32.8:11883: connect: no route to host"

When I do a docker-compose ps, I see the following:

                      Name                                     Command               State                                             Ports
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
loraserver-docker_chirpstack-application-server_1   /usr/bin/chirpstack-applic ...   Up      0.0.0.0:18001->18001/tcp,:::18001->18001/tcp, 0.0.0.0:18080->18080/tcp,:::18080->18080/tcp
loraserver-docker_chirpstack-gateway-bridge_1       /usr/bin/chirpstack-gatewa ...   Up      0.0.0.0:11700->11700/udp,:::11700->11700/udp
loraserver-docker_chirpstack-geolocation-server_1   ./chirpstack-geolocation-s ...   Up      0.0.0.0:18005->18005/tcp,:::18005->18005/tcp
loraserver-docker_chirpstack-network-server_1       /usr/bin/chirpstack-networ ...   Up      0.0.0.0:18003->18003/tcp,:::18003->18003/tcp
loraserver-docker_mosquitto_1                       /docker-entrypoint.sh /usr ...   Up      0.0.0.0:11883->11883/tcp,:::11883->11883/tcp, 1883/tcp
loraserver-docker_postgresql_1                      docker-entrypoint.sh postgres    Up      5432/tcp
loraserver-docker_redis_1                           docker-entrypoint.sh redis ...   Up      6379/tcp

The mosquitto logs seems to think a client is connecting.

1706026256: mosquitto version 2.0.18 starting
1706026256: Config loaded from /mosquitto/config/mosquitto.conf.
1706026256: Opening ipv4 listen socket on port 11883.
1706026256: Opening ipv6 listen socket on port 11883.
1706026256: mosquitto version 2.0.18 running
1706026257: New connection from 192.168.32.2:56064 on port 11883.
1706026257: New client connected from 192.168.32.2:56064 as auto-3B9FDABB-D0FE-A0C0-E802-616DD1D7610D (p2, c1, k30).
1706026258: New connection from 192.168.32.5:49720 on port 11883.
1706026258: New client connected from 192.168.32.5:49720 as auto-A2D8AED2-1EAA-24A1-7990-0A8939BD69E0 (p2, c1, k30).
1706026264: New connection from 172.16.1.25:56662 on port 11883.
1706026264: New client connected from 172.16.1.25:56662 as nodered_5d624415f6ed64a6 (p2, c1, k60).

Why would this stop working? One thing I noticed that when the bridge is starting up that the version is blank. Where else can I check what version of Chirpstack is running? It doesn’t appear to be anywhere in the web interface.

time="2024-01-22T13:35:57.761993149Z" level=info msg="starting ChirpStack Gateway Bridge" docs="https://www.chirpstack.io/gateway-bridge/" version=
time="2024-01-22T13:35:57.762085311Z" level=info msg="backend/semtechudp: starting gateway udp listener" addr="0.0.0.0:11700"

Any help would be greatly appreciated.

Well, this is interesting. In the process of trying to resolve this, I ran docker system prune, which removes all unused containers, networks, images (both dangling and unused). After restarting Chirpstack, everything now works again.

As the prune command removes network interfaces and one of the error messages was no route to host, I’m wondering if somehow an interface became corrupted. If that’s the case, the corrupted interface survived across the server being restarted.

I have a shell script that starts Chirpstack, and perhaps I’ll add pruning all the Chirpstack volumes as a good measure.