Docker installation does not start because of "Error: Setup MQTT integration"

[dma75]

I am in the process of up updating an existing v3 installation to the v4. I am using a Docker installation. While installation and migration runs smooth on my test server, I encounter a rather strange problem on the production server.

I can reproduce the problem very simple by doing a "git clone https://github.com/chirpstack/chirpstack-docker.git" on both servers and bringing the containers up. (Of course, I stopped the v3 instances before). I made no changes to the configuration of to the cloned git repo.

1. on my test server (Debian, 11.7) the containers start up as expected, no problems.

2. on the production server (Debian, 10.13), the chirpstack container container exits and restarts over and over, because of an “Error: Setup MQTT integration”. Other containers, like the gateway bridge, successfully connect with the MQTT broker. Also, I can publish/subcribe messages on the exposed mosquitto. The error I see when bringing up the containers is:

chirpstack-docker-chirpstack-1 | 2023-09-21T13:37:18.316003Z INFO chirpstack::integration::mqtt: Connecting to MQTT broker server_uri=tcp://mosquitto:1883/ client_id=64dd73fb1d23dbb3 clean_session=false chirpstack-docker-chirpstack-1 | Error: Setup MQTT integration chirpstack-docker-chirpstack-1 | chirpstack-docker-chirpstack-1 | Caused by: chirpstack-docker-chirpstack-1 | 0: Connect to MQTT broker chirpstack-docker-chirpstack-1 | 1: [-1] TCP/TLS connect failure chirpstack-docker-chirpstack-1 exited with code 1

Can someone give me a hint on what the problem could be and/or how I can narrow it down?

[sp193]

It looks like a problem connecting to the MQTT broker. Have you checked that the credentials match, the MQTT broker address is correct and that the broker is actually running and can accept connections?

There may be relevant log messages from the MQTT broker.

[dma75]

Mosquitto is in default config, so no credentials needed. And other parts of the dockerized installation have no problems in connecting to Mosquitto. Even other services running on the machine connect to the exposed Mosquitto port and can subscribe/publish messages. So the issue is most likely coming from the chirpstack-image itself.
As already said, I pulled the docker-compose.yml from GitHub without making any changes. On one machine it works, on the other not.

[brocaar]

Are you sure that the Mosquitto service was started correctly? A potential issue could be an other Mosquitto instance which is running which conflicts with the port (1883) that Docker Compose tries to forward to the Docker host.

[dma75]

I stopped my ‘regular’ Mosquitto before. And I see that other services on my machine successfully connect into the container’s Mosquitto. I verified this by means of the on-screen log output (when running the container non-detached). I further used mosquitto_pub/sub outside the container to check if connects are shown by the container’s Mosquitto.
Also, docker compose would yield an error when bringing the container up if the the host port is already allocated. I intentionally tried this as well - it was also my first guess that there is something wrong with the container’s Mosquitto.

So yes, I am really sure that Mosquitto has started inside the container.