All gateways offline

ChirpStack (v4)
1

My Chirpstack instance decided to auto upgrade yesterday and now all my gateways are offline.
This is most likely due to version 4.6 change of the chirpstack gateway bridge, however when i add the “v4_migrate=true” to the config file it still doesn’t work.

I’m uncertain if i added it right in the config file.

 # MQTT configuration.
      [regions.gateway.backend.mqtt]
      v4_migrate=true
        # Topic prefix.
        #
        # The topic prefix can be used to define the region of the gateway.
        # Note, there is no need to add a trailing '/' to the prefix. The trailing
        # '/' is automatically added to the prefix if it is configured.
        topic_prefix="eu868"
2

Are you certain you are on 4.6 and not 4.7? I would assume an auto-update would bring you to the newest version, being 4.7.

If you are on 4.7 you must use the command:

chirpstack -c /etc/chirpstack migrate-device-sessions-to-postgres

To migrate the device session keys from redis to postgres.

I’m not familiar with using the v4_migrate though so unfortunately I can’t help if that is the issue, although it looks like it should be placed in your servers regional .toml file and not the main file. Is that where you put it?

3

By auto update i meant that some sysadmin did an “apt upgrade”.

I’m on 4.7 but i was on 4.4 before the auto update.

All my gateways use chirpstack gateway bridge 3.14.x and all gateways are offline so this must be the issue?
I’ve added the line in the regional section.

4

Just to confirm by “regional section” you mean the region_eu868.toml file?

5

Yes indeed.
However, now i noticed that all gateways check in to the mqtt server but all of them are offline in the chirpstack webinterface.

6

I believe you must receive your first device uplink for the gateway to appear online, are your device session keys still there after updating?

7

Where can i see those?

8

In the web UI, on any devices page click activation

9

I get This device has not (yet) been activated.

10

Ah, so as of 4.7 the device’s session keys got migrated to the postgres database, where they used to be kept in the redis database.

After the update, your keys are still there in redis but you need to migrate the keys to the postgres database where Chirpstack is now expecting them.

Use the command:

chirpstack -c /etc/chirpstack migrate-device-sessions-to-postgres

If you did the ubuntu install that should just work, but if you’re using docker you will have the enter the chirpstack container first

11

I have already done that command and it seem to be successfull in the cli.

12

Huh strange, if they are in postgres you should see them UI. Did you check that all the keys are missing (or atleast enough to confirm this is the issue). You could also do some manual PSQL to check the database yourself and see if the keys have been moved. Listing the tables would be enough as a new table should be created for the session keys after the command is executed.

13

Perhaps the chirpstack logs after executing the commands could show something

14

Hmm, the devices show that they’re activated now. I think i just clicked on one that wasn’t before.

15

Im running out of ideas. just reading through the migration documentation, Is the gateway bridges mashalar set to protobuf?

16

Yes it is set to protobuff.

17

I get this in the logs:

chirpstack[3216]: #033[2m2024-04-23T00:02:10.049647Z#033[0m #033[31mERROR#033[0m #033[2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033[0m MQTT error #033[3merror#033[0m#033[2m=#033[0mI/O: connection closed by peer

18

Last shot from me, did you configure the MQTT topic prefixes in the gateway bridge? I also don’t understand why any of this would have been a problem if you simply upgraded from 4.4 to 4.7, and in the original post you say you tried adding v4_migrate=true but didn’t you need the v4_migrate=true even for 4.4 if you are using 3.10.x gateway bridges?

19

No, that change went live with 4.6 wich led me to my first conclusion.
However i found this topic:

20

Interesting, whats your mosquitto version?