Chirpstack V4 Docker - Handle uplink Error=Object does not exist

ChirpStack (v4)
#1

Hello!

I am running V4 - Docker on opensuse 15.4. Everything runs smoothly except one (important) thing. I can’t figure out why Chirpstack drops uplinks from devices occasionally. The logged error looks as follows:

chirpstack-gateway-bridge-eu868_1  | time="2022-11-28T06:37:28.365593599Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/b827ebfffe0a8083/event/up uplink_id=43897
chirpstack_1                       | 2022-11-28T06:37:28.421438Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_name="eu868" topic="eu868/gateway/b827ebfffe0a8083/event/up" qos=0 json=false
chirpstack_1                       | 2022-11-28T06:37:28.625379Z  INFO up{deduplication_id=86fbf797-5cc8-4ffa-9dcc-89d4fb12742a}: chirpstack::uplink: Uplink received m_type="UnconfirmedDataUp"
chirpstack_1                       | 2022-11-28T06:37:28.630961Z  INFO up{deduplication_id=86fbf797-5cc8-4ffa-9dcc-89d4fb12742a}:data_up: chirpstack::storage::device_session: Device-session saved dev_eui=4ab1df35680f3a3f dev_addr=260bd5b0
chirpstack_1                       | 2022-11-28T06:37:28.632389Z ERROR up{deduplication_id=86fbf797-5cc8-4ffa-9dcc-89d4fb12742a}: chirpstack::uplink::data: Handle uplink error error=Object does not exist (id: 4ab1df35680f3a3f)

I have the same error on both RAK WIsGate Developer gateway and several 1-channel test gateways (EU868 and EU433). The uplink is visible in gateway logs on Chirpstack and on gateways, but error is reported in Chirpstack log.

#2

I think a little more information is needed. To clarify, 4ab1df35680f3a3f is a device you have registered and not an unknown device? When the uplinks are dropped, does the problem correct itself or is a rejoin required (either manually or via logic on the device)? If a rejoin is required, could you have Redis configured for non-persistent storage and may be inadvertently dumping the activation data on restart?

#3

Thank you for response. I am not sure where this 4ab1df35680f3a3f comes from, since the registered device’s EUI is different (631e2a1f4ef39970 according to what is visible using web interface), and I use only one device. The problem corrects itself without rejoining. The uplinks are dropped irregularely, every couple of messages. When I join the device to the TTN, no such problems occur, so I suppose the device doesn’t cause it. And I observe the same behavior also with other devices, on EU433 plan (I have single Chirpstack instance, two gateway bridges, single MQTT broker).

I use Docker install configured as-is, the only change I have made was with additional gateway bridge, but the problem was present even before that.

#4

With regard to 4ab1df35680f3a3f, it sounds like the network server is dropping traffic from an unrecognized device. This is working as intended.

Do you have any logs that show the issue with the device you control?

#5

I am not aware of any logs that could help to detect the reason of dropping uplink messages. And the issue is not related to the specific device. I am also not sure that the eui of the dropped message comes from some device which is not joined to the network, because the device address in gateway frames is always correct. For example:

2022-11-28 21:53:41 DevAddr: 260bd5b0
2022-11-28 21:51:41 DevAddr: 260bd5b0
2022-11-28 21:49:40	DevAddr: 260bd5b0
2022-11-28 21:47:39	DevAddr: 260bd5b0
2022-11-28 21:45:39	DevAddr: 260bd5b0
2022-11-28 21:43:38 DevAddr: 260bd5b0
2022-11-28 21:41:38	DevAddr: 260bd5b0

Here the device sends uplink message each two minutes. But some of these messages, which were received by gateway are dropped, and don’t appear in the event list for the device:

2022-11-28 21:53:42		DR: 5 Data: 3edc172b000000000000 FPort: 1
2022-11-28 21:51:42		DR: 5 Data: c5db172b000000000c00 FPort: 1
2022-11-28 21:43:39		DR: 5 Data: e3d9172b000000001a00 FPort: 1
2022-11-28 21:41:39		DR: 5 Data: 6ad9172b000000000e00 FPort: 1

I have noticed that for some reason wrong dev_eui appears (4ab1df35680f3a3f instead of 631e2a1f4ef39970) in dropped uplinks, although the dev_addr is correct (260bd5b0). Could it be because I have created and deleted the same device several times, and that the old record hasn’t been purged from the database?

#6

I think that I successfully resolved the problem. While experimenting since installation of Chirpstack, I have created and deleted devices several times. Each time, the address of certain device was the same, but the dev_eui was different. It seems that the deleted dev_eui remained in the database, and sometimes (I don’t understand why), when the uplink message from certain device arrives, Chirpstack retrieves the deleted dev_eui, based on its dev_addr. And this happens from time to time, not always (in most cases Chirpstack finds the correct dev_eui).

Anyhow, when I changed addresses of all devices to those not used previously, errors disappeared.

#7

So you had overlapping dev_addr between different EUIs? That doesn’t seem right.

#8

I installed Chirpstack for the first time and played with it, to understand the interface and procedures of creating applications, adding devices, gateways etc. I added a device (with address dev_addr), and this device got a random dev_eui. Then after some experimentation, I deleted this device. Then I added again the same device (the same dev_addr), and it got another dev_eui. I never had two devices (two different dev_eui) with the same dev_addr concurrently.

Obviously, deleting device doesn’t mean that its dev_eui would be fully removed from the database. Is it a bug or a feature? Or I didn’t follow the complete procedure of deletion of device?

#9

Actually, ChirpStack does handle this fine :slight_smile: In Redis the DevAddr → DevEUI is a key holding an array of DevEUIs. ChirpStack will iterate over the device-sessions and for each device-session validating the MIC. If the MIC is valid, it then knows the DevEUI for that specific uplink.