I have been debugging some timing issues that are occurring when queuing back-to-back downlink packets on two class C devices. I have placed the devices on two separate (but identical configs) gateways in order to further demonstrate this potential limitation. I am looking for some clarity on why this is occurring (including if we are violating some rules):
Setup
- Ubuntu 18.04
- chirpstack-network-server/stable,now 3.9.0 amd64 [installed]
- chirpstack-application-server/stable,now 3.10.0 amd64 [installed]
- Semtech packet forwarder & chirpstack-gateway-bridge running on MTConduit
- Class C devices
- 2 gateways, with 1 device each
- Downlink to /tx topic for both devices - 2nd device downlink publish is roughly 200ms after first
Problem
Network server handling of back-to-back downlink packets (even to separate gateways) has issues with internal state/context. Both packets seem to be successfully sent to the gateways, and then received at the end-device (although I have not verified their unique payloads). The issue arises when the event/ack message comes back from the respective gateway. The network server seems to lose uniqueness of state information about each queued item.
You can see from the below logfile, both downlink packets are created (as mentioned above, roughly 200ms apart). The first payload is queued to gateway A, then the 2nd queued packet is handled, and is queued to gateway B. Notice how the ctx_id is identical for the 2 separate packets – this seems suspicious. Both packets are queued at the gateway, and their respective bridge publishes the event/ack message to let the server know they have been received/queued at the gateway. However, as you can see - the network server is treating them as the same txack.
“sent tx ack to application-server” occurs twice, as expected, but the content of the txack is identical. It published BOTH txack messages to the topic of the first EUI (in fact, when sniffing via mqtt.fx, the tx ack packets are identical - I posted the mqtt payload log as well below)
Log
21:56:15 msg="device-queue item created" ctx_id=5ab5b44c-439d-44c1-acfc-57bb9b3392c3 dev_eui=c6f2fc8b5d47ddb8 f_cnt=16
21:56:16 msg="device-queue item created" ctx_id=43a35209-52e9-4f68-a008-e69e897b6964 dev_eui=2e57abe2eae086bb f_cnt=18
21:56:16 msg="device-queue deleted" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d id=32048
21:56:16 msg="pending mac-command block set" cid=LinkADRReq commands=2 ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d dev_eui=2e57abe2eae086bb
21:56:16 msg="gateway/mqtt: publishing gateway command" command=down downlink_id=90b57afb-661a-4fe1-99ae-271df54ea20d gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
21:56:16 msg="device-session saved" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d dev_addr=012f5c2d dev_eui=2e57abe2eae086bb
21:56:16 msg="downlink-frames saved" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d token=37045
21:56:16 msg="device-queue deleted" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d id=32047
21:56:16 msg="pending mac-command block set" cid=LinkADRReq commands=2 ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d dev_eui=c6f2fc8b5d47ddb8
21:56:16 msg="gateway/mqtt: publishing gateway command" command=down downlink_id=90b57afb-661a-4fe1-99ae-271df54ea20d gateway_id=00800000a000387a qos=1 topic=gateway/00800000a000387a/command/down
21:56:16 msg="device-session saved" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d dev_addr=0198700f dev_eui=c6f2fc8b5d47ddb8
21:56:16 msg="downlink-frames saved" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d token=37045
21:56:16 msg="backend/gateway: downlink tx acknowledgement received" downlink_id=90b57afb-661a-4fe1-99ae-271df54ea20d gateway_id=00800000a000387a
21:56:16 msg="sent downlink meta-data to network-controller" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d
21:56:16 msg="sent tx ack to application-server" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d
21:56:16 msg="backend/gateway: downlink tx acknowledgement received" downlink_id=90b57afb-661a-4fe1-99ae-271df54ea20d gateway_id=00800000a0005a49
21:56:16 msg="sent downlink meta-data to network-controller" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d
21:56:16 msg="sent tx ack to application-server" ctx_id=90b57afb-661a-4fe1-99ae-271df54ea20d
