Acknowledged packets failing on devices created after Chirpstack Upgrade

I am running Chirpstack V 3.14 on a hosted server (Debian).
The server started off with LoRaServer but this was removed and Chirpstack installed when it was released.
The server has been running reliably but we have recently hit a problem.
We use devices on AU915, in ABP mode and always run withADR off and Acknowledged Packets (in effort to get a continuous data set from our devices which are sending long packets every 15 minutes)

  • the Device Profile has been set to match the settings of teh LoRa WAN units: version 1.02 of the LoRa WAN stack and Regional Paramaters Version A
  • at the start of a transmission cycle, the devices send a Link Check packet. If this is acknowledged, they start sending the sensor readings.
    We set up a batch of devices on Chirpstack some time ago and they all ran fine with Acknowledged Packets. This would have been with an earlier version of the server (but I am not sure which version it was at the time)
    Recently, we have added some new devices but they are not working in Acknowledge mode.
    If we change the Node to Unacknowledged mode, they work fine.
    So the device address, EUI, NwkSKey and AppSKey are all OK, otherwise the Server . But for some reason the downlink packet is being missed.
    To make sure I had a proper handle on this, I took one of the new devices and loaded it with the config from one of the old devices (I downloaded the working config from the old device and loaded it to the new one):
  • the new device (B) works fine when I load it with the credentials for a devce which was created when Chirpstack was at the earlier version
  • the new device (B) does not work with a configuration created under Chirpstack Version 3.14
  • if I load device (A) with the new configuration, it too stops working
    My conclusion from this is that there is nothing wrong with the hardware.
    But configurations created after the Chirpstack upgrade will not work in Acknowledged mode.
    I suspect that something has changed with the settings for the downlink packets: either frequency, data rate or delay:
  • we can see the Link Check packet go out and a reply come back. But the Node is missing the reply.
    Maybe the devices created under the previous Chirpstack build may have an empty or null parameter for one of the settings; wherease new devices may have a value set which is causing the issue.
    So my question from this is, does anyone have any idea on where next to look? Answering this may entail some additional knowledge on how the Device configurations are stored and from that a line by line comparison to see if there is a difference between the old and new.
    Any help or thoughts would be appreciated.

Hi, what do you mean by a Link Check packet?
As they work in ABP mode, devices should not have Join procedure with the Server and they should start sending Uplink messages inmeadiately.

Thanks for the reply.
The product manufacturer has written the firmware so that when the device is ready, it first sends a Link Check (AT+MSG if used on the Rising HF modems for this purpose). It then waits for the acknowledgement - Link Check packets are always acknowledged by the server.
If the acknowledgement is received, it begins to send the data packets
If the Link Check is not aknowledged, the device goes back to sleep and waits for the next read cycle. At that time it tries again and, if the Link Check succeeds, sends the oldest packets then works its way to the most recent packets

  • if the device has been set to use Unacknowledged packets, it will then send each packet one or more times, according to the number set in the configuration (AT+MSG=“abcede…”
  • if teh device has been set to use Acknowledged packets, it will send the packet once and wait for an ACK (AT+CMSG=“abcde…”
    The problem as it currently stands is that the device is not seeing the Acknowledgement to the Link Check message.
    But this only applies to configurations which have been created with the more recent Chirpstack versions.

Hi @petertoome,

I understand now the procedure.
It’s right, it is used as a mechanism for not losing some uplink packets in case the server is not ready.

Did you have a look at Mqtt transmissions of both Link Check downlinks to compare them?

I am monitoring the MQTT traffic with MQTT-FX (the topic is : application/1/device/6e341d533b047796/#). I can see the uplink packets but no reply from Chirpstack (i.e no"downlink" or “error” packets)

application/1/device/6e341d533b047796/event/up
{“applicationID”:“1”,“applicationName”:“TOIP-Local-Nodes”,“deviceName”:“tbsl1trhbpsr”,“devEUI”:“6e341d533b047796”,“rxInfo”:[{“gatewayID”:“b827ebfffe140bf6”,“uplinkID”:“4bc117c5-3b71-4adc-b791-9a6fa98323a2”,“name”:“RHF2S001”,“rssi”:-41,“loRaSNR”:9,“location”:{“latitude”:-34.066489717707924,“longitude”:137.59002685546878,“altitude”:0}}],“txInfo”:{“frequency”:915400000,“dr”:5},“adr”:false,“fCnt”:0,“fPort”:0,“data”:null}

Same thing if I send a confirmed message (AT+CMSG=“Help” on the RIsing HF RHF076 modem:
{“applicationID”:“1”,“applicationName”:“TOIP-Local-Nodes”,“deviceName”:“tbsl1trhbpsr”,“devEUI”:“6e341d533b047796”,“rxInfo”:[{“gatewayID”:“b827ebfffe140bf6”,“uplinkID”:“d76594c1-49b7-4168-8ba5-8de79c827c08”,“name”:“RHF2S001”,“rssi”:-43,“loRaSNR”:9,“location”:{“latitude”:-34.066489717707924,“longitude”:137.59002685546878,“altitude”:0}}],“txInfo”:{“frequency”:915200000,“dr”:5},“adr”:false,“fCnt”:0,“fPort”:8,“data”:“SGVscA==”}

I have been doing some more logging, this time capturing the logs to a text file to make it easier to decipher.
My earlier comment about no seeing the downlink packets was incorrect: I can confirm that Chirpstack is (2) receiving the Link Check message and (2) sending an Acknowledgement.
But the Node is not seeing the Acknowledgement
To rule out a problem with the Node, I have quickly set the Node up with a configuration for TTN and confirmed that the LinK Check is always acknowledged on TTN (my Node responds with the Signal strength).
This means that the ACK is either:

  • being received outside of the RX1 and WX2 time windows
  • is on the wrong frequency
  • or is on a data rate that can not be decided
    I am currently working through the chirpstack-network-server configuration settings to see if changing them has any bearing

Hi @petertoome

Glad to know that you’re going ahead.

Do you have the Downlinks with Link Check Answers from Network Server?

Yes, I can see the downlinks being sent from the Network Server to the Gateway.
But the Nodes never see the transmission from the Gateway.

After a few days away from this issue, I have made some progress towards resolving the problem.
I managed to get the devices to run with Acknowledged packets by forcing them to use Version 1.03 of the LoRa WAN Mac and version B of the Regional Paramaters.
But I also had to go to the Rising HF modems and force them to use Version 1.0.3 MAC. The command for this is AT+LW=VER,V103
The modems also allow you to check the packet counter values: AT+LW=ULDL
Previously I would send a Link Check and see the Uplink counter go up but see no change on the Downlink; whereas on Chirpstack I could see the downlink packet going out. Now I can watch the uplink and downlink counters going up correctly.
I am not sure of the actual reason for the problem & its fix. But it is still possible that when I initially upgraded Chirpstack from an early build to the current build, that some of the parameters in the Gateway Bridge Configuration ended up being missed/lost/mis-configured. There could still be a bug in Chirpstack but I need to rule out the other options first.
But at least I am back to being able to add new Nodes again

After another break I have spent some more time on this.
The problems seem to only affect ABP mode. OTAA works fine.
And the same problem also happens on TTN V3 console.
I now suspect that there is something out of step between Rising HF and TTN/Chirpstack in relation to the RX1 & RX2 downlink windows. It could be DR, frequency or delay. But I need to do some more measurements to try and work out which.
Or, just ignore the problem and switch new devices to OTAA