I am running Chirpstack V 3.14 on a hosted server (Debian).
The server started off with LoRaServer but this was removed and Chirpstack installed when it was released.
The server has been running reliably but we have recently hit a problem.
We use devices on AU915, in ABP mode and always run withADR off and Acknowledged Packets (in effort to get a continuous data set from our devices which are sending long packets every 15 minutes)
the Device Profile has been set to match the settings of teh LoRa WAN units: version 1.02 of the LoRa WAN stack and Regional Paramaters Version A
at the start of a transmission cycle, the devices send a Link Check packet. If this is acknowledged, they start sending the sensor readings.
We set up a batch of devices on Chirpstack some time ago and they all ran fine with Acknowledged Packets. This would have been with an earlier version of the server (but I am not sure which version it was at the time)
Recently, we have added some new devices but they are not working in Acknowledge mode.
If we change the Node to Unacknowledged mode, they work fine.
So the device address, EUI, NwkSKey and AppSKey are all OK, otherwise the Server . But for some reason the downlink packet is being missed.
To make sure I had a proper handle on this, I took one of the new devices and loaded it with the config from one of the old devices (I downloaded the working config from the old device and loaded it to the new one):
the new device (B) works fine when I load it with the credentials for a devce which was created when Chirpstack was at the earlier version
the new device (B) does not work with a configuration created under Chirpstack Version 3.14
if I load device (A) with the new configuration, it too stops working
My conclusion from this is that there is nothing wrong with the hardware.
But configurations created after the Chirpstack upgrade will not work in Acknowledged mode.
I suspect that something has changed with the settings for the downlink packets: either frequency, data rate or delay:
we can see the Link Check packet go out and a reply come back. But the Node is missing the reply.
Maybe the devices created under the previous Chirpstack build may have an empty or null parameter for one of the settings; wherease new devices may have a value set which is causing the issue.
So my question from this is, does anyone have any idea on where next to look? Answering this may entail some additional knowledge on how the Device configurations are stored and from that a line by line comparison to see if there is a difference between the old and new.
Any help or thoughts would be appreciated.
Hi, what do you mean by a Link Check packet?
As they work in ABP mode, devices should not have Join procedure with the Server and they should start sending Uplink messages inmeadiately.
Thanks for the reply.
The product manufacturer has written the firmware so that when the device is ready, it first sends a Link Check (AT+MSG if used on the Rising HF modems for this purpose). It then waits for the acknowledgement - Link Check packets are always acknowledged by the server.
If the acknowledgement is received, it begins to send the data packets
If the Link Check is not aknowledged, the device goes back to sleep and waits for the next read cycle. At that time it tries again and, if the Link Check succeeds, sends the oldest packets then works its way to the most recent packets
if the device has been set to use Unacknowledged packets, it will then send each packet one or more times, according to the number set in the configuration (AT+MSG=“abcede…”
if teh device has been set to use Acknowledged packets, it will send the packet once and wait for an ACK (AT+CMSG=“abcde…”
The problem as it currently stands is that the device is not seeing the Acknowledgement to the Link Check message.
But this only applies to configurations which have been created with the more recent Chirpstack versions.
I am monitoring the MQTT traffic with MQTT-FX (the topic is : application/1/device/6e341d533b047796/#). I can see the uplink packets but no reply from Chirpstack (i.e no"downlink" or “error” packets)
Same thing if I send a confirmed message (AT+CMSG=“Help” on the RIsing HF RHF076 modem:
{“applicationID”:“1”,“applicationName”:“TOIP-Local-Nodes”,“deviceName”:“tbsl1trhbpsr”,“devEUI”:“6e341d533b047796”,“rxInfo”:[{“gatewayID”:“b827ebfffe140bf6”,“uplinkID”:“d76594c1-49b7-4168-8ba5-8de79c827c08”,“name”:“RHF2S001”,“rssi”:-43,“loRaSNR”:9,“location”:{“latitude”:-34.066489717707924,“longitude”:137.59002685546878,“altitude”:0}}],“txInfo”:{“frequency”:915200000,“dr”:5},“adr”:false,“fCnt”:0,“fPort”:8,“data”:“SGVscA==”}
I have been doing some more logging, this time capturing the logs to a text file to make it easier to decipher.
My earlier comment about no seeing the downlink packets was incorrect: I can confirm that Chirpstack is (2) receiving the Link Check message and (2) sending an Acknowledgement.
But the Node is not seeing the Acknowledgement
To rule out a problem with the Node, I have quickly set the Node up with a configuration for TTN and confirmed that the LinK Check is always acknowledged on TTN (my Node responds with the Signal strength).
This means that the ACK is either:
being received outside of the RX1 and WX2 time windows
is on the wrong frequency
or is on a data rate that can not be decided
I am currently working through the chirpstack-network-server configuration settings to see if changing them has any bearing
After a few days away from this issue, I have made some progress towards resolving the problem.
I managed to get the devices to run with Acknowledged packets by forcing them to use Version 1.03 of the LoRa WAN Mac and version B of the Regional Paramaters.
But I also had to go to the Rising HF modems and force them to use Version 1.0.3 MAC. The command for this is AT+LW=VER,V103
The modems also allow you to check the packet counter values: AT+LW=ULDL
Previously I would send a Link Check and see the Uplink counter go up but see no change on the Downlink; whereas on Chirpstack I could see the downlink packet going out. Now I can watch the uplink and downlink counters going up correctly.
I am not sure of the actual reason for the problem & its fix. But it is still possible that when I initially upgraded Chirpstack from an early build to the current build, that some of the parameters in the Gateway Bridge Configuration ended up being missed/lost/mis-configured. There could still be a bug in Chirpstack but I need to rule out the other options first.
But at least I am back to being able to add new Nodes again
After another break I have spent some more time on this.
The problems seem to only affect ABP mode. OTAA works fine.
And the same problem also happens on TTN V3 console.
I now suspect that there is something out of step between Rising HF and TTN/Chirpstack in relation to the RX1 & RX2 downlink windows. It could be DR, frequency or delay. But I need to do some more measurements to try and work out which.
Or, just ignore the problem and switch new devices to OTAA
The issue with ABP mode comes down to the RX1 & RX2 Window timing. Earlier versions all assumed RX1 delay =1 and RX2 delay = 2. But as is happening on TTN, users are now being encouraged to use RX1 dleay = 5 (and hence RX2=6).
When OTAA is used, the initial delay settings must be 1 & 2 sec. When the device Joins, the LNS then sends the keys along with the desired RX1 delay. Which is why ABP devices will fail but OTAA will work.
After sorting this out, we started to also see issues on OTAA with some devices working and others not. This turned out to be a mismatch in the LoRa WAN MAC version. Although the RHF076 modems we use look externally the same, internally they carry different firmware revisions with different MAC versions set as the default. The command AT+LW=VER shows the currently selected MAC version and this can be changed at the command line e.g AT+LW=VER,V103.
So pay careful attention to the LoRa WAN version (and Reginal Parameters version) used in your nodes so you don’t fall victim to this
And just for the sake of completenesss, it is worth adding one more thing to check if you can’t get ABP mode devices to work. For us, this came about when we could get devices running on TTN but not in Chirpstack.
This time around the parameter to check is the Data Rate Offset (DRO) which on Chirpstack is part of the ABP mode settings in the Device Profile (Uncheck “device uses OTAA” to switch the profile to ABP).
Working from the Regional Parameters doc and using AU915 as an example, the downlink DR is a product of the uplink DR and an Offset. We are using DR5 and 6 for our uplinks. This is to allow us to capture very long packets from multi-parameter sensors. We also run with ADR off. TTN uses a downlink DR of 13 (SF7/BW500k). Looking at the DR offset table (table 47) for DR5 this requires a data rate offset of 0. Our default device profile had an offset of 5, so the downlink packets were being sent at DR8 (SF12/BW500k). Modifying the Device Profile and then re-activating the device put the downlink DR back to the nominal settings, allowing the ACK packets to be properly received and processed by the device.
In addition, make sure that you change the default Data Rate Offset parameter in the chirpstack-network-server.toml configuration file.
RX1 data-rate offset
rx1_dr_offset=0
If you leave this on another value (e…g 5) Chirpstack will over-write the value you set in the Node configuration: this will be evident as the SF in the downlink packets changing from the expected value (e…g SF7) to the value obtained with the default offset (e.g 12).
I suspect that this is a bug, as the Node configuration should take precedence over the default settings in the config file.