Devices don't seem to join succesfully

ChirpStack Network Server
1

Hi.

We have a network of sensors in a remote building. The gateway forwards everything to our chirpstack-network-server.

Three sensors haven’t been able to send any data for months, although they used to work. Other sensors of same model are transmitting correctly.

In “device data” we only see dev-nonce errors. We assumed we had dev-nonce issues. After playing with the database then using the new “Clear Devnonce” feature (thanks for that!) the dev-nonce issue still appears from times to times but I’m beginning to think the issue lies elsewhere and the dev-nonce errors are due to the device sending so many join requests (they seem to be random so they may happen to match) but are not an issue by themselves.

In “lorawan frames”, we see join requests with varying freq/SF but I can’t find anything saying if the join requests was accepted or not.

In “activation” tab, everything is greyed out. Device address, network session key and application session key contain stuff. UL/DL frame counters are 0.

In database, I see many rows in device_activation table.

select * from device_activation where dev_eui=E'\\x24e1641094896540';

Does a row here mean join was successful?

The only explanation I can think of is that the device is authorized to join but never receives the response so it sends join requests again and again. We have range issues indeed, but I’d find it strange that we get so many join requests and the device doesn’t get any response.

Here’s the kind of things I can see in the logs:

ul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.040625236Z" level=info msg="gateway/mqtt: uplink frame received" gateway_id=24e124fffef0a6d4 uplink_id=5544e82b-2b12-40ec-a3d8-8f41c003ec96
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.243670604Z" level=info msg="uplink: frame(s) collected" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb mtype=JoinRequest uplink_ids="[5544e82b-2b12-40ec-a3d8-8f41c003ec96]"
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.29751888Z" level=info msg="lorawan/backend: finished backend api call" message_type=JoinReq protocol_version=1.0 receiver_id=24e124c0002a0001 result_code=Success sender_id=000000 transaction_id=1037645265
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.297677252Z" level=info msg="sent uplink meta-data to network-controller" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb dev_eui=24e1641094896540
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.298701879Z" level=info msg="device-queue flushed" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb dev_eui=24e1641094896540
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.29946541Z" level=info msg="device-session saved" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb dev_addr=0197fe9c dev_eui=24e1641094896540
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.300742709Z" level=info msg="device-activation created" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb dev_eui=24e1641094896540 id=47925
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.301604735Z" level=info msg="device updated" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb dev_eui=24e1641094896540
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.301845424Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=001c511f-0230-4fc5-b5ba-2131eebaedfb gateway_id=24e124fffef0a6d4 qos=0 topic=gateway/24e124fffef0a6d4/command/down
Jul 18 07:31:22 delorean chirpstack-network-server[30338]: time="2022-07-18T07:31:22.302319032Z" level=info msg="storage: downlink-frame saved" ctx_id=001c511f-0230-4fc5-b5ba-2131eebaedfb token=54909

Jul 18 07:31:22 delorean chirpstack-application-server[30431]: time="2022-07-18T07:31:22.248709365Z" level=info msg="backend/joinserver: request received" message_type=JoinReq receiver_id=24e124c0002a0001 sender_id=000000 transaction_id=1037645265
Jul 18 07:31:22 delorean chirpstack-application-server[30431]: time="2022-07-18T07:31:22.296755559Z" level=info msg="device-keys updated" ctx_id="<nil>" dev_eui=24e1641094896540
Jul 18 07:31:22 delorean chirpstack-application-server[30431]: time="2022-07-18T07:31:22.296901525Z" level=info msg="backend/joinserver: sending response" dev_eui=24e1641094896540 message_type=JoinAns receiver_id=000000 result_code=Success sender_id=24e124c0002a0001 transaction_id=1037645265

Does this provide any useful information?

The “result_code=Success” sounds like the join succeeds. Is this correct?

Thanks for any hint.

I setup this sensor network a while ago (> 12 months) and I haven’t been doing much LoRa since then. I just peek into the web interface from times to times. It is totally possible that I’m missing a basic concept.

I do regular software updates. We’re using latest chirpstack versions.

2

The device nonce cannot be reused. It is a 16-bit value, so there can be 65536 possible values. It is unlikely to get exhausted, unless your device somehow keeps trying to join so many times. If this is unexpected, then perhaps there is something wrong (might be related to the conditions for rejoining - like perhaps it needs a response from the LNS and cannot receive downlinks well).

If it often reuses the same nonce values, then perhaps there is something wrong with its LoRaWAN implementation.

If the join request was accepted, there will be a join-accept sent by the LNS.

3

Thanks. Yes, it looks like the joins succeed and the devices don’t get the join-accept.

This is what I would like to confirm. I also find it strange that the device manages to reach the gateway but the gateway never reaches the device when sending the response. I would expect the range issue to be more symmetric.

4

Does the device often have difficulties receiving downlinks? If so, could it be that you need to relocate the device?

I was once told it could be because devices have lower RF sensitivity than gateways. It might be possible that the device cannot receive downlinks, even if its uplinks can be received.

5

The device not receiving downlink data is the most likely explanation I have.

Is there a way to confirm or infirm this from the logs?

Relocating the devices in not really an option. We could try to move the gateways but even this requires a round-trip to that remote building. Before doing so, I’d rather be sure the problem is correctly diagnosed.

6

Perhaps spend time observing whether it can receive downlinks? If you have multiple of this same type of device installed and only this handful of them have a problem, surely it cannot be an issue with the LNS.

Chirpstack has a function to periodically send DevStatusReq, which will make signal strength information available… although I don’t know how useful that information really is because it reports the signal strength above the demodulation floor (rather than RSSI). But if the device never seems to respond to DevStatusReq despite supporting this MAC command, it might meant that it has difficulties receiving downlinks.