No Join Accept in GUI, but in application server output mqtt gateway/+/command/down

We started seeing this behavior with a subset of the devices on our platform (we have many other devices joining successfully and showing join accepts in the GUI). The application server shows join requests, but no join accepts. However, I have an application that consumes from the gateway/+/command/down mqtt topic, which if I understand correctly is the topic used to send packets down to the end devices and I DO see the join accept packets there.

image

Any suggestions on where to look for the problem? I have tried several things (noted below) and cannot pinpoint the issue.

  • Rebooting the gateways
  • Restarting the Chirpstack Gateway Bridge
  • Restarting the Chirpstack Application and Network servers
  • Pointing the gateways at an older Lora Gateway Bridge (pre-Chirpstack namechange)

I will share what we find. We were able to resolve the join accept issue by rolling back to version 3.2.1 of the gateway bridge.

Looking further - the only thing I’m noting so far is that the CRC status is set in the uplink here and that changed between these releases.

In these screen shots you can see the crcStatus is “NO_CRC” in the attempt that succeeded with version 3.2.1 and “CRC_OK” in the failed attempt on version 3.8.0.

Still digging to figure what the issue is here.

Please note that the downlink at the MQTT level means that the NS forwarded the downlink to the Gateway Bridge. You would have to check the Gateway Bridge logs to see if it actually forwarded the downlink to the packet-forwarder (and check the packet-forwarder logs to see if there are any errors returned by the HAL).

This might be helpful: https://www.chirpstack.io/guides/troubleshooting/

We were able to identify and recreate the issue by severing the downstream UDP connection from the gateway bridge to the gateway.

You were spot on that the packets were not ever making it to the PF.