LNS hiccup: stopped talking to devices; restart gateway-bridge appeared to clear it

We have a Chirpstack server (v3.17.4 running on Linux x64) with 18 gateways connected (most of them local to the devices), and 59 devices in an application. Some of the devices periodically do a Confirmed UL, and after missing these for 24h, they start a re-join.

Today we noticed that many of those devices were in re-join, though they’d been reporting via the LNS correctly until about 4 hours earlier (we have an alert when we don’t hear from a mote in 4 hours). Turns out, the inbound ULs were being processed corrected, but apparently no DLs were being done for the confirmed ULs.

Once re-join starts, we see network-server logs of no such object, even though the ChirpStack web UI clearly shows these devices and a look at postgres also shows them.

In the process of trying to understand things, I changed log_level on the components from info to debug and restarted the components. After restarting the gateway-bridge, this apparently cleared the issue.

I don’t know how to reproduce this (yet) but wanted to report the hiccup in case anyone else sees it.