A firmware bug and less than ideal response pattern on my part appears to have put chirpstack into an unusual state, for a particular device, that I’d like to understand. Hoping someone here can help.
- Running chirpstack in docker, don’t know how to get the version number.
- Number of devices registered and happily sending data every 15 minutes, and a special report every 24 hours.
- One of the devices entered a strange state and it started sending only the 24 hour report.
- Tried to queue a HardwareRest command, via our Python controller, to this broken device. Intending to get the next 24 hour report but got the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = “enqueue downlink payload error: create device-queue item error: rpc error: code = InvalidArgument desc = device security-context out of sync”
debug_error_string = “{“created”:”@1645724595.350538822",“description”:“Error received from peer ipv4:10.202.90.51:8080”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1069,“grpc_message”:“enqueue downlink payload error: create device-queue item error: rpc error: code = InvalidArgument desc = device security-context out of sync”,“grpc_status”:13}"
It turned out the 24hr reports had also stopped a day or so before I tried to queue the reset request. A second bug, sigh.
It appears that the Chirpstack Network log entries corresponding to this event are:
time=“2022-02-24T18:46:43.228290164Z” level=info msg=“finished unary call with code OK” ctx_id=10e50711-6ae8-44a1-83dd-8f7711f16db9 grpc.code=OK grpc.method=GetNextDownlinkFCntForDevEUI grpc.service=ns.NetworkServerService grpc.start_time=“2022-02-24T18:46:43Z” grpc.time_ms=1.23 peer.address=“172.18.0.7:37556” span.kind=server system=grpc
time=“2022-02-24T18:46:43.233474801Z” level=info msg=“finished unary call with code InvalidArgument” ctx_id=066f4de9-6f32-4399-8cee-6cadbdf226e1 error=“rpc error: code = InvalidArgument desc = device security-context out of sync” grpc.code=InvalidArgument grpc.method=CreateDeviceQueueItem grpc.service=ns.NetworkServerService grpc.start_time=“2022-02-24T18:46:43Z” grpc.time_ms=1.077 peer.address=“172.18.0.7:37556” span.kind=server system=grpc
Chirpstack will happily queue/send the reset command to any of the other devices.
Is there any way to discover that Chirpstack won’t allow commands to be queued or is this exception the only way?
Is this because the ‘session’ with the device has lapsed because it was silent for to long? If so is the only way to correct the problem for the device to rejoin?
Haven’t yet been able to recover the device, load the fixed firmware and get it to rejoin the network.