Hello, I have a problem with one of my ChirpStack. I have 2 completely separate (even separated redis and mqtt) ChirpStack envs - test and staging.
Both have the following configuration:
App-server v3.17.4
Network-server v3.15.5
Gateway-bridge v3.13.2
Redis v5.0.14
Mosquitto v2
Running on the single machine in docker v20.10.10 + docker-compose, uses common bridge network.
Both are using the RU864 frequency plan and have additional frequencies in the NS configuration (866+).
A few days ago, data of income frames from gateway stopped decipher on staging.
In NS logs I can see theese messages for problematic devices:
level=info msg="gateway/mqtt: uplink frame received" gateway_id=98f07bfffea7ba83 uplink_id=6e2a5552-2bae-4bd3-bc49-833cf159726f
level=info msg="uplink: frame(s) collected" ctx_id=78645052-46fc-44e2-9de3-4a07b832d814 mtype=UnconfirmedDataUp uplink_ids="[6e2a5552-2bae-4bd3-bc49-833cf159726f]"
level=warning msg="creating insecure application-server client" server="192.168.40.128:8703"
level=error msg="uplink: processing uplink frame error" ctx_id=78645052-46fc-44e2-9de3-4a07b832d814 error="get application-server client error: create application-server api client error: dial application-server api error: context deadline exceeded"
Not much informative, so I tried to use debug logging-mode:
level=warning msg="creating insecure application-server client" server="192.168.40.128:8703"
level=debug msg="[core]parsed scheme: \"\""
level=debug msg="[core]scheme \"\" not registered, fallback to default scheme"
level=debug msg="[core]ccResolverWrapper: sending update to cc: {[{192.168.40.128:8703 <nil> 0 <nil>}] <nil> <nil>}"
level=debug msg="[balancer]base.baseBalancer: got new ClientConn state: {{[{192.168.40.128:8703 <nil> 0 <nil>}] <nil> <nil>} <nil>}"
level=debug msg="[core]Subchannel Connectivity change to CONNECTING"
level=debug msg="[core]Subchannel picks a new address \"192.168.40.128:8703\" to connect"
level=debug msg="[balancer]base.baseBalancer: handle SubConn state change: 0xc0002c8f50, CONNECTING"
level=debug msg="[core]Channel Connectivity change to CONNECTING"
level=debug msg="[core]Subchannel Connectivity change to TRANSIENT_FAILURE"
level=debug msg="[balancer]base.baseBalancer: handle SubConn state change: 0xc0002c8f50, TRANSIENT_FAILURE"
level=debug msg="[core]Channel Connectivity change to TRANSIENT_FAILURE"
level=debug msg="[transport]transport: loopyWriter.run returning. connection error: desc = \"transport is closing\""
At first I check the connectivity between AS and NS using telnet - works fine.
But then I noticed - at same moment I have these errors on the application-server side:
level=error msg="api/js: unmarshal base payload error" error="unexpected end of JSON input"
level=error msg="backend/joinserver: error handling request" error="unexpected end of JSON input"
So if I have a problem with input JSON for appserver, it could be related to income data and some bugs on LoraWAN-station and/or devices. I tried to connect Lora station from fine-working test-env to broken staging-env. But I got same result and same errors with Lora-station from test-env.
I also saw earlier on this forum that similar problems can be caused by a database misconsistency(?).
I Deleted the databases and re-created all the entities needed to work (first using postman via the API, then just manually). The problem has unchanged.
How can I debug this?
What additional info can I provide you to help?