Hi there,
Since days ago Im started to see a ton of this errors
2024-10-31T15:24:57.708365Z ERROR chirpstack::downlink::tx_ack: Handling tx ack error error=Object does not exist (id: 1135066439)
During that moment the server starts to decrease its performance and my monitoring system reports a drop in the outgoing server data.
I have around 300 gateways and 8000 devices.
The LoRa server is running within container in a linux VM with 32 cores and 64 MB of ram.
Im using the Chirpstack v4 repo, with all the default configuration that the repo provides. Chirpstack and its dependencies are running on docker containers
I have collected the entire sequence of logs
chirpstack-1 | 2024-10-29T10:08:57.882068Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Downlink response flow
chirpstack-1 | 2024-10-29T10:08:57.882088Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Selecting downlink gateway
chirpstack-1 | 2024-10-29T10:08:57.882136Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting tx-info for RX1
chirpstack-1 | 2024-10-29T10:08:57.882161Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting tx-info for RX2
chirpstack-1 | 2024-10-29T10:08:57.882177Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Getting next device queue-item
chirpstack-1 | 2024-10-29T10:08:57.887528Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Requesting custom channel re-configuration
chirpstack-1 | 2024-10-29T10:08:57.887541Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Requesting channel-mask reconfiguration
chirpstack-1 | 2024-10-29T10:08:57.887916Z INFO up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::storage::mac_command: Pending mac-command block set dev_eui=70b3d5a4d31047af cid=LinkADRReq
chirpstack-1 | 2024-10-29T10:08:57.887968Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Requesting ADR change
chirpstack-1 | 2024-10-29T10:08:57.888036Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Requesting device-status
chirpstack-1 | 2024-10-29T10:08:57.888044Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Requesting rejoin param setup
chirpstack-1 | 2024-10-29T10:08:57.888066Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting ping-slot parameters
chirpstack-1 | 2024-10-29T10:08:57.888072Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting rx parameters
chirpstack-1 | 2024-10-29T10:08:57.888078Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting tx parameters
chirpstack-1 | 2024-10-29T10:08:57.888105Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Setting downlink PHYPayloads
chirpstack-1 | 2024-10-29T10:08:57.888518Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Updating device queue-item
chirpstack-1 | 2024-10-29T10:08:57.888530Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Saving downlink frame
chirpstack-1 | 2024-10-29T10:08:57.889126Z INFO up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::storage::downlink_frame: Downlink-frame saved downlink_id=2393941469
chirpstack-1 | 2024-10-29T10:08:57.889148Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Updating device
chirpstack-1 | 2024-10-29T10:08:57.901289Z INFO up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::storage::device: Device partially updated dev_eui=70b3d5a4d31047af
chirpstack-1 | 2024-10-29T10:08:57.901317Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::downlink::data: Sending downlink frame
chirpstack-1 | 2024-10-29T10:08:57.901379Z INFO up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::gateway::backend::mqtt: Sending downlink frame region_id=us915_1 gateway_id=c0ee40ffff2a7790 topic=us915_1/gateway/c0ee40ffff2a7790/command/down json=false
chirpstack-1 | 2024-10-29T10:08:57.901410Z TRACE up{deduplication_id=fe41eacb-5559-4768-9648-37bd78ee8df9}:data_up{dev_eui="70b3d5a4d31047af"}:data_down{downlink_id=2393941469}: chirpstack::gateway::backend::mqtt: Message published
chirpstack-gateway-bridge-1 | 2024-10-29T10:08:57.904818841Z level=info msg="integration/mqtt: downlink frame received" downlink_id=2393941469 gateway_id=c0ee40ffff2a7790
chirpstack-gateway-bridge-1 | 2024-10-29T10:08:57.914282669Z level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=us915_1/gateway/c0ee40ffff2a7790/event/ack
chirpstack-1 | 2024-10-29T10:08:57.983484Z TRACE tx_ack{downlink_id=2393941469}: chirpstack::downlink::tx_ack: Get downlink-frame from Redis
chirpstack-1 | 2024-10-29T10:08:57.984001Z TRACE tx_ack{downlink_id=2393941469}: chirpstack::downlink::tx_ack: Decoding PhyPayload
chirpstack-1 | 2024-10-29T10:08:57.984015Z TRACE tx_ack{downlink_id=2393941469}: chirpstack::downlink::tx_ack: Getting device data
chirpstack-1 | 2024-10-29T10:08:57.992377Z TRACE tx_ack{downlink_id=2393941469}: chirpstack::downlink::tx_ack: Logging tx ack error
I was trying to increase the max connection for redis, postgress but nothing fix the issue.
Any thoughts? Is there any arch limit for chirpstack in term of number of devices?
My devices are sending uplinks every 5-10 secs