Environment
- AWS EC2 ubuntu running:
- NS 3.13.0
- AS 3.15.0
- Localhost mosquitto broker
- Gateway-bridge 3.11.0-r1 running on Conduit GW
- 4 dummy Class-C devices (powered down for test, so that NO uplinks can mess with this test). All 4 last uplinks via single gateway, so all 4 downlinks are sent via one gateway
Class-C settings, if applicable
scheduler_interval="1s"
gateway_downlink_lock_duration="0s"
Issue Summary
When multiple downlink items are queued back to back, when the class-C scheduler runs it sends all 4 command downs to the Gateway, but it looks like the downlink_id is incorrectly setup. I queued 4 items back-to-back via application-server mqtt endpoint for 4 different devices. Creation of device-queue item is where I started the logfile (filtered for easier reading of queue-item and downlink publish. Let me know if you want the full unfiltered log)
time="2021-06-22T19:44:23.279227325Z" level=info msg="device-queue item created" ctx_id=dbcf2ee0-8f4d-4438-bf72-ab1d96faf527 dev_eui=aaaaaa0001000033 f_cnt=55
time="2021-06-22T19:44:23.317445163Z" level=info msg="device-queue item created" ctx_id=45f92973-e53b-4182-926c-e233292f3686 dev_eui=aaaaaa0001000034 f_cnt=40
time="2021-06-22T19:44:23.321068071Z" level=info msg="device-queue item created" ctx_id=3c7e08db-aa09-41a0-81a9-adf1d2c473c0 dev_eui=aaaaaa0001000035 f_cnt=49
time="2021-06-22T19:44:23.330589121Z" level=info msg="device-queue item created" ctx_id=f7022664-6d92-4b47-ac95-9872973cd187 dev_eui=aaaaaa0001000036 f_cnt=57
time="2021-06-22T19:44:24.237144371Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T19:44:24.239012358Z" level=info msg="device-queue item updated" ctx_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c dev_eui=aaaaaa0001000033 emit_at_time_since_gps_epoch="<nil>" f_cnt=55 is_pending=false timeout_after="<nil>"
time="2021-06-22T19:44:24.240368752Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T19:44:24.241711181Z" level=info msg="device-queue item updated" ctx_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c dev_eui=aaaaaa0001000034 emit_at_time_since_gps_epoch="<nil>" f_cnt=40 is_pending=false timeout_after="<nil>"
time="2021-06-22T19:44:24.243063519Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T19:44:24.244424000Z" level=info msg="device-queue item updated" ctx_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c dev_eui=aaaaaa0001000035 emit_at_time_since_gps_epoch="<nil>" f_cnt=49 is_pending=false timeout_after="<nil>"
time="2021-06-22T19:44:24.245742161Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T19:44:24.247157639Z" level=info msg="device-queue item updated" ctx_id=bc5b4b45-a7b1-4445-8896-988ad4911a0c dev_eui=aaaaaa0001000036 emit_at_time_since_gps_epoch="<nil>" f_cnt=57 is_pending=false timeout_after="<nil>"
As you can see, the same downlink_id is set for all 4 downlinks. The gateway-bridge does in fact ACK each downlink_id, but I think the LNS only processes one (since its the same id?). The result is that one of the 4 downlinks is removed from the queue, and the next interval the next 3 are sent (again with a duplicate downlink_id), then 2, then 1. See below:
time="2021-06-22T20:10:46.727027720Z" level=info msg="device-queue item created" ctx_id=7d62eb01-6d8a-429c-a162-d2909d310436 dev_eui=aaaaaa0001000033 f_cnt=61
time="2021-06-22T20:10:46.765936077Z" level=info msg="device-queue item created" ctx_id=2b0150af-ec98-4d46-a01a-a14318c7335f dev_eui=aaaaaa0001000035 f_cnt=55
time="2021-06-22T20:10:46.767708952Z" level=info msg="device-queue item created" ctx_id=d96f552c-c8b4-4aa5-9d97-3f88282751d1 dev_eui=aaaaaa0001000034 f_cnt=46
time="2021-06-22T20:10:46.777393139Z" level=info msg="device-queue item created" ctx_id=4fea5fdc-68c6-4c92-8d96-12a16dc5aa7a dev_eui=aaaaaa0001000036 f_cnt=63
time="2021-06-22T20:10:47.097388891Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:47.099754506Z" level=info msg="device-queue item updated" ctx_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b dev_eui=aaaaaa0001000033 emit_at_time_since_gps_epoch="<nil>" f_cnt=61 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:47.101095120Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:47.102459822Z" level=info msg="device-queue item updated" ctx_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b dev_eui=aaaaaa0001000034 emit_at_time_since_gps_epoch="<nil>" f_cnt=46 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:47.103759913Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:47.105112317Z" level=info msg="device-queue item updated" ctx_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b dev_eui=aaaaaa0001000035 emit_at_time_since_gps_epoch="<nil>" f_cnt=55 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:47.107484704Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:47.109074614Z" level=info msg="device-queue item updated" ctx_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b dev_eui=aaaaaa0001000036 emit_at_time_since_gps_epoch="<nil>" f_cnt=63 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:47.150034708Z" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b gateway_id=00800000a0005a49
time="2021-06-22T20:10:47.151591651Z" level=info msg="device-queue deleted" ctx_id=a6d56c62-3f14-4f7a-beec-9ff2aa04b16b id=936
time="2021-06-22T20:10:49.115899713Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=542a45b3-5cba-4716-b209-3be7529fa3e5 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:49.117777055Z" level=info msg="device-queue item updated" ctx_id=542a45b3-5cba-4716-b209-3be7529fa3e5 dev_eui=aaaaaa0001000033 emit_at_time_since_gps_epoch="<nil>" f_cnt=61 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:49.119186249Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=542a45b3-5cba-4716-b209-3be7529fa3e5 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:49.120542765Z" level=info msg="device-queue item updated" ctx_id=542a45b3-5cba-4716-b209-3be7529fa3e5 dev_eui=aaaaaa0001000034 emit_at_time_since_gps_epoch="<nil>" f_cnt=46 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:49.121864296Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=542a45b3-5cba-4716-b209-3be7529fa3e5 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:49.123197843Z" level=info msg="device-queue item updated" ctx_id=542a45b3-5cba-4716-b209-3be7529fa3e5 dev_eui=aaaaaa0001000035 emit_at_time_since_gps_epoch="<nil>" f_cnt=55 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:49.177815248Z" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=542a45b3-5cba-4716-b209-3be7529fa3e5 gateway_id=00800000a0005a49
time="2021-06-22T20:10:49.179336350Z" level=info msg="device-queue deleted" ctx_id=542a45b3-5cba-4716-b209-3be7529fa3e5 id=935
time="2021-06-22T20:10:51.130338429Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:51.132058963Z" level=info msg="device-queue item updated" ctx_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 dev_eui=aaaaaa0001000033 emit_at_time_since_gps_epoch="<nil>" f_cnt=61 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:51.133485208Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:51.134889787Z" level=info msg="device-queue item updated" ctx_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 dev_eui=aaaaaa0001000034 emit_at_time_since_gps_epoch="<nil>" f_cnt=46 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:51.193371124Z" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 gateway_id=00800000a0005a49
time="2021-06-22T20:10:51.195046832Z" level=info msg="device-queue deleted" ctx_id=f9664ad9-790f-4b7b-97e2-c3c149e2c827 id=934
time="2021-06-22T20:10:53.141731743Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=3c02104e-7372-453f-bb60-37ae0e364e66 gateway_id=00800000a0005a49 qos=1 topic=gateway/00800000a0005a49/command/down
time="2021-06-22T20:10:53.143711096Z" level=info msg="device-queue item updated" ctx_id=3c02104e-7372-453f-bb60-37ae0e364e66 dev_eui=aaaaaa0001000033 emit_at_time_since_gps_epoch="<nil>" f_cnt=61 is_pending=false timeout_after="<nil>"
time="2021-06-22T20:10:53.203819659Z" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=3c02104e-7372-453f-bb60-37ae0e364e66 gateway_id=00800000a0005a49
time="2021-06-22T20:10:53.205371067Z" level=info msg="device-queue deleted" ctx_id=3c02104e-7372-453f-bb60-37ae0e364e66 id=933
Hi.
I reproduced a similar error for Class-B too, with a similar setup (3 nodes powered down and 2 powered up).
In my test, I am ussing the lastest versions of chirpstack and sentech lorapkg_forwarder and lora_gateway, all in the same machine based in raspberry pi 4 (RAK4722C).
In my case, the error occurs when queue of application-server has 2 queued downlinks or more for a particular node and this node sends a Class-A pkg uplink. Network-server stops working propiertily in class B for thesse node and try to flush queue and create a downlinks for each second beacons (it is weird because beacon period is set to 128s in device-profile, lorapkg_forwarder and any configuration of chirpstack and semteck_udp). Network-server try to create a new "device-queue" for each packet in the queue of the node each 128 seconds (for the same iteration, “beacon_start_time_s increase” each time in one second and f_cnt increase by 1 each time and ctx_id is always the same).
This malfuntion cause that semtech lorapkg_forwarder must reject a lot of packages because other package was previously in the same pingslot for those beacon period. After a while, system collapse and all nodes stop reciving pakages in class B from gateway.
In previous versions of network-server (3.11) it works fine (but has 4% of rejection pkg in lorapkg_forwarder)
Syslog:
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.577110376+02:00" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 gateway_id=dca632fffe0ca79b qos=0 topic=gateway/dca632fffe0ca79b/command/down
Jun 28 12:36:07 rak-gateway chirpstack-gateway-bridge[493]: time="2021-06-28T12:36:07.57771997+02:00" level=info msg="integration/mqtt: downlink frame received" downlink_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 gateway_id=dca632fffe0ca79b
Jun 28 12:36:07 rak-gateway chirpstack-gateway-bridge[493]: time="2021-06-28T12:36:07.578637406+02:00" level=info msg="integration/mqtt: publishing event" downlink_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 event=ack qos=0 topic=gateway/dca632fffe0ca79b/event/ack
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.579268776+02:00" level=info msg="device-session saved" ctx_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 dev_addr=0141551f dev_eui=123456789a000003
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.581165109+02:00" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 gateway_id=dca632fffe0ca79b
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.581386195+02:00" level=info msg="storage: downlink-frame saved" ctx_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 token=21836
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.582867337+02:00" level=info msg="device-queue item updated" ctx_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 dev_eui=123456789a000003 emit_at_time_since_gps_epoch=363586h36m26.08s f_cnt=44 is_pending=false timeout_after="2021-06-28 11:36:16.08 +0100 BST"
Jun 28 12:36:07 rak-gateway chirpstack-application-server[489]: time="2021-06-28T12:36:07.586780722+02:00" level=error msg=COLLISION_PACKET dev_eui=123456789a000003 type=DATA_DOWN_GATEWAY
Jun 28 12:36:07 rak-gateway chirpstack-application-server[489]: time="2021-06-28T12:36:07.587865894+02:00" level=info msg="finished unary call with code OK" ctx_id=ed38753d-3ce0-49b4-9559-0a4d4708f336 grpc.code=OK grpc.method=HandleError grpc.service=as.ApplicationServerService grpc.start_time="2021-06-28T12:36:07+02:00" grpc.time_ms=4.308 peer.address="[::1]:33050" span.kind=server system=grpc
Jun 28 12:36:07 rak-gateway chirpstack-application-server[489]: time="2021-06-28T12:36:07.588056203+02:00" level=info msg="integration/mqtt: publishing event" ctx_id=ed38753d-3ce0-49b4-9559-0a4d4708f336 dev_eui=123456789a000003 qos=0 retain=true topic=application/5/device/123456789a000003/event/error
Jun 28 12:36:07 rak-gateway chirpstack-application-server[489]: time="2021-06-28T12:36:07.587869561+02:00" level=info msg="integration/logger: logging event" ctx_id=ed38753d-3ce0-49b4-9559-0a4d4708f336 dev_eui=123456789a000003 type=error
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.588418785+02:00" level=info msg="finished client unary call" ctx_id=554c4d36-2606-4da6-b8d7-74a3df724ff3 grpc.code=OK grpc.ctx_id=ed38753d-3ce0-49b4-9559-0a4d4708f336 grpc.duration=5.555596ms grpc.method=HandleError grpc.service=as.ApplicationServerService span.kind=client system=grpc
Jun 28 12:36:07 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:07.588508097+02:00" level=info msg="sent error to application-server" ctx_id=554c4d36-2606-4da6-b8d7-74a3df724ff3
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.596733538+02:00" level=info msg="device-queue flushed" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.599189465+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=96080 beacon_start_time_s=10225874 dev_addr=0141551f ping_nb=1 ping_offset_ms=3132 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.600753882+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=44
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.602307948+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=31040 beacon_start_time_s=10225875 dev_addr=0141551f ping_nb=1 ping_offset_ms=964 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.603524986+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=45
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.605112051+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=81050 beacon_start_time_s=10225876 dev_addr=0141551f ping_nb=1 ping_offset_ms=2631 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.606402957+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=46
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.607774084+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=78980 beacon_start_time_s=10225877 dev_addr=0141551f ping_nb=1 ping_offset_ms=2562 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.609097693+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=47
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.610720701+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=106910 beacon_start_time_s=10225878 dev_addr=0141551f ping_nb=1 ping_offset_ms=3493 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.612025885+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=48
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.613733335+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=89000 beacon_start_time_s=10225879 dev_addr=0141551f ping_nb=1 ping_offset_ms=2896 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.615057926+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=49
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.616619398+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=72140 beacon_start_time_s=10225880 dev_addr=0141551f ping_nb=1 ping_offset_ms=2334 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.617884028+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=50
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.619434427+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=16280 beacon_start_time_s=10225881 dev_addr=0141551f ping_nb=1 ping_offset_ms=472 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.620765147+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=51
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.622278417+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=31340 beacon_start_time_s=10225882 dev_addr=0141551f ping_nb=1 ping_offset_ms=974 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.623534492+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=52
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.625053206+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=73970 beacon_start_time_s=10225883 dev_addr=0141551f ping_nb=1 ping_offset_ms=2395 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.626418203+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=53
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.627987768+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=102740 beacon_start_time_s=10225884 dev_addr=0141551f ping_nb=1 ping_offset_ms=3354 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.629246954+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=54
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.631396798+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=107450 beacon_start_time_s=10225885 dev_addr=0141551f ping_nb=1 ping_offset_ms=3511 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.632650261+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=55
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.634195271+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=34070 beacon_start_time_s=10225886 dev_addr=0141551f ping_nb=1 ping_offset_ms=1065 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.636011699+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=56
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.637521951+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=93560 beacon_start_time_s=10225887 dev_addr=0141551f ping_nb=1 ping_offset_ms=3048 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.638770544+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=57
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.640345331+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=28700 beacon_start_time_s=10225888 dev_addr=0141551f ping_nb=1 ping_offset_ms=886 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.642070966+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=58
[.....]
[.....]
[similar fcnt 59 to 107 ]
[.....]
[.....]
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.798346305+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=26780 beacon_start_time_s=10225938 dev_addr=0141551f ping_nb=1 ping_offset_ms=822 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.79959362+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=108
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.801005356+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=114530 beacon_start_time_s=10225939 dev_addr=0141551f ping_nb=1 ping_offset_ms=3747 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.8022963+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=109
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.803694203+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=103250 beacon_start_time_s=10225940 dev_addr=0141551f ping_nb=1 ping_offset_ms=3371 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.804984147+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=110
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.806349866+02:00" level=info msg="get next ping-slot timestamp" after_beacon_start_time_ms=39110 beacon_start_time_s=10225941 dev_addr=0141551f ping_nb=1 ping_offset_ms=1233 ping_slot_n=0
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.807587682+02:00" level=info msg="device-queue item created" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 f_cnt=111
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.809847023+02:00" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 gateway_id=dca632fffe0ca79b qos=0 topic=gateway/dca632fffe0ca79b/command/down
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.811637859+02:00" level=info msg="device-session saved" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_addr=0141551f dev_eui=123456789a000003
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.812148622+02:00" level=info msg="storage: downlink-frame saved" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 token=2030
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.812501186+02:00" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 gateway_id=dca632fffe0ca79b
Jun 28 12:36:08 rak-gateway chirpstack-gateway-bridge[493]: time="2021-06-28T12:36:08.810541206+02:00" level=info msg="integration/mqtt: downlink frame received" downlink_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 gateway_id=dca632fffe0ca79b
Jun 28 12:36:08 rak-gateway chirpstack-gateway-bridge[493]: time="2021-06-28T12:36:08.811270796+02:00" level=info msg="integration/mqtt: publishing event" downlink_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 event=ack qos=0 topic=gateway/dca632fffe0ca79b/event/ack
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.81389146+02:00" level=info msg="device-queue item updated" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 dev_eui=123456789a000003 emit_at_time_since_gps_epoch=363586h39m28.08s f_cnt=44 is_pending=false timeout_after="2021-06-28 11:39:18.08 +0100 BST"
Jun 28 12:36:08 rak-gateway chirpstack-network-server[23774]: time="2021-06-28T12:36:08.816393071+02:00" level=error msg="uplink: handle downlink tx ack error" ctx_id=07eefac6-0f96-4b65-8535-32ddb0dc1600 error="get device-queue item error: object does not exist" gateway_id=dca632fffe0ca79b token=2030
Here is another instance of this occurring out of the blue, it starts spamming the same downlink_id over and over only a millisecond apart. Our gateway so far today has 355,000 downlink packets and crippled the site 
Jun 28, 2021 @ 09:10:10.079 time="2021-06-28T09:10:10.078892275Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.079 time="2021-06-28T09:10:10.079414075Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=44212
Jun 28, 2021 @ 09:10:10.080 time="2021-06-28T09:10:10.080238961Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.080 time="2021-06-28T09:10:10.080637581Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=13137
Jun 28, 2021 @ 09:10:10.081 time="2021-06-28T09:10:10.081437161Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.081 time="2021-06-28T09:10:10.081809257Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=51908
Jun 28, 2021 @ 09:10:10.082 time="2021-06-28T09:10:10.082570696Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.083 time="2021-06-28T09:10:10.082951126Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=47668
Jun 28, 2021 @ 09:10:10.083 time="2021-06-28T09:10:10.083721656Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.084 time="2021-06-28T09:10:10.084052071Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=36
Jun 28, 2021 @ 09:10:10.084 time="2021-06-28T09:10:10.084819974Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.085 time="2021-06-28T09:10:10.085174784Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=61267
Jun 28, 2021 @ 09:10:10.086 time="2021-06-28T09:10:10.086012107Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.086 time="2021-06-28T09:10:10.086384177Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=8483
Jun 28, 2021 @ 09:10:10.087 time="2021-06-28T09:10:10.087217603Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.087 time="2021-06-28T09:10:10.087595468Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=27094
Jun 28, 2021 @ 09:10:10.088 time="2021-06-28T09:10:10.088390358Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.088 time="2021-06-28T09:10:10.088748064Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=32472
Jun 28, 2021 @ 09:10:10.089 time="2021-06-28T09:10:10.089497265Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.089 time="2021-06-28T09:10:10.089884439Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=9108
Jun 28, 2021 @ 09:10:10.090 time="2021-06-28T09:10:10.090633764Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Jun 28, 2021 @ 09:10:10.091 time="2021-06-28T09:10:10.090984794Z" level=info msg="storage: downlink-frame saved" ctx_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb token=62514
Jun 28, 2021 @ 09:10:10.091 time="2021-06-28T09:10:10.091742503Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=32b3521d-78ef-46bc-97c1-3fef2e0e0ffb gateway_id=00800000a0006af2 qos=0 topic=gateway/00800000a0006af2/command/down
Would you create a tag for installing like a package (apt-get install)?
I have tried to install with docker and souce code and I couldn’t install correctly.
What’s also interesting is the timings, since we have gateway_downlink_lock_duration set to 200ms, yet it was sending command/down to the same GW back to back.
Could be a related issue? It honored the gateway_downlink_lock_duration earlier, but all the sudden stopped
@brocaar
I see the uuid.NewV4() call was added to the normal downlink loop, but not the multicast loop.
Does loop chirpstack-network-server/scheduler.go at master · brocaar/chirpstack-network-server · GitHub need this bug fix as well?
I traced a bit into the multicast enqueuing, and it looks to me that the fix from Increment by multicast gateway delay after first item. · brocaar/chirpstack-network-server@6e51d2a · GitHub makes it so that ALL multicast items queued get the same scheduled_at setting (if only one gateway). Before, it was automatically adding multicastGatewayDelay before queuing, which spaced the downlinks out by 2 seconds (even if only 1 gateway was involved).
The problem is that fuota-server queues ALL fuota packets at the same time, so I think 400 downlinks in our last fuota were all attempted to be sent down at once. 380 out of 400 are rejected by the packet forwarder due to JIT queue full, but more importantly 399 out of those 400 are not ack’d due to the duplicate downlink_id bug.
I suspect there is another bug hiding here due to that above commit. It seems like we need a ts = ts.Add(x) after GetMaxScheduleAtForMulticastGroup call (i.e. Get latest schedule_at for this group and schedule it x seconds after.
Hi @brocaar
I have compiled network-server from source code (last commit from github) and I continue seeing the same bug in class-B.
It tries to send all queued packages in the same beacon period and it generates many wrong pingslot downlinks each second, it doesn’t respect beacon periods, beacon_start_time_s always increments in 1 seg by each f_cnt in the same iteration.
I think that it doesn’t considere that beacon period is 128 seg (Default por EU868) and it interpretes that beacon period is always 1 seg.
It is ignoring [network_server.scheduler] → scheduler_interval="2s" too. It continues generating a pingslot downlink each 30ms (Default for LoRaWAN Region EU868) instead scheduler_interval time. (.toml is located in ~/.config/chirpstack-network-server/chirpstack-network-server.toml )
This behavour causes that sentech package_forwarder rejects a lot of packages.
Network-server error LOG (ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35):
INFO[2058] device-queue flushed ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=120620 beacon_start_time_s=10228606 dev_addr=0048e04c ping_nb=1 ping_offset_ms=3950 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=129
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=16310 beacon_start_time_s=10228607 dev_addr=0048e04c ping_nb=1 ping_offset_ms=473 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=130
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=8960 beacon_start_time_s=10228608 dev_addr=0048e04c ping_nb=1 ping_offset_ms=228 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=131
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=96260 beacon_start_time_s=10228609 dev_addr=0048e04c ping_nb=1 ping_offset_ms=3138 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=132
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=71300 beacon_start_time_s=10228610 dev_addr=0048e04c ping_nb=1 ping_offset_ms=2306 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=133
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=28040 beacon_start_time_s=10228611 dev_addr=0048e04c ping_nb=1 ping_offset_ms=864 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=134
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=116840 beacon_start_time_s=10228612 dev_addr=0048e04c ping_nb=1 ping_offset_ms=3824 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=135
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=76850 beacon_start_time_s=10228613 dev_addr=0048e04c ping_nb=1 ping_offset_ms=2491 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=136
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=73130 beacon_start_time_s=10228614 dev_addr=0048e04c ping_nb=1 ping_offset_ms=2367 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=137
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=49430 beacon_start_time_s=10228615 dev_addr=0048e04c ping_nb=1 ping_offset_ms=1577 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=138
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=47660 beacon_start_time_s=10228616 dev_addr=0048e04c ping_nb=1 ping_offset_ms=1518 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=139
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=8570 beacon_start_time_s=10228617 dev_addr=0048e04c ping_nb=1 ping_offset_ms=215 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=140
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=8960 beacon_start_time_s=10228618 dev_addr=0048e04c ping_nb=1 ping_offset_ms=228 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=141
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=32030 beacon_start_time_s=10228619 dev_addr=0048e04c ping_nb=1 ping_offset_ms=997 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=142
INFO[2058] get next ping-slot timestamp after_beacon_start_time_ms=55730 beacon_start_time_s=10228620 dev_addr=0048e04c ping_nb=1 ping_offset_ms=1787 ping_slot_n=0
INFO[2058] device-queue item created ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 f_cnt=143
INFO[2058] gateway/mqtt: publishing gateway command command=down downlink_id=ffff3440-fea2-43ee-b220-3e74423a8a35 gateway_id=dca632fffe0ca79b qos=0 topic=gateway/dca632fffe0ca79b/command/down
INFO[2058] backend/gateway: downlink tx acknowledgement received downlink_id=ffff3440-fea2-43ee-b220-3e74423a8a35 gateway_id=dca632fffe0ca79b
INFO[2058] device-session saved ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_addr=0048e04c dev_eui=123456789a000004
INFO[2058] storage: downlink-frame saved ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 token=65535
INFO[2058] device-queue item updated ctx_id=ffff3440-fea2-43ee-b220-3e74423a8a35 dev_eui=123456789a000004 emit_at_time_since_gps_epoch=363683h48m8.62s f_cnt=129 is_pending=false timeout_after=“2021-07-02 12:47:58.62 +0100 BST”
Note that scheduler_interval is just how often the task runs, it doesn’t affect how often downlinks can be sent down, that is done with other settings/locks (e.g. if 200 packets are in the queue and not locked, 1 interval execution would handle all 200). However, this thread is about class-c and your instance is possibly a bit different with class-b
@Curt_Black
What parameter do I change for correct that network-server handle correctly pingslot? How can I lock packages/queue for send one by one in a beacon period?