Case let’s say I have an application-server integrated with MQTT. I published in downstream >2 messages atm. Only for 2 of the published messages, I received the txAck and response from the device.
Issue: all messages except the first 2 do not send to the device if I have published more then 2 messages in MQTT simultaneously.
Device: Class C
mosquito log:
application/1/device/0101010101010101/tx {"data": "AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB", "devEUI": "0101010101010101", "fPort": 12, "fCnt": "1", "confirmed": false}
application/1/device/0101010101010101/tx {"data": "AgEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB", "devEUI": "0101010101010101", "fPort": 12, "fCnt": "3", "confirmed": false}
application/1/device/0101010101010101/tx {"data": "AwEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB", "devEUI": "0101010101010101", "fPort": 12, "fCnt": "2", "confirmed": false}
application/1/device/0101010101010101/txack {"applicationID":"1","applicationName":"APPLICATION_NAME","deviceName":"DEV_NAME","devEUI":"0101010101010101","fCnt":184}
application/1/device/0101010101010101/txack {"applicationID":"1","applicationName":"APPLICATION_NAME","deviceName":"DEV_NAME","devEUI":"0101010101010101","fCnt":185}
application/1/device/0101010101010101/rx {"applicationID":"1","applicationName":"APPLICATION_NAME","deviceName":"DEV_NAME","devEUI":"0101010101010101","txInfo":{"frequency":902700000,"dr":3},"adr":false,"fCnt":338,"fPort":10,"data":"AQEBAQ=="}
application/1/device/0101010101010101/rx {"applicationID":"1","applicationName":"APPLICATION_NAME","deviceName":"DEV_NAME","devEUI":"0101010101010101","txInfo":{"frequency":902500000,"dr":3},"adr":false,"fCnt":339,"fPort":10,"data":"AQEBAg=="}
chirpstack log:
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="integration/mqtt: data-down payload received" topic=application/1/device/0101010101010101/tx
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="integration/mqtt: data-down payload received" topic=application/1/device/0101010101010101/tx
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:03Z" level=info msg="finished unary call with code OK" ctx_id=26453909-6580-4aca-8358-579a8620d8a0 grpc.code=OK grpc.method=GetNextDownlinkFCntForDevEUI grpc.service=ns.NetworkServerService grpc.start_time="2020-08-05T14:02:03Z" grpc.time_ms=1.045 peer.address="127.0.0.1:43036" span.kind=server system=grpc
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="finished client unary call" ctx_id="<nil>" grpc.code=OK grpc.ctx_id=26453909-6580-4aca-8358-579a8620d8a0 grpc.duration=2.061645ms grpc.method=GetNextDownlinkFCntForDevEUI grpc.service=ns.NetworkServerService span.kind=client system=grpc
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="integration/mqtt: data-down payload received" topic=application/1/device/0101010101010101/tx
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="integration/mqtt: data-down payload received" topic=application/1/device/0101010101010101/tx
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:03Z" level=info msg="device-queue item created" ctx_id=f206a0d9-62a6-49b3-b584-c2b67da3e764 dev_eui=0101010101010101 f_cnt=138
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:03Z" level=info msg="finished unary call with code OK" ctx_id=f206a0d9-62a6-49b3-b584-c2b67da3e764 grpc.code=OK grpc.method=CreateDeviceQueueItem grpc.service=ns.NetworkServerService grpc.start_time="2020-08-05T14:02:03Z" grpc.time_ms=2.303 peer.address="127.0.0.1:43036" span.kind=server system=grpc
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="finished client unary call" ctx_id=2c026cf1-f5b5-47fd-9558-85b14c4400ca grpc.code=OK grpc.ctx_id=f206a0d9-62a6-49b3-b584-c2b67da3e764 grpc.duration=2.898591ms grpc.method=CreateDeviceQueueItem grpc.service=ns.NetworkServerService span.kind=client system=grpc
Aug 5 14:02:03 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:03Z" level=info msg="downlink device-queue item handled" confirmed=false dev_eui=0101010101010101 f_cnt=138
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="device-queue deleted" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a id=2612
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="gateway/mqtt: publishing gateway command" command=down downlink_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a gateway_id=1111111111111111 qos=0 topic=gateway/1111111111111111/command/down
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="device-session saved" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a dev_addr=00672d1d dev_eui=0101010101010101
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="downlink-frames saved" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a token=16635
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="backend/gateway: downlink tx acknowledgement received" downlink_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a gateway_id=1111111111111111
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="sent downlink meta-data to network-controller" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:04Z" level=info msg="downlink tx acknowledged by gateway" dev_eui=0101010101010101
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:04Z" level=info msg="finished unary call with code OK" ctx_id=a722d4a0-3384-4d01-8b29-3452de262073 grpc.code=OK grpc.method=HandleTxAck grpc.service=as.ApplicationServerService grpc.start_time="2020-08-05T14:02:04Z" grpc.time_ms=1.411 peer.address="127.0.0.1:51516" span.kind=server system=grpc
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:04Z" level=info msg="integration/mqtt: publishing message" ctx_id=a722d4a0-3384-4d01-8b29-3452de262073 qos=0 topic=application/1/device/0101010101010101/txack
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="finished client unary call" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a grpc.code=OK grpc.ctx_id=a722d4a0-3384-4d01-8b29-3452de262073 grpc.duration=3.927406ms grpc.method=HandleTxAck grpc.service=as.ApplicationServerService span.kind=client system=grpc
Aug 5 14:02:04 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:04Z" level=info msg="sent tx ack to application-server" ctx_id=40fb21dc-1d32-4f2d-ac20-40dd04ec241a
Aug 5 14:02:05 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:05Z" level=info msg="gateway/mqtt: uplink frame received" gateway_id=1111111111111111 uplink_id=2ba8098f-2ba3-4d71-8a6a-bcdc3fd93df9
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:06Z" level=info msg="uplink: frame(s) collected" ctx_id=abe63d46-e91d-42d5-ae04-3d3554675d57 mtype=UnconfirmedDataUp uplink_ids="[2ba8098f-2ba3-4d71-8a6a-bcdc3fd93df9]"
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:06Z" level=info msg="device gateway rx-info meta-data saved" ctx_id=abe63d46-e91d-42d5-ae04-3d3554675d57 dev_eui=0101010101010101
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:06Z" level=info msg="device-session saved" ctx_id=abe63d46-e91d-42d5-ae04-3d3554675d57 dev_addr=00672d1d dev_eui=0101010101010101
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:06Z" level=info msg="sent uplink meta-data to network-controller" ctx_id=abe63d46-e91d-42d5-ae04-3d3554675d57 dev_eui=0101010101010101
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:06Z" level=info msg="device last-seen and dr updated" ctx_id=4a47f327-43b2-4a16-89c3-d40db73a8b68 dev_eui=0101010101010101
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:06Z" level=info msg="finished unary call with code OK" ctx_id=4a47f327-43b2-4a16-89c3-d40db73a8b68 grpc.code=OK grpc.method=HandleUplinkData grpc.request.deadline="2020-08-05T14:02:07Z" grpc.service=as.ApplicationServerService grpc.start_time="2020-08-05T14:02:06Z" grpc.time_ms=4.554 peer.address="127.0.0.1:51516" span.kind=server system=grpc
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-application-server[891904]: time="2020-08-05T14:02:06Z" level=info msg="integration/mqtt: publishing message" ctx_id=4a47f327-43b2-4a16-89c3-d40db73a8b68 qos=0 topic=application/1/device/0101010101010101/rx
Aug 5 14:02:06 ip-172-31-79-8 chirpstack-network-server[1652938]: time="2020-08-05T14:02:06Z" level=info msg="finished client unary call" ctx_id=abe63d46-e91d-42d5-ae04-3d3554675d57 grpc.code=OK grpc.ctx_id=4a47f327-43b2-4a16-89c3-d40db73a8b68 grpc.duration=8.495783ms grpc.method=HandleUplinkData grpc.service=as.ApplicationServerService span.kind=client system=grpc
UPD: I suspect that this is somehow connected with DB transactions, but this does not explain why messages silently disappear. I also tried to increase number of idle connections to DB (by default: 2) without any success