Assistance with invalid MIC on ABP devices

ChirpStack Application Server
1

First things first: I have already read the previous posts on this and attempted the advice in e.g. Error: device-session does not exist or invalid fcnt or mic - #2 by brocaar and attempted it.

The problem: I have a number of devices that occasionally have Unconfirmed Uplinks fail with Invalid MIC errors (from the MQTT bus, this on messages:"type":"UPLINK_MIC","error":"invalid MIC",). Sometimes they self resolve.

I have attempted to reset counters. I have attempted to toggle the framecounter validation flag for the affected devices.

In addition to this I have also done some light reading of the source in an attempt to find out why this might be happening, as well as looking at the redis keys (although this was less useless since I’d have to decode the protobuf object, and didn’t have things set up for that).

These devices definitely do occasionally reboot (battery changes, software crashes), and frame counter resets are typical. But sometimes some of them just stop working entirely and it’s always this failure.

Maybe there is something incorrect with the device-session data in redis? I’m not sure how to tell, the logs don’t seem to provide any more information on this either. Similarly uncertain about how to recreate the session information or so.

System is on the latest versions of chirpstack, on a Debian installation from the https://artifacts.chirpstack.io/packages/3.x/deb repo.

ii  chirpstack-application-server 3.16.0       amd64        ChirpStack Application Server
ii  chirpstack-gateway-bridge     3.12.0       amd64        ChirpStack Gateway Bridge
ii  chirpstack-network-server     3.14.0       amd64        ChirpStack Network Server

Notes: these seem to only work on ABP when using Chirpstack. They worked on OTAA with TTN, for reasons unclear to me as I didn’t have time to investigate, so it was necessary to roll them out with ABP (and OTAA is not an option).

2

I have the same problem.
With OTAA ClassC

example:

image
image827×526 29.9 KB

every error are:

type:“UPLINK_MIC”
error:“invalid MIC”
fCnt:9

the fCnt are the same of the last good values received

that’s probably the problem

GOOD–> value receive with fCnt:7
image

BAD → next one with MIC error and same fCnt:7
image

GOOD → next one after 1 hour
image

3

Indeed, unconfirmed uplinks with repeated frame count are illegal (confirmed ones could be unnecessary retries where it was the ack not the uplink that got lost).

Unfortunately there are some node firmwares where the manufacturer has implemented this out-of-spec behavior.

You basically have to take it up with the manufacturer or firmware author and get them to bring their device into compliance with the LoRaWan spec.

Or else you ignore them - dropping the packet with or without notification is spec-compliant behavior by the server.

4

To briefly reiterate some parts in my original post, I’ve attempted to reset counters, restart devices, as well as to disable frame-counter validation, and this error kept happening. So I’m not sure that it was the cause of the issue I observed.

Once it happens again on another device I’ll have to doublecheck to be absolutely certain, but I also don’t recall observing duplicated frame counters at the time.

I have also attempted to upgrade the stack in the intervening time, so that’s been changed. Hopefully - for the sake of answers - it hasn’t eliminated the issue. But if I don’t see this happen again I’ll update the thread in some time.