Random "invalid MIC" errors

We have 9 power meters at a construction site, each hooked up to a Milesight UC300 IoT Controller, which is the device that sends the power meter data that it collects via Modbus over LoRaWAN. 12 modbus registers are read, which results in 3 separate packets being sent over LoRaWAN every 10 minutes.

The gateway is about 20 metres from the IoT Controllers.

Here is the weird thing… about 1/3 of the devices seems to send all 3 data packets correctly to Chirpstack, while the rest only send about 2 packets correctly, and then the “invalid MIC” error is thrown for the third packet. Then about every 3 to 4 hours, the third packet is also sent correctly, but then the errors return for another 3-4 hours. We have many other LoRaWAN devices in the same area, mostly Elsys models, and they work fine.

I have tried to run the devices with and without “Disable frame-counter validation” enabled. We have restarted the devices a couple of times, but that didn’t help. They are set up for OTAA.

Edit: Also, very often duplicate packets are sent only seconds apart.

The MIC error is unrelated to duplicated frame-counters. These are two checks, e.g. the MIC might be valid, but the counter did not increment. In your case the best thing to do would be to calculate the MIC “by hand” using the session-keys and check if this is a firmware issue with the device.