Hi, short: I could solve same problems just by setting the server time right.
I used the whole server (server-full on raspi3) disconnected from the internet and therefore the time of the server was wrong. When a device-node connected the first time, data was always right. But after a restart of the client (or of the server) a new security context had to be used. While the Network Session Key(s) had been renewed, the Application Session Key (AppSKey) was sometimes not changed. I could verify that in the Chirpstack WebInterface. The cases when the AppSKey was not updated produced wrong payload data, without any warning or any other obvious effect. There was no way to convince the Server to update the AppSKey, only removing the registered device completely and adding it again worked immediately.
Giving the server the right synchronized time solved the issue permanently.
According to the standard:
Any time the network server processes a Join-Request and generates a Join-accept frame, it shall maintain both the old security context (keys and counters, if any) and the new one until it receives the first successful uplink frame using the new context, after which the old context can be safely removed.
(copied from Trouble generating app session key, not from original standard)
So chirpstack is obviously accepting the new NetworkSessionKey(s) in any case, but the new AppSKey only if it’s newer by any timestamp. This might give some trouble also in the wild if any server did not get the right time maybe shortly after a restart or during an internet connectivity outage.
And I think this is a major bug because it can lead to data corruption. The security context is imho. both the NetworkSessionKey(s) and the AppSessionKey. Both keys must be updated or not, doing half the job is not right. By now some application might receive wrong data without noticing that, that’s bad.
I’m completely new to LoraWan/chirpstack, so I might got some things wrong, sorry. Regards,