We have problems with some nodes.
At same time 10 nodes enters rejoin procedure. (could be reboot or power outage, but shuld be normal)
All 10 has now same error.
I can see they send join
I can see that server did send JoinAccept
And we did send some downlink with confirmed data down
After that in frame logs I do not see any messages from node.
Node activation in lora app is showing old keys with frame counter: 6155
so I have checked loraserver log and it has shown node session error
node-session does not exist or invalid fcnt or mic"
without info about devEUI
for one node I have deleted it from lora app and recreated it.
now loraserver is reporting error:
%snode-session does not exist" dev_addr=07419f2a dev_eui=xxxxxxxxxx
How can I check new keys in lora-app or redis or pg I do not know where to look?
Is this some known bug?
And who has now wrong keys node or loraserver 
You can see the current keys in the web-interface. When you click on the device, you’ll see a tab “Device keys (OTAA)”. This info comes from Redis and should reflect the actual state.
And we did send some downlink with confirmed data down
So are you saying that all the 10 devices activate succesfully (seen from the device) and all 10 are able to send confirmed data and then after a while everything stops working? Or the activation failed?
1 Like
All 10 devices acting same.
From device perspective all is working (they send unconfirmed data to server).
For them on loraserver I just see (“node session does not exists”)
In web-interface are wrong keys I can see frame counter before rejoin.
So web-interface still has old keys.
But on web-interface I see JoinAccept (in frame logs)
So new keys are missing somehow…
I can send pictures if needed…
Yes, please share some logs of LoRa Gateway Bridge, LoRa Server and LoRa App Server. Also if you could post some screenshots of the activations. Either on the forum or in a private message.
Just to confirm, are you on the latest versions of all components?
1 Like
I cannot send private messages so here it is:
Pictures of two nodes join procedure and current key in web, and loraserver log
In log you can see one devEUI - that is the one I have deleted and recreated (for them keys are now 000000) this one is not on pictures.
https://yadi.sk/d/PA21K0vi3RaPRt
Lora gw bridge - you can just see IP and same data as in loraserver and log of appserver does not show a thing - loraserver does not forward that message to appserver (probably because error)
So I will just send loraserver log.
No it is not newest, it was run for some longer period and we did not noticed any noticeable problems with this version, until now.
If we confirm that problem is in server side then we will update to newest.
We still are not sure is it a node problem (but nodes are already deployed so we need to get to them and re flash them of debug them on place if we discover that this is not server problem).
I’m not sure if the logs can helpful. Your screenshot is from Jan 15, the logs from Jan 18. It would be helpful to have logs of the time between when things were OK and when things started to break. Ideally this includes all the .../rx and ../tx frames sent / received by the gateways so that I can replay the events locally
What is the vendor / type of your device?
yes in current log that I have captured now you can only see that device is active and it thinks it is joined.
This is microchip RN2483 with newest firmware available.
I have logs for all nodes tx data, and last I see is Jan 15.
Then all 10 rejoined but not completely.
I do not think there is loraserver log from Jan 15.
By default I have only “today” log?
I could log that for next “bad” event, and send you if it happens.
But one think I could do now, but do not know how to find that.
So loraserver did send keys and node did receive keys (he thinks he is joined).
So on lora server new keys are in database?
Where can I search for new stored keys, as you see in web I only see old keys before join with old frame counter.
Then I could just maybe change those active keys to new ones and check will loraserver forward data to app server.
Would that work (if new keys are correct)?
In the latest versions this will be easier to debug. In loraserver_ns there is a table with device_activation containing the nwk_s_key of each join. In loraserver_as there is a table with device_activation with both the app_s_key and nwk_s_key of each activation.
In the older versions the activations are not logged.
1 Like
Tnx,
we will move to a newest version of loraserver.
And I am now logging that gw TX and RX data.
If I catch it again I will send you logs.
Goran
1 Like
Ok, now we have new versions for tests.
If I check psql loraserver_ns and loraserver_as I can find all session keys.
Can you just help me with one thing.
If I would have same situation on field it would be nice if I can manually change session keys to previews one.
For now Im simulating wrong keys by letting other node with same EUID start join and disconnect him after sending Join request.
I have tried that in psql for ns and as, but I can see in lora web interface keys are still not changed.
I have tried to restart servers, but keys are still present.
Can you just tell me from where does app server reads current keys?
I will report feature problems (if any.)
We manage to debug, and find out that loraserver_ns) did have wrong keys.
Network session key was old and wrong and Application session key was correctly updated.
If we use messages that loraserver did not manage to decrypt, and if we manually generate old keys from data available in join, we can decrypt “lost” node messages.
So if you can just please point us how to change current keys (if problem repeats).
Those nodes are on field for a while and not have implemented check if messages are arriving so we need to travel to them and reboot them if it happens again…
Got it…
To change current keys in web you first need to change to ABP join
Then change keys
Save, and then you can change back to OTAA join
All working you just need to calculate old keys from join request and join accept…