Lost OTAA Activation Data With Redis "appendonly yes" set

Hi,

I am running the latest gateway, LNS and application server on an ARM 9 platform running Ubuntu 16.04 with Redis 5.0.2 (00000000/0) 32 bit.

All my devices are loosing their OTAA activation data (“This device has not (yet) been activated”) after a power cycle/reboot with data is still coming into the gateway, even though I have the following set in /etc/redis/redis.conf :

Please check http://redis.io/topics/persistence for more information.

appendonly yes

The name of the append only file (default: “appendonly.aof”)

appendfilename “appendonly.aof”

I am now running really late on project delivery, so if anyone can help I would be really grateful.

Many thanks,

Mark

I’m not a fan of appendonly and prefer snapshotting at an interval where I can tolerate potential, small loss. I like the following as a starting point:

redis-server --save 300 1 --save 60 100 --appendonly no

You also haven’t mentioned where the data is stored. Is this in a Docker volume or running natively in an OS (I’m assuming the latter, given ARM9 and Ubuntu)?

With either approach, you should be able to see your persistence file being written while Redis is running.

Thank you ever so much for the prompt reply, I really appreciate it.

This is just running natively in the OS.

So I made these changes to /etc/redis/redis.conf :

save 300 1
save 60 100
appendonly no

Now I can see the following in redis-log:
823:M 28 Feb 2021 09:54:55.923 * 100 changes in 60 seconds. Saving…
823:M 28 Feb 2021 09:54:55.923 * Background saving started by pid 29374
29374:C 28 Feb 2021 09:54:55.934 * DB saved on disk
29374:C 28 Feb 2021 09:54:55.936 * RDB: 0 MB of memory used by copy-on-write
823:M 28 Feb 2021 09:54:56.024 * Background saving terminated with success

Yet, I am still losing my OTAA activation data after a power cycle/reboot, is there anything else that might need sorting out?

Thanks again.

I would try manually writing data to Redis via the CLI and see if it persists across service restarts.

Thanks a lot, stupid question perhaps, but how do I do that as I could not find a CLI option to do that, only a way to launch the CLI as per:

sudo systemctl stop chirpstack-network-server
sudo chirpstack-network-server --config /etc/chirpstack-network-server/chirpstack-network-server.toml

I checked the redis db and unless I am mistaken, both the NS & AS are writing to the db OK, but there are no keys being stored (the device in question is deveui 00137a100000d1c1, addr 00 21 4f 23, network session key 2b bc b7 dc 0f e4 f3 3c 8e c1 49 5e a5 3a b8 3c, app session key ef 20 92 16 62 de e8 1c 1a 40 5c cd db 1b e6 3b):

127.0.0.1:6379> KEYS *

  1. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589440”
  2. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614585600”
  3. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587580”
  4. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587160”
  5. “lora:ns:device:00137a100000d1c1:mac:pending:5”
  6. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614560400”
  7. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589200”
  8. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590220”
  9. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586860”
  10. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584820”
  11. “lora:ns:device:00137a100000d1c1:mac:pending:8”
  12. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590640”
  13. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590040”
  14. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585480”
  15. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585060”
  16. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588240”
  17. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587220”
  18. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586500”
  19. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589680”
  20. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590940”
  21. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584400”
  22. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590100”
  23. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587940”
  24. “lora:ns:gw:meta:0080000000019c85”
  25. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588900”
  26. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585840”
  27. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585540”
  28. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591240”
  29. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585420”
  30. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614574800”
  31. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586260”
  32. “lora:ns:devaddr:004e5191”
  33. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587340”
  34. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588420”
  35. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590160”
  36. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586680”
  37. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586620”
  38. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586560”
  39. “lora:ns:device:00137a100000d1c1:gwrx”
  40. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586020”
  41. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584340”
  42. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590760”
  43. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590820”
  44. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587760”
  45. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614546000”
  46. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584880”
  47. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584640”
  48. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589920”
  49. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589260”
  50. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588960”
  51. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585600”
  52. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591180”
  53. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588780”
  54. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589320”
  55. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614582000”
  56. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584460”
  57. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590520”
  58. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589080”
  59. “lora:ns:device:00137a100000d1c1:mac:pending:7”
  60. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587820”
  61. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585960”
  62. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586380”
  63. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586440”
  64. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586740”
  65. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591480”
  66. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589140”
  67. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587100”
  68. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585120”
  69. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587700”
  70. “lora:as:metrics:{gw:0080000000019c85}:MONTH:1612137600”
  71. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590340”
  72. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586800”
  73. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585780”
  74. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585180”
  75. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586200”
  76. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585360”
  77. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614538800”
  78. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590280”
  79. “lora:ns:device:00137a100000d1c1”
  80. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586980”
  81. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591420”
  82. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589980”
  83. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588060”
  84. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587280”
  85. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589620”
  86. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589560”
  87. “lora:as:metrics:{gw:0080000000019c85}:MONTH:1614556800”
  88. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589740”
  89. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586080”
  90. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589500”
  91. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585000”
  92. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614542400”
  93. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588840”
  94. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591360”
  95. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588600”
  96. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584700”
  97. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614531600”
  98. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591000”
  99. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586320”
  100. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590700”
  101. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614556800”
  102. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584580”
  103. “lora:as:metrics:{gw:0080000000019c85}:DAY:1614556800”
  104. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587640”
  105. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585660”
  106. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587520”
  107. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614567600”
  108. “lora:ns:devaddr:00214f23”
  109. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589800”
  110. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584520”
  111. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588720”
  112. “lora:as:metrics:{gw:0080000000019c85}:DAY:1614470400”
  113. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588300”
  114. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587400”
  115. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587460”
  116. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588180”
  117. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588480”
  118. “lora:ns:sp:f24ad6a9-d9e1-4e02-a0ce-3b39d2ab61d6”
  119. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589380”
  120. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586920”
  121. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587880”
  122. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584940”
  123. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590460”
  124. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588360”
  125. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585240”
  126. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591300”
  127. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614549600”
  128. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588660”
  129. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591540”
  130. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614571200”
  131. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588000”
  132. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614553200”
  133. “lora:ns:dp:86ac0868-7067-4cec-9aaf-4fad83e46285”
  134. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591120”
  135. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589020”
  136. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614584760”
  137. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585720”
  138. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588120”
  139. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614587040”
  140. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614589860”
  141. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614588540”
  142. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614564000”
  143. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590880”
  144. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585900”
  145. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614591060”
  146. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590580”
  147. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614578400”
  148. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614590400”
  149. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614586140”
  150. “lora:as:metrics:{gw:0080000000019c85}:MINUTE:1614585300”
  151. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614589200”
  152. “lora:as:metrics:{gw:0080000000019c85}:HOUR:1614535200”