Filesystem Full (Redis AOF) - LoRa messages not processed

DeBuffel · March 25, 2021, 10:19am

Hi,

I am using Chirpstack (fullOS) on a RPi 4 (4GB) and a 32GB SD card. I noticed that the gateway is no longer working while all processes etc. are all working correct.

After checking log files I found out the the filesystem is full and seems that Redis is not able to save RDB snapshots to disk due to a file system that is full (see snapshot from log)

raspberrypi4:~$ tail -f /var/log/messages
Jan  7 11:57:16 raspberrypi4 local0.warn redis[480]: Background saving error
Jan  7 11:57:22 raspberrypi4 local0.notice redis[480]: 1 changes in 120 seconds. Saving...
Jan  7 11:57:22 raspberrypi4 local0.notice redis[480]: Background saving started by pid 14004
Jan  7 11:57:22 raspberrypi4 local0.warn redis[14004]: Write error saving DB on disk: No space left on device
Jan  7 11:57:22 raspberrypi4 local0.warn redis[480]: Background saving error
Jan 07 11:57:24 raspberrypi4 user.info chirpstack-concentratord-sx1301[516]: Publishing stats event, stats_id: 2730ac58-fff1-4bec-9c0b-30ef3799667b, rx_received: 1, rx_received_ok: 1, tx_received: 0, tx_emitted: 0
Jan  7 11:57:24 raspberrypi4 user.info chirpstack-gateway-bridge[507]: time="2021-01-07T11:57:24Z" level=info msg="backend/concentratord: stats event received" stats_id=2730ac58-fff1-4bec-9c0b-30ef3799667b
Jan  7 11:57:24 raspberrypi4 user.info chirpstack-gateway-bridge[507]: time="2021-01-07T11:57:24Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=2730ac58-fff1-4bec-9c0b-30ef3799667b topic=gateway/dca632fffe817e1c/event/stats
Jan  7 11:57:24 raspberrypi4 user.err chirpstack-network-server[498]: time="2021-01-07T11:57:24Z" level=error msg="gateway/mqtt: acquire lock error" error="acquire lock error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on dis
Jan  7 11:57:28 raspberrypi4 local0.notice redis[480]: 1 changes in 120 seconds. Saving...
Jan  7 11:57:28 raspberrypi4 local0.notice redis[480]: Background saving started by pid 14007
Jan  7 11:57:28 raspberrypi4 local0.warn redis[14007]: Write error saving DB on disk: No space left on device

The filesystem is full - see below:

raspberrypi4:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
none                      1.7G      8.0K      1.7G   0% /dev
/dev/mmcblk0p2          479.9M    275.5M    168.6M  62% /mnt/root
/dev/mmcblk0p4          479.9M    469.7M         0 100% /data
overlay                 479.9M    469.7M         0 100% /
tmpfs                     1.9G    232.0K      1.9G   0% /run
tmpfs                     1.9G    320.0K      1.9G   0% /var/volatile
/dev/mmcblk0p1           39.9M     21.8M     18.1M  55% /boot
raspberrypi4:~$

Due to the above problem no LoRa messages are processed at all.

Has anybody experienced the same? And how can I prevent this from happening (e.g. can I enlarge the mounted filesystems?)

Thanks in advance

bconway · March 25, 2021, 1:12pm

I would check to see what’s using all your disk space. Logging should not be all that verbose in a default configuration.

DeBuffel · March 25, 2021, 1:53pm

It appear to be the appendonly.aof (used by redis)

raspberrypi4:/data/upperdir/var/lib/redis$ ls -la
drwxr-xr-x    2 redis    redis         4096 Jan  7 12:56 .
drwxr-xr-x   10 root     root          4096 Dec 30 15:01 ..
-rw-r--r--    1 root     root     414289320 Jan  6 20:36 appendonly.aof
-rw-r--r--    1 root     root         37994 Jan  6 18:35 dump.rdb
raspberrypi4:/data/upperdir/var/lib/redis$

However this is weird as the redis config (raspberrypi4:/etc/redis$ nano redis.conf) is configured with

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

I also tried to manually start BGREWRITEAOF from redis-cli - no result and the appendonly.aof file remained the same

Only after deleting the appendonly.aof file manually and reboot the Chirpstack started to work again

bconway · March 25, 2021, 1:55pm

That makes sense (sort of), it’s keeping the full history of everything that ran through Redis. I prefer not to use append persistence with Redis, and using the following as a starting point, and then tune based on requirements:

redis-server --save 300 1 --save 60 100 --appendonly no

However, I do not use ChirpStack’s gateway OS, and cannot give rationale for that being the (presumable) default.

All that being said, that is a TON (415 MB!) of updates going on in Redis. Unless this system has been running for a very long time, that does not seem normal.

DeBuffel · March 25, 2021, 2:05pm

Should the

auto-aof-rewrite-percentage / min-size

not ensure that the filesize is controlled / minimazed to configured size?

Yes system was running for quite a while however it should not hang due to file size which is too big

Maybe I have to switch of append persistence in Redis. I am not that familiar with Redis (need to readup) and maybe some other Chirpstack users have experience with it

DeBuffel · March 26, 2021, 8:58am

I changed the appendonly to “no” // for my Chirpstack GW this is ok and now I avoid that the filesystem get full (and the Gateway stops working).

Tested it and all good. Thanks for your support

brocaar · March 30, 2021, 4:11pm

Would you like to create a pull-request for this, to update the ChirpStack Gateway OS Redis config?