Filesystem Full (Redis AOF) - LoRa messages not processed

Hi,

I am using Chirpstack (fullOS) on a RPi 4 (4GB) and a 32GB SD card. I noticed that the gateway is no longer working while all processes etc. are all working correct.

After checking log files I found out the the filesystem is full and seems that Redis is not able to save RDB snapshots to disk due to a file system that is full (see snapshot from log)

raspberrypi4:~$ tail -f /var/log/messages
Jan  7 11:57:16 raspberrypi4 local0.warn redis[480]: Background saving error
Jan  7 11:57:22 raspberrypi4 local0.notice redis[480]: 1 changes in 120 seconds. Saving...
Jan  7 11:57:22 raspberrypi4 local0.notice redis[480]: Background saving started by pid 14004
Jan  7 11:57:22 raspberrypi4 local0.warn redis[14004]: Write error saving DB on disk: No space left on device
Jan  7 11:57:22 raspberrypi4 local0.warn redis[480]: Background saving error
Jan 07 11:57:24 raspberrypi4 user.info chirpstack-concentratord-sx1301[516]: Publishing stats event, stats_id: 2730ac58-fff1-4bec-9c0b-30ef3799667b, rx_received: 1, rx_received_ok: 1, tx_received: 0, tx_emitted: 0
Jan  7 11:57:24 raspberrypi4 user.info chirpstack-gateway-bridge[507]: time="2021-01-07T11:57:24Z" level=info msg="backend/concentratord: stats event received" stats_id=2730ac58-fff1-4bec-9c0b-30ef3799667b
Jan  7 11:57:24 raspberrypi4 user.info chirpstack-gateway-bridge[507]: time="2021-01-07T11:57:24Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=2730ac58-fff1-4bec-9c0b-30ef3799667b topic=gateway/dca632fffe817e1c/event/stats
Jan  7 11:57:24 raspberrypi4 user.err chirpstack-network-server[498]: time="2021-01-07T11:57:24Z" level=error msg="gateway/mqtt: acquire lock error" error="acquire lock error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on dis
Jan  7 11:57:28 raspberrypi4 local0.notice redis[480]: 1 changes in 120 seconds. Saving...
Jan  7 11:57:28 raspberrypi4 local0.notice redis[480]: Background saving started by pid 14007
Jan  7 11:57:28 raspberrypi4 local0.warn redis[14007]: Write error saving DB on disk: No space left on device

The filesystem is full - see below:

raspberrypi4:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
none                      1.7G      8.0K      1.7G   0% /dev
/dev/mmcblk0p2          479.9M    275.5M    168.6M  62% /mnt/root
/dev/mmcblk0p4          479.9M    469.7M         0 100% /data
overlay                 479.9M    469.7M         0 100% /
tmpfs                     1.9G    232.0K      1.9G   0% /run
tmpfs                     1.9G    320.0K      1.9G   0% /var/volatile
/dev/mmcblk0p1           39.9M     21.8M     18.1M  55% /boot
raspberrypi4:~$

Due to the above problem no LoRa messages are processed at all.

Has anybody experienced the same? And how can I prevent this from happening (e.g. can I enlarge the mounted filesystems?)

Thanks in advance

I would check to see what’s using all your disk space. Logging should not be all that verbose in a default configuration.

It appear to be the appendonly.aof (used by redis)

raspberrypi4:/data/upperdir/var/lib/redis$ ls -la
drwxr-xr-x    2 redis    redis         4096 Jan  7 12:56 .
drwxr-xr-x   10 root     root          4096 Dec 30 15:01 ..
-rw-r--r--    1 root     root     414289320 Jan  6 20:36 appendonly.aof
-rw-r--r--    1 root     root         37994 Jan  6 18:35 dump.rdb
raspberrypi4:/data/upperdir/var/lib/redis$

However this is weird as the redis config (raspberrypi4:/etc/redis$ nano redis.conf) is configured with

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

I also tried to manually start BGREWRITEAOF from redis-cli - no result and the appendonly.aof file remained the same

Only after deleting the appendonly.aof file manually and reboot the Chirpstack started to work again

That makes sense (sort of), it’s keeping the full history of everything that ran through Redis. I prefer not to use append persistence with Redis, and using the following as a starting point, and then tune based on requirements:

redis-server --save 300 1 --save 60 100 --appendonly no

However, I do not use ChirpStack’s gateway OS, and cannot give rationale for that being the (presumable) default.

All that being said, that is a TON (415 MB!) of updates going on in Redis. Unless this system has been running for a very long time, that does not seem normal.

1 Like

Should the

auto-aof-rewrite-percentage / min-size

not ensure that the filesize is controlled / minimazed to configured size?

Yes system was running for quite a while however it should not hang due to file size which is too big :slight_smile:

Maybe I have to switch of append persistence in Redis. I am not that familiar with Redis (need to readup) and maybe some other Chirpstack users have experience with it

I changed the appendonly to “no” // for my Chirpstack GW this is ok and now I avoid that the filesystem get full (and the Gateway stops working).

Tested it and all good. Thanks for your support

1 Like

Would you like to create a pull-request for this, to update the ChirpStack Gateway OS Redis config?