Hello,
in my testing I discovered an issue in the connection between Chirpstack and a Redis cluster in Kubernetes.
I use a headless service which should resolve all 6 of the present Redis instance IPs, so that the Chirpstack Pods can open their dedicated connections to the Redis instances directly.
I discovered, that Chirpstack always connects to the first 3 of them.
Additionally when a Redis pod restarts for some reason, Chirpstack is not resolving the IPs again and throws errors until the Chirpstack pod is restarted and the IPs are resolved again.
What is the preferred way to use Redis cluster in Kubernetes?
# Redis configuration.
[redis]
# Server address or addresses.
# Set multiple addresses when connecting to a cluster.
servers=[
"redis://:$CHIRPSTACK_REDIS_PASSWORD@chirpstackv4-redis-cluster-headless",
]
# Redis Cluster.
#
# Set this to true when the provided URLs are pointing to a Redis Cluster
# instance.
cluster=true
~ $ nslookup chirpstackv4-redis-cluster-headless
Server: 172.29.0.10
Address: 172.29.0.10:53
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.58.33
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.78.17
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.129.12
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.131.17
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.37.30
Name: chirpstackv4-redis-cluster-headless.siot-qs.svc.cluster.local
Address: 172.28.145.15
~ $ netstat -nputw | grep 172.28.58.33
tcp 0 0 172.28.106.125:58418 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58468 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58416 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58536 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58520 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58454 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58560 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58506 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58422 172.28.58.33:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:58474 172.28.58.33:6379 ESTABLISHED 1/chirpstack
~ $ netstat -nputw | grep 172.28.78.17
tcp 0 0 172.28.106.125:52654 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52696 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52656 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52658 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52710 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52740 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52682 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52666 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52726 172.28.78.17:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:52742 172.28.78.17:6379 ESTABLISHED 1/chirpstack
~ $ netstat -nputw | grep 172.28.129.12
tcp 0 0 172.28.106.125:47842 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47856 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47880 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47872 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47892 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47860 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47850 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47838 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47824 172.28.129.12:6379 ESTABLISHED 1/chirpstack
tcp 0 0 172.28.106.125:47884 172.28.129.12:6379 ESTABLISHED 1/chirpstack
~ $ netstat -nputw | grep 172.28.131.17
~ $ netstat -nputw | grep 172.28.37.30
~ $ netstat -nputw | grep 172.28.145.15
~ $
Thanks.