Hi, we are running a Chirpstack v4.3.1 instance in a Kubernetes cluster (AKS in Azure).
The applications starts and connects to MQTT successfully, but calling the /health endpoint gets stuck for no obvious reasons:
- /metrics endpoint works well on the same port, so there are no networking issues
- no errors or relevant warnings in the log
- Postgres schemas are created, so there is probably no Postgres issue (we are using Azure Database for PostgreSQL)
- Redis is empty, so the Redis is probably the issue, but connecting the Redis instance (we are using Azure Cache for Redis) from a pod inside the cluster works fine (TLS enabled, no cluster).
How can we debug this issue deeper or get more information to help us fix it? (logLevel trace is already enabled)
Thanks!
Redis config:
[redis]
servers=[
"redis://$REDIS_PASSWORD@$REDIS_HOST:$REDIS_PORT",
]
tls_enabled=true
cluster=false
Full log:
2023-04-13T11:26:10.840360Z INFO chirpstack::cmd::root: Starting ChirpStack LoRaWAN Network Server version="4.3.1" docs="https://www.chirpstack.io/"
2023-04-13T11:26:10.840424Z INFO chirpstack::storage: Setting up PostgreSQL connection pool
2023-04-13T11:26:10.923461Z INFO chirpstack::storage: Applying schema migrations
2023-04-13T11:26:10.930026Z INFO chirpstack::storage: Setting up Redis client
2023-04-13T11:26:11.394672Z INFO chirpstack::region: Setting up regions
2023-04-13T11:26:11.394755Z INFO setup{common_name=EU868 region_id=eu868}: chirpstack::region: Configuring region
2023-04-13T11:26:11.394872Z INFO chirpstack::backend::joinserver: Setting up Join Server clients
2023-04-13T11:26:11.394878Z INFO chirpstack::backend::roaming: Setting up roaming clients
2023-04-13T11:26:11.394883Z INFO chirpstack::adr: Setting up adr algorithms
2023-04-13T11:26:11.394914Z INFO chirpstack::integration: Setting up global integrations
2023-04-13T11:26:11.394920Z INFO chirpstack::integration::redis: Initializing Redis integration
2023-04-13T11:26:11.394926Z INFO chirpstack::gateway::backend: Setting up gateway backends for the different regions
2023-04-13T11:26:11.394931Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=eu868 region_common_name=EU868
2023-04-13T11:26:11.395509Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_config_id=eu868 server_uri=tcp://emqx-listeners.infra:1883 clean_session=false client_id=6f1b23245d1f1b4b
2023-04-13T11:26:11.528331Z INFO chirpstack::downlink: Setting up Class-B/C scheduler loop
2023-04-13T11:26:11.528510Z INFO chirpstack::downlink: Setting up multicast scheduler loop
2023-04-13T11:26:11.527924Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT consumer loop
2023-04-13T11:26:11.528700Z INFO chirpstack::api: Setting up API interface bind=0.0.0.0:8080
2023-04-13T11:26:11.529023Z WARN chirpstack::api::backend: Backend interfaces API is disabled
2023-04-13T11:26:11.529106Z INFO chirpstack::api::monitoring: Setting up monitoring endpoint bind=0.0.0.0:8090
2023-04-13T11:26:11.529242Z INFO chirpstack::gateway::backend::mqtt: Connected to MQTT broker region_config_id=eu868
2023-04-13T11:26:11.529418Z INFO chirpstack::gateway::backend::mqtt: Subscribing to gateway event topic region_config_id=eu868 event_topic=chirpstack/eu868/gateway/+/event/+