Chirpstack services don't connect to Postgres anymore

EmiHR · May 3, 2021, 12:14pm

Hi all

I’ve been using a RAK gateway for a while now and everything was going excellent, until today. I couldn’t enter to the app-server, and my nodes didn’t join, so I took a peek into the services status. It showed me that psql wasn’t running, so I restarted everything, to no avail.

These are the status of all services:

● chirpstack-application-server.service - ChirpStack Application Server
   Loaded: loaded (/lib/systemd/system/chirpstack-application-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-05-03 09:00:22 -03; 1min 14s ago
     Docs: https://www.chirpstack.io/
 Main PID: 3671 (chirpstack-appl)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/chirpstack-application-server.service
           └─3671 /usr/bin/chirpstack-application-server

May 03 09:01:18 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:18-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:20 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:20-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:22 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:22-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:24 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:24-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:26 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:26-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:28 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:28-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:30 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:30-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:32 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:32-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:34 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:34-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:01:36 rak-gateway chirpstack-application-server[3671]: time="2021-05-03T09:01:36-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
root@rak-gateway:/home/pi# service chirpstack-network-server status
● chirpstack-network-server.service - ChirpStack Network Server
   Loaded: loaded (/lib/systemd/system/chirpstack-network-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-05-03 09:00:54 -03; 1min 33s ago
     Docs: https://www.chirpstack.io/
 Main PID: 3773 (chirpstack-netw)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/chirpstack-network-server.service
           └─3773 /usr/bin/chirpstack-network-server

May 03 09:02:08 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:08-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:10 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:10-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:12 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:12-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:14 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:14-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:16 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:16-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:18 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:18-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:20 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:20-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:22 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:22-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:24 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:24-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
May 03 09:02:26 rak-gateway chirpstack-network-server[3773]: time="2021-05-03T09:02:26-03:00" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp [::1]:5432: connect: connection refused"
root@rak-gateway:/home/pi# service postgresql status
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2021-05-03 08:44:59 -03; 17min ago
  Process: 851 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 851 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/postgresql.service

May 03 08:44:59 rak-gateway systemd[1]: Starting PostgreSQL RDBMS...
May 03 08:44:59 rak-gateway systemd[1]: Started PostgreSQL RDBMS.

Steps done so far:
.

Reset the gateway
Restarted all services (chirpstack and postgresql)
Checked psql dsn in both .toml files

Any ideas?

Thank you in advance.

cls02 · May 3, 2021, 4:06pm

Similar problem. Clean install of gateway-os-full on raspberrypi-4

monit summary gives all green
/var/log/messages full of postgresql errors

Obviously cannot connect to Application Server

Casper

eugenev · May 3, 2021, 4:50pm

inspect postgresql logs, why it can’t be start.

EmiHR · May 3, 2021, 5:04pm

So, after searching what may be causing the issue I found out I had around 12GB of log files. Those filled my SD card to the brim and no service would start correctly because of that. Removed those logs, rebooted, and problem solved. Anyways, thanks to the people that responded.

petertoome · July 6, 2023, 3:24am

I have just seen a similar error condition after upgrading from postgres 9.6 to 13. After the upgrade, Chirpstack failed with a message that it was unable to connect to postgres.
After hunting around I found that the postgres 13 config file has the default port as 5433 instead of 5432 which was used previously.
So if you upgrade postgres or start with a new install and version 13, make sure you set the port back to 5432 in the postgres config