NS Database connection times out randomly


I am running Chirpstack AS and NS servers in docker containers on an azure VM and these connect to hosted PostgreSQL database also in azure.

Everything runs perfectly until once every 6 or 7 days on average, the NS seems to loose connection with the database. The AS continues to function perfectly but I need to restart the NS container to establish connection again.

It seems like a single post to the database causes the error (i presume its random, below is the latest one) and then subsequently every database connection fails from the network server.

time=“2021-08-23T03:36:41.017769356Z” level=error msg=“class-b / class-c scheduler error” ctx_id=a038718a-ba99-4d4a-adb0-fb15dabcc255 error=“storage: begin transaction error: write tcp>XXX.XXX.XXX.XXX:5432: write: connection reset by peer”

Subsequent error example:

time=“2021-08-23T03:36:44.253322762Z” level=error msg=“uplink: processing uplink frame error” ctx_id=7fca74c0-aec3-46b4-a42e-e7816b0a046e error=“run uplink response flow error: get next device-queue item error: select error: write tcp>XXX.XX.XXX.XXX:5432: write: connection reset by peer”

I have checked the logs of the PostgreSQL server and there is no error at the time the NS gets its first error but post that timestamp there are no further connections from the NS logged in the database at all.

Please how can I troubleshoot further?

Thank you

Chirpstack has to be updated to use lib/pq v1.9 or higher. Can you file a bug with chirpstack as and ns?

I have updated lib/pq and will issue a bugfix release soon.

Thank you for the update. Fixing this issue will help us greatly.