Can't create additional Network Server with docker-compose - Postgresql authentication fails

NoelzeN · January 30, 2020, 10:53pm

What I’m trying to achieve:
I have a setup using docker-compose that consist of Redis and Postgresql, Geolocation-, Network- and Application-Server. Now I have a network server for Region A but want to add an additional Network server for region B.

What I’ve done so far:
Since I understand that the initdb files in the postgres configuration folder are only read once, I created a new postgres user by connecting as postgres user and executing the commands

create role chirpstack_ns_in with login password 'PASSWORD';
create database chirpstack_ns_in with owner chirpstack_ns_in;

The user creation seems to have been successful, issuing the “\l” command I see the new Database and owner:

   Name       |      Owner       | Encoding |  Collate   |   Ctype    |   Access privileges   
------------------±-----------------±---------±-----------±-----------±----------------------
chirpstack_as | chirpstack_as | UTF8 | en_US.utf8 | en_US.utf8 |
chirpstack_ns | chirpstack_ns | UTF8 | en_US.utf8 | en_US.utf8 |
chirpstack_ns_in | chirpstack_ns_in | UTF8 | en_US.utf8 | en_US.utf8 |

After that, I added a new container in the docker-compose.yml file:

  chirpstack-network-server-in:
    image: chirpstack/chirpstack-network-server:3
    volumes:
      - ./configuration/chirpstack-network-server-in:/etc/chirpstack-network-server-in
    networks:
      ToolNetwork:
        ipv4_address: 172.23.0.26

and also copied the complete network server configuration folder. In the new folder, I updated the configuration file to match the different band and inserted the postgresql parameters:

[postgresql]
dsn=“postgres://chirpstack_ns_in:PASSWORD@postgresql/chirpstack_ns_in?sslmode=disable”

What the issue is:
When I now start the new container, I get an error that the connection to the postgresql database failed:

time=“2020-01-29T18:57:54Z” level=warning msg=“storage: ping PostgreSQL database error, will retry in 2s” error=“dial tcp 127.0.0.1:5432: connect: connection refused”

Actually I noticed https://github.com/brocaar/chirpstack-docker/issues/27 when debugging the described issue. I tried to login with the credentials which worked perfectly but then noticed also without password I could connect which makes it even more strange to me why I get this error.

Does anyone have an idea what is happening here or what I’m doing wrong? Thanks a lot for any hints.

NoelzeN · February 6, 2020, 1:20pm

No one else ever had this issue?

sagarpatel · February 7, 2020, 4:16am

Hii @NoelzeN

Have you update chirpstack-network-server-in.toml file Postgres connection
check docker-compose-env.yml file

# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
dsn="postgres://localhost/chirpstack_ns?sslmode=disable"

NoelzeN · February 7, 2020, 9:51am

Hi @sagarpatel

I did configure the .toml file in configuration folder, however I didn’t touch the docker-compose-env.yml file so far. I also adjusted the settings in that file now but still get the same error. Configuration (Network Server 1 is ok, Network Server 2 (IN) shows the postgres error):

docker-compose.yml

[...]
  chirpstack-network-server:
    image: chirpstack/chirpstack-network-server:3
    volumes:
      - ./configuration/chirpstack-network-server:/etc/chirpstack-network-server
    networks:
      ToolNetwork:
        ipv4_address: 172.23.0.21

  chirpstack-network-server-in:
    image: chirpstack/chirpstack-network-server:3
    volumes:
      - ./configuration/chirpstack-network-server-in:/etc/chirpstack-network-server-in
    networks:
      ToolNetwork:
        ipv4_address: 172.23.0.26

  postgresql:
    image: postgres:9.6-alpine
    volumes:
      - ./configuration/postgresql/initdb:/docker-entrypoint-initdb.d
      - postgresqldata:/var/lib/postgresql/data
    networks:
      ToolNetwork:
        ipv4_address: 172.23.0.24
[...]

docker-compose-env.yml

[...]
      chirpstack-network-server:
        image: chirpstack/chirpstack-network-server:3
        environment:
          - POSTGRESQL.DSN=postgres://chirpstack_ns:[PASSWD1]@postgresql/chirpstack_ns?sslmode=disable
          - REDIS.URL=redis://redis:6379
          - NETWORK_SERVER.BAND.NAME=EU_863_870
          - NETWORK_SERVER.GATEWAY.BACKEND.MQTT.SERVER=ssl://XXX:8883
          - JOIN_SERVER.DEFAULT.SERVER=http://chirpstack-application-server:8003
          - GEOLOCATION_SERVER.SERVER=chirpstack-geolocation-server:8005

  chirpstack-network-server-in:
    image: chirpstack/chirpstack-network-server:3
    environment:
      - POSTGRESQL.DSN=postgres://chirpstack_ns_in:[PASSWD2]@postgresql/chirpstack_ns_in?sslmode=disable
      - REDIS.URL=redis://redis:6379
      - NETWORK_SERVER.BAND.NAME=IN_865_867
      - NETWORK_SERVER.GATEWAY.BACKEND.MQTT.SERVER=ssl://XXX:8883
      - JOIN_SERVER.DEFAULT.SERVER=http://chirpstack-application-server:8003
      - GEOLOCATION_SERVER.SERVER=chirpstack-geolocation-server:8005

  postgresql:
    image: postgres:9.6-alpine
    volumes:
      - ./configuration/postgresql/initdb:/docker-entrypoint-initdb.d
      - postgresqldata:/var/lib/postgresql/data
[...]

configuration/chirpstack-network-server/chirpstack-network-server.toml

[...]
[postgresql]
dsn="postgres://chirpstack_ns:[PASSWD1]@postgresql/chirpstack_ns?sslmode=disable"
[...]

configuration/chirpstack-network-server-in/chirpstack-network-server-in.toml

[postgresql]
    dsn="postgres://chirpstack_ns_in:[PASSWD2]@postgresql/chirpstack_ns_in?sslmode=disable"

One thing I see is that when I manually connect to Postgres and open the Database, of course the chirpstack_ns_in DB is empty while the chirpstack_ns DB contains some Relations. Might that be an issue? Or are they only generated once the NS connects? Looking at the init scripts from the postgres configuration Folder I don’t see any command that would have created those Relations.

sagarpatel · February 7, 2020, 10:21am

@NoelzeN
Please set below Network Server 2 (IN) compose file setting and check

volumes:
          - ./configuration/chirpstack-network-server-in:/etc/chirpstack-network-server

NoelzeN · February 7, 2020, 10:57am

@sagarpatel

Thanks for the help so far. I adjusted the docker-compose.yml file as you suggested:

[...]
  chirpstack-network-server-in:
    image: chirpstack/chirpstack-network-server:3
    volumes:
      - ./configuration/chirpstack-network-server-in:/etc/chirpstack-network-server
    networks:
      ToolNetwork:
        ipv4_address: 172.23.0.26
[...]

Running docker-compose down and then docker-compose up, still getting the same error:

chirpstack-network-server-in_1   | time="2020-02-07T10:54:42Z" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp 127.0.0.1:5432: connect: connection refused"
chirpstack-network-server-in_1   | time="2020-02-07T10:54:44Z" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp 127.0.0.1:5432: connect: connection refused"
chirpstack-network-server-in_1   | time="2020-02-07T10:54:46Z" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp 127.0.0.1:5432: connect: connection refused"
chirpstack-network-server-in_1   | time="2020-02-07T10:54:48Z" level=warning msg="storage: ping PostgreSQL database error, will retry in 2s" error="dial tcp 127.0.0.1:5432: connect: connection refused"

I’ll try to export the relations from the NS DB to the NS_IN DB today afternoon to see whether that is an issue

sagarpatel · February 7, 2020, 10:58am

@NoelzeN,

can you please share chirpstack-network-server-in .toml file?

NoelzeN · February 7, 2020, 12:17pm

@sagarpatel FYI, I just copied the chirpstack_ns database over to a new database chirpstack_ns_in, granted privileges on tables and sequences, tested access by connecting via CLI, all works fine, but didn’t solve the issue. My config looks like this:

[xxx@xxx chirpstack-docker]$ cat configuration/chirpstack-network-server-in/chirpstack-network-server-in.toml 

# See https://www.chirpstack.io/network-server/install/config/ for a full
# configuration example and documentation.
#
# This file is for the EU868 band. See the examples/ folder for more
# configuration examples.

[postgresql]
dsn="postgres://chirpstack_ns_in:[PASSWD2]@postgresql/chirpstack_ns_in?sslmode=disable"

[redis]
url="redis://redis:6379"

[network_server]
net_id="000000"

[network_server.band]
name="IN_865_867"

[network_server.gateway.backend.mqtt]
server="ssl://XXX:8883"
username="xxx"
password="xxx"
client_id="lora_networkserver_in"

[join_server.default]
server="http://chirpstack-application-server:8003"

[geolocation_server]
server="chirpstack-geolocation-server:8005"

What confuses me is this part of the error message:

error="dial tcp 127.0.0.1:5432

It looks like it wants to connect to some localhost instance of postgres instead of the docker one?

NoelzeN · February 7, 2020, 1:17pm

@sagarpatel I think I solved the issue now
I renamed the config file from

configuration/chirpstack-network-server-in/chirpstack-network-server-in.toml

to

configuration/chirpstack-network-server-in/chirpstack-network-server.toml

and now with docker-compose up I don’t get the error anymore. Didn’t test whether the Network Server is also functional though but I hope so. Thanks a lot for your hints, renaming the volume path in the docker compose file brought me on the path that I simply also tried renaming the config file itself

Tom_Z · July 5, 2020, 8:48am

Thanks guys this is exactly what I needed to do and this method worked

ccall48 · December 22, 2020, 1:16pm

Thanks… I just wasted over 8 hours on this and config settings in a docker compose instance today all because of the name of this file…!!!