Can't add second network server

[almaz]

this is continuation of [Handling EU868 and US915 same time](http://handling EU868 and US915 same time).
I have chirpstack-network-server docker container configured to work on EU868 frequency plan.
So I need to run second chirpstack-network-server configured to another required frequency plan (US915).
For that I simply change port in chirpstack-network-server-us915.toml

  [network_server.api]
  # ip:port to bind the api server
  bind="0.0.0.0:7999"

And run second contailer:

docker run -d -p 127.0.0.1:7999:7999 --network="common" --name lora_server_US915 loraserver_chirpstack-network-server ./chirpstack-network-server -c chirpstack-network-server-us915.toml

Seems it starts as expected:

time="2020-04-24T23:34:56Z" level=info msg="starting ChirpStack Network Server" band=US915 docs="https://www.chirpstack.io/" net_id=000000 version=3.9.0-30-gbe5d7a1
time="2020-04-24T23:34:56Z" level=info msg="disabling all channels"
time="2020-04-24T23:34:56Z" level=info msg="enabling channels" channels="[0 1 2 3 4 5 6 7]"
time="2020-04-24T23:34:56Z" level=info msg="storage: setting up storage module"
time="2020-04-24T23:34:56Z" level=info msg="storage: setting up Redis client"
time="2020-04-24T23:34:56Z" level=info msg="storage: connecting to PostgreSQL"
time="2020-04-24T23:34:56Z" level=info msg="storage: applying PostgreSQL data migrations"
time="2020-04-24T23:34:56Z" level=info msg="storage: PostgreSQL data migrations applied" count=0
time="2020-04-24T23:34:56Z" level=info msg="gateway/mqtt: connecting to mqtt broker" server="tcp://lora_mosquitto:1883"
time="2020-04-24T23:34:56Z" level=info msg="no geolocation-server configured"
time="2020-04-24T23:34:56Z" level=info msg="configuring join-server client" ca_cert= server="http://lora_appserver:8003" tls_cert= tls_key=
time="2020-04-24T23:34:56Z" level=info msg="backend/gateway: connected to mqtt server"
time="2020-04-24T23:34:56Z" level=info msg="gateway/mqtt: subscribing to gateway event topic" qos=0 topic=gateway/+/event/+
time="2020-04-24T23:34:56Z" level=info msg="api: starting network-server api server" bind="0.0.0.0:7999" ca-cert= tls-cert= tls-key=
time="2020-04-24T23:34:56Z" level=info msg="starting downlink device-queue scheduler"
time="2020-04-24T23:34:56Z" level=info msg="starting multicast scheduler"

But when I try to create new network server in chirpstack-application-sever I get following error from application server:

Here is error messages in logs of chirpstack-application-server:

time="2020-04-24T23:45:32Z" level=warning msg="creating insecure network-server client" server="172.18.0.11:7999"
time="2020-04-24T23:45:32Z" level=info msg="finished client unary call" ctx_id=4d25e158-5d18-4afb-bfd5-7c7b231e5e27 error="rpc error: code = AlreadyExists desc = object already exists" grpc.code=AlreadyExists grpc.ctx_id=3d18f461-2677-4b69-9827-7b7479e7dddb grpc.duration=1.075654ms grpc.method=CreateRoutingProfile grpc.service=ns.NetworkServerService span.kind=client system=grpc
time="2020-04-24T23:45:32Z" level=info msg="finished unary call with code AlreadyExists" ctx_id=4d25e158-5d18-4afb-bfd5-7c7b231e5e27 error="rpc error: code = AlreadyExists desc = object already exists" grpc.code=AlreadyExists grpc.method=Create grpc.service=api.NetworkServerService grpc.start_time="2020-04-24T23:45:32Z" grpc.time_ms=4.21 peer.address="127.0.0.1:38928" span.kind=server system=grpc

and chirpstack-network-server:

time="2020-04-24T23:45:32Z" level=info msg="finished unary call with code AlreadyExists" ctx_id=3d18f461-2677-4b69-9827-7b7479e7dddb error="rpc error: code = AlreadyExists desc = object already exists" grpc.code=AlreadyExists grpc.method=CreateRoutingProfile grpc.service=ns.NetworkServerService grpc.start_time="2020-04-24T23:45:32Z" grpc.time_ms=0.571 peer.address="172.18.0.9:52744" span.kind=server system=grpc

Seems that lora-app-server compares something else than just nerwork name and server address?

[almaz]

Fixed by using separate database for each network server.
But now I’m facing following error messages:

time="2020-04-25T02:07:36Z" level=info msg="gateway/mqtt: gateway stats packet received" gateway_id=aa555a0000000005 stats_id=40a1ff96-8a59-4da1-b4ea-b2b31fc52706
time="2020-04-25T02:07:36Z" level=error msg="gateway: handle gateway stats error" ctx_id=40a1ff96-8a59-4da1-b4ea-b2b31fc52706 error="update gateway error: object does not exist"

or like that:

time="2020-04-25T02:14:24Z" level=info msg="gateway/mqtt: uplink frame received" gateway_id=aa555a0000000008 uplink_id=f03fc5a7-4355-480b-9afb-8488077bb429
time="2020-04-25T02:27:41Z" level=error msg="uplink: processing uplink frame error" ctx_id=fbc83e69-bbbf-4cb3-9a33-3875221b5fe5 error="run uplink response flow error: get rx1 frequency error: lorawan/band: unknown channel for frequency: 868500000"

Seems there are how network server reacts on packets from gateways registered in another network.

But the most unpleasant is that application server begins to ignore join requests without any error messages. The only reaction for joing request is log message of network server that uplink:

time="2020-04-25T02:14:24Z" level=info msg="gateway/mqtt: uplink frame received" gateway_id=aa555a0000000008 uplink_id=f03fc5a7-4355-480b-9afb-8488077bb429

After few join requests it sends join responce:

What is an issue of such stranges?
I think to modify source code of network server to subsribe only for MQTT messages from gateways registered for this network. Will it help?

[bconway]

Are you sending messages to different topic prefixes for each network server (configured by the gateway bridge)? You don’t want network servers seeing messages intended for the other.

[almaz]

Thank you!
Now it works like a charm!
Started understand how powerful is chirpstack!

[girts]

Hi,
Could you, please, explain how can you tell at gateway bridge that one message should be sent to US network server and the other to the EU network server? I am receiving all the messages on both network servers resulting in the same unwanted error log messages and I do not understand how can I filter the messages at gateway bridge. I would very appreciate if you could share the contents of your gateway-bridge.toml.

[bconway]

The gateway bridge configuration supports multiple servers in integration.mqtt.auth.generic, however I prefer/recommend different network bridge instances (either cloud or on the gateways) to point to the respective network servers. Debugging is a lot easier when you have clear demarcation in your network topology.

[girts]

Thank you for your answer, but I do not see what parameters in integration.mqtt.auth.generic would help me to filter uplink messages and send them to corresponding network server. Having another instance of gateway bridge would require also another instance of mosquitto as I understand.

[bconway]

You are correct, the currently supported approach is to send the traffic to both network servers and let the server drop the traffic that doesn’t apply.

May I ask why you are using the same gateway for both US and EU traffic? Presumably it is not located in both regions at the same time.