Multiple Network servers with similar (but not same) bands and single Application server

Hello.
First af all, thanks to all Chirpstack Application Server authors and contributors for their great work ).

I have a question regarding to multiple network servers and single application server setup. I searched forum and found some topics about such configuration but unfourtunately everything not works as I expected. I see interference (in how end node served by application server via 2 network servers) when both gateways turned on and connected to chirpstack during working with single end-node.
My setup:

So I installed all Chirpstack components to Ubuntu18.04 server (all prerequisites were installed of most recent versions of course) from repository. All versions of Chirpstack components are on picture. I disabled default gateway bridge and netwok server systemd services and then created instead 2 instances of systemd services for each with manually specified config file path.
Networking server config for _Generic is default EU868 config + additional 867.xMHz frequencies. On the other side _UA config is based on 3 default EU868 frequencies and have specific frequencies set for DR2, forced RX1 window only and so on - local RF regulations specific.
Path of RF frame arrival to Application server is separated (as far as I can see) - it goes via separate GW bridge each running at own port, specifically prefixed MQTT broker topic and then goes to its networking server.
In Application server WEB API I added 2 Networking servers which also separated by port.
Then I created 2 service profiles and 2 device profiles, each is same but binded to _UA or _Generic Networking server.
Then I created application binded to _UA service profile and added device binded to _UA Networking server.
As I can see creating any instance of service profile, device profile, gateway and so on is accompanied with selecting Networking server. So my thought was that if end-node in the end is binded to _UA networking server via all those profiles then it should be served only by _UA network server and Microtik gateway in my case.
However it seems it is not.

At first stage if I turn off Mikrotik gateway and leave only Kona Micro and device sends Join requests - they are refused by _Generic Networking server (level=error msg=“uplink: processing uplink frame error” ctx_id=064823fd-ec84-493c-9745-23f75bd3b1a2 error=“get device error: object does not exist”
). _EU networking server receives nothing in logs. When I turn on Mikrotik - device authorizes, but then device does not served properly. Various frustrating behaviour may occur. For example gateways does not send ACK for confirmed uplinks (seen in radio spectrum and from gateways live frames) or both these Networking servers with different configuration try to operate. If only KonaMicro left turned on _Generic Networking server adds additional frequencies (867.xMHz) which are set in its configuration but should not be applied to device in application binded to _UA Networking server!. If only Mikrotik will be left - _UA Networking server will set default channels mask 0x07. In general normal device operation is seen when only one gateway turned on and seems like then each gateway operated by corresponding Network server with its specific configuration.

This question risen primarily from misunderstanding how Application server works with multiple Networking servers after setting up and experimenting with such dual Networking server setup. We need to set up multiple Networking server instances for our needs in future but after testing setup I cant understand the logic behind multiple Networkink servers operating with single Application server. This setup was intentionally made such that regional standard is same but specific configurations vary. However it possible if both gateways operated near but in different countries, where overall regional standard is same but each country/region has it’s own region specifics.
So, plese, can anybody explain or point me to answers for following questions:

  • why do I see such behaviour?
  • how does Application server differentiates between Networking servers?
  • what is the reasoning for binding service/device profiles to specific Networking server and what was the initial aim of such binding?
  • what may be wrong in my setup?

P.S. I will also attach configuration files for Chirpstack components.
https://drive.google.com/drive/folders/1uUO-fJlKgztLh31hwyWfbMlQHOejJ2en?usp=sharing

In general, the idea behind multiple network-servers and a single application server is to manage multiple regions within a single web-interface and also have the network-server close to the gateways (to limit the roundtrip time between the ns and gateways).

How this is intended to use is that every network-server instance has its own pool of gateways. Each pool of gateways would then have its own MQTT broker, or would have their own prefix.

E.g. for US915 it would be us915/gateway/...., EU868 eu868/gateway/.... and so on.

how does Application server differentiates between Networking servers?

With the above setup, received data is automatically routed to the correct NS responsible for handing the gateway data.

what is the reasoning for binding service/device profiles to specific Networking server and what was the initial aim of such binding?

Since this information is stored in the NS database, the AS database only contains a kind of “pointer record”.

1 Like

Thanks for your reply.

This is kind what I expected with such hierarchy and from information in other topics about multiple Networking servers connected to single Application service.
However, why does my setup doesn’t work as I expected? I expected that if I have 2 “separate” Networking servers with their gateways (and gateway bridges) then even if gateways located at the neighborhood - only single Networking server should operate with device binded to exact Network server (via service and device profile).

The setup described in my first post. There is only one end node in that setup and 2 gateways physically.

I have 2 Networking server instances binded each to own database with similar EU868 regional frequency plan configuration for test (which are not exact same, one have additional frequencies).
A have also 2 gateway bridges on different ports listening 2 different gateways. Paths between gateway bridges and Networking servers also separated with different MQTT topic prefix.

But device operates normally only if one of gateways is turned off. When device does not joined yet (OTAA used) - Join accepted only from Mikrotik gateway (which is as expected) to _UA Network server. Kona Micro forwards Join requests but _Generic Network server refuzes (object does not exixst in log) them.
However, when authorization stage was passed by device it begins to operate with not expected behaviour. Device will proceed to work “normally” via any gateway if only one left turned on. Depending of which gateway actually left to work - corresponding Network server specific frequencies and channel mask will be updated in the end-node(i can see it in device debug log). When both gateways turned on - test end-node even does not receive ACK from Network server properly and various similar improper behaviour occurs. It looks like both networking server try to operate.

I described more in my first post. Also config files attached to Google Drive and I put link at the end of first post.

Any ideas? Please help to understand where is an error )