Multiple network servers

ChirpStack Network Server
1

For context, we tested out Chirpstack Simulator to send an uplink every 1 minute from 10k devices with 300 gateways. This led to our EC2 server that were hosting Chirpstack to be very sluggish, rendering it unusable, as it was consuming up to more than 90% of the server’s CPU.

So, does having multiple Chirpstack Network Servers (which is then connected to a single Chirpstack App Server) alleviate this problem? We were thinking of having multiple Chirpstacks on different servers, linked to a load balancer that will then distribute the uplink accordingly based on the server load.

If yes, how do we sync the data between the different servers? Do I need a dedicated server to host all the Postgresql and Redis?

1 Like
2

All the network servers must connect to the same redis for de-duplication and should consume gateway messages over the same backend for example MQTT. They also should share the same database. If using MQTT you should enable shared topics if your broker supports it, else you might find one of the NSs winning most of the time. On the AS side is a bit trickier. Load-Balancing GRPC is not as straight forward as HTTP/1.1 - you will need a DNS based service discovery which round-robins or randomly selects an address to return. GRPC connections are long live TCP.

3

I thought it was straight forward by just adding the network servers’ IP and port on the AS web UI?

We’re planning on deploying with AWS Elastic Beanstalk that will automatically create a new EC2 Chirpstack server whenever one is about to overload. Which then the load balancer can re-route the traffic to that new server. Something like this

future
future934×580 55 KB

By having multiple NSs, will that means we will have multiple IPs to work with? Even though we’re only using a single AS? Because we want to use Chirpstack REST API and would prefer to keep it to only one endpoint.

1 Like
4

The NS will be behind an LB, but the AS opens a steady connection to ONE NS, even if it’s running over an LB. It’s long lived TCP. If the NS starts the connection (most of the time) they will then land, in your case, on a single AS. You could have the AS behind an LB also, but the same limitations apply.

An if you are LoadBalancing the Gateway traffic, then it’s a job of the backend configured. If you are using MQTT you will need shared topics.

5

There is a configuration option on Application Server side:

# gRPC default resolver scheme.
#
# Set this to "dns" for enabling dns round-robin load balancing.
grpc_default_resolver_scheme="passthrough"
6

Yes, that’s the configuration. You service discovery (DNS in this case) needs to support that thou.

7

And is this something that might happen? Because at the moment we have hard-coded our current Chirpstack’s server address to utilize its REST API

8

The RESTful API is provided by the AS and it’s a grpc->HTTP/1.1 API. So the regular LB Mechanism for HTTP/1.1 applies. The NS has afaik, no RESTful API. I would suggest replicating your infra, as it is all virtual with AWS and add a test gateway. That way you dont have to trust, what this monkey is saying and you can even work a Disaster Recovery Plan or use your DRP to bring up the test infra. :slight_smile: