Production deployment recommendation?

[happidad]

I had a v3 chirpstack production deployment for a couple of years, without any issues. Kudos to @brocaar for his awesome work. I have a huge amount of trust in his implementation.

My v3 deployment was a single ec2 instance in AWS, running the application server, network server and a gateway bridge in separate containers. The postgresql DB and redis were running in their own servers. The system was rock solid and I never had any issues with any part of it.

I recently migrated to v4, running the chirpstack server and a gateway bridge running in docker containers on a single ec2 instance. I had accidentally under-provisioned the ec2 instance (a smaller instance than what I had used for my v3 instance). The system was working, but then started getting overloaded and it appears that the gateway bridge docker container was getting starved because the chirpstack container was very busy.

This leads me to conclusion that it’s not the best production solution to have the gateway bridge container running on the same instance as chirpstack container. It is very simple to separate these onto separate instances.

Adding additional gateway bridge instances seems easy, by just segmenting the gateways that connect to each gateway bridge. Otherwise, just increasing the ec2 instance size is a very easy way to increase performance.

What are the possible ways to improve throughput through the chirpstack server? Can you have multiple chirpstack servers behind a load balancer? Since the communication between chirpstack and the gateway bridge is mqtt, what happens if multiple chirpstacks receive the same mqtt message? Will all instances try to process the mqtt message, defeating the parallelism? Otherwise, just increasing the ec2 instance size is a very easy way to increase performance.

Thanks to everyone in this forum. I really appreciate the support!

[Liam_Philipp]

@happidad

what happens if multiple chirpstacks receive the same mqtt message? Will all instances try to process the mqtt message, defeating the parallelism?

Chirpstack uses what is called a “shared mqtt subscription” which means each MQTT event will only be read by one of the Chirpstack instances connected to it. This allows you to very easily spin up multiple Chirpstack instances pointing at the same broker and it will just work. Throw a load-balancer in front of them and you’re done.

I am also working on a production Chirpstack deployment right now so you might be able to learn a trick or two from my plan:

HA and redundancy for Chirpstack

Thanks for the help guys I’m still open to any improvements, but I’ll leave my plan here for anyone else researching Chirpstack redundancy and high availability in the future: General Network Flow [Chirpstack Redundancy Flow] Overview: Chirpstack depends on two databases: PostgreSQL for persistent storage and Redis for caching and temporary data. It uses the MQTT broker (EMQX) to communicate with gateways. HAProxy will use LetsEncrypt certificates from certbot to …

Another consideration with your setup, where gateway bridges are on the server rather than on the gateway, is what are the gateways sending over the air? Do you have a way to secure the UDP forwarder, or are you using basicstation with TLS?