Scaling Up and MQTT topics

Hi there

Maybe a “stupid” question.
When scaling up the solution, for example, having 2…3…N dockers running NS/AS/GB. What is the recommended use of MQTT topics ?

I mean, all subscribers will receive events. I can use $share on topics to let MQTT manage the situation and deliver the message to only one subscriber or there is an internal mechanism on CS that manage it ?

Thanks

ChirpStack NS will do its own locking/deduping, it is fine to run multiple containers without needing to partition MQTT topics.

The NS is the one consuming MQTT. I dont think you can run the GB in an N:M Gateways, as they dont seem to have locking mechanism, but maybe @brocaar can clarify that. Running them directly on the GW seems to be the preferred option. NS <-> AS is gRPC (HTTP2) so you are fine running a bunch of them.

We are running 3 replicas for NS/AS and GB on the Gateways. I need to have a better look at the metrics to see if they get equally loaded over time or if we should move the topics to “$share”, at the moment it’s mostly providing a “rolling-update” mechanism or in case a Node (VM) fails.
If we see a load-increase is probably the AS, as is does a lot of things (running decoders, integrations, join-server), but without Numbers right now, it’s a guess. GB/NS/AS provide prometheus endpoints…we havent had the time to do proper dashboards yet.
@bconway you dont happend to have some grafana-dashboards laying around getting dusty in a repo somewhere at work, do you? :slight_smile:

The preferred way is running the ChirpStack Gateway Bridge on the gateway. One thing that some MQTT brokers support is so called “consumer groups”, in which the MQTT broker will round-robin messages across the consumers within a single group :slight_smile: Else ChirpStack Network Server will handle the de-duplication for you, but it means some additional load as indeed each instance will get the same messages from the MQTT broker.

1 Like

Hi @brocaar

Exactly, the question is regarding not using GB on gateways.
Is there any counter effect using consumer groups on NS subscribing topics to avoid this overhead on it ?

Thanks very much

Once you start using “shared” topics the load gets balanced out and the network-instances record just the mqtt packages received.

You can see one instance “winning” most of the time. After “shared” it balances out.

That leads to an application-server getting the most load. With “shared” topics the load is balanced out, most of the time. Due to instance starting order it may happen that (in our case) two of the instances get the most traffic. It should solvable with a better configured/tuned LB in-front of the AS-instances.

The GW definitely needs to use “shared” topics, as running many of them will get them all receiving the downlink-messages. Afaik the GW has no locking mechanism. Might test it in 2-3 Weeks