Hi all,
I was assigned to help solve a deployment problem Chirpstack, at work. Chirpstack runs within Kubernetes, within an on-prem installation. The cluster has no ingress controller, thus the GWB service is exposed with NodePort. All gateways use the UDP Packet Forwarder.
Normally, the GW traffic goes through a Nginx proxy, which routes to the computers containing the Kubernetes cluster. But the problem persists, even if we point the GW directly to one of the VMs running Kubernetes.
Problem: if I have one instance of GWB, there is no problem with downlinks. If I have multiple instances of the GWB, downlinks do not really seem to work - they usually do not get logged within the LoRaWAN frames view and neither does the GW receive them. It is not a 100% failure, as it would work with more attempts.
Unfortunately, I do not own this system and so I cannot just collect log messages and their configuration files from it. My relationship with that team is also rather complex, so the best I can do would be to try things out myself and ask on the community forum. It can be said that I belong to the parent company of this team’s.
So I would like to ask for tips - what sort of settings for Kubernetes might be required to ensure that downlinks will work properly, when we have a cluster containing multiple GWB instances? I cannot directly make changes the shape of this deployment either. But if there is a strong reason to change it, I could make a suggestion.
Since this problem affects downlinks and I also saw errors getting logged about GWB not having the downlink frame within its cache for the Tx ACK it receives, I once suggested that they enable ClientIP sessionAffinity for the GWB service - to ensure that the TX_ACK goes back to the GWB which triggered the downlink. It seemed to work for a short period of time, until they reboot the services while testing whether the problem was truly taken care of. This change was rolled back every since.
I believe I have a deep understanding of LoRaWAN and the gateway backhaul protocols, and also have some understanding of how Chirpstack’s data flows work from studying its logs and code. But I am quite clueless when it comes to Kubernetes.
I would be grateful for any help and tips provided! Thank you in advance. Especially since there is very little information I have, to begin with.