Basics Station behind reverse-proxy with TLS termination

ChirpStack Gateway Bridge
1

Hello!

I’m having some issues trying to set up a server-side GatewayBridge service for listening BasicsStation LNS protocol.

I have an NGINX Ingress controller which does the TLS termination so all TLS certificates are managed by a cert-manager controller using Let’sEncrypt.
Following the de-facto configuration for all of out services I configured GatewayBridge to work without TLS like this:

[general]
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4

# Log in JSON format.
log_json=false

# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false


# Filters.
#
# These can be used to filter LoRaWAN frames to reduce bandwith usage between
# the gateway and ChirpStack Gateway Bridge. Depending the used backend, filtering
# will be performed by the Packet Forwarder or ChirpStack Gateway Bridge.
[filters]

# NetIDs filters.
#
# The configured NetIDs will be used to filter uplink data frames.
# When left blank, no filtering will be performed on NetIDs.
#
# Example:
# net_ids=[
#   "000000",
#   "000001",
# ]
net_ids=[
]

# JoinEUI filters.
#
# The configured JoinEUI ranges will be used to filter join-requests.
# When left blank, no filtering will be performed on JoinEUIs.
#
# Example:
# join_euis=[
#   ["0000000000000000", "00000000000000ff"],
#   ["000000000000ff00", "000000000000ffff"],
# ]
join_euis=[
]


# Gateway backend configuration.
[backend]

# Backend type.
#
# Valid options are:
#   * semtech_udp
#   * concentratord
#   * basic_station
type="basic_station"
  # Basic Station backend.
  [backend.basic_station]

  # ip:port to bind the Websocket listener to.
  bind=":3001"

  # TLS certificate and key files.
  #
  # When set, the websocket listener will use TLS to secure the connections
  # between the gateways and ChirpStack Gateway Bridge (optional).
  tls_cert=""
  tls_key=""

  # TLS CA certificate.
  #
  # When configured, ChirpStack Gateway Bridge will validate that the client
  # certificate of the gateway has been signed by this CA certificate.
  ca_cert=""

  # Stats interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge forwards
  # the uplink / downlink statistics.
  stats_interval="30s"

  # Ping interval.
  ping_interval="1m0s"

  # Timesync interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge sends
  # a timesync request to the gateway. Setting this to 0 disables sending
  # timesync requests.
  timesync_interval="0s"

  # Read timeout.
  #
  # This interval must be greater than the configured ping interval.
  read_timeout="1m5s"

  # Write timeout.
  write_timeout="1s"

  # Region.
  #
  # Please refer to the LoRaWAN Regional Parameters specification
  # for the complete list of common region names.
  region="AU915"

  # Minimal frequency (Hz).
  frequency_min=915200000

  # Maximum frequency (Hz).
  frequency_max=923300000

  # Concentrator configuration.
  #
  # This section contains the configuration for the SX1301 concentrator chips.
  # Example:
  [[backend.basic_station.concentrators]]
  
    # Multi-SF channel configuration.
    [backend.basic_station.concentrators.multi_sf]
  
    # Frequencies (Hz).
    frequencies=[
      915200000,
      915400000,
      915600000,
      915800000,
      916000000,
      916200000,
      916400000,
      916600000,
    ]
  
    # LoRa STD channel.
    [backend.basic_station.concentrators.lora_std]
  
    # Frequency (Hz).
    frequency=915900000
  
    # Bandwidth (Hz).
    bandwidth=500000
  
    # Spreading factor.
    spreading_factor=8
  
    # FSK channel.
    [backend.basic_station.concentrators.fsk]
  
    # Frequency (Hz).
    frequency=921800000


# Integration configuration.
[integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf:  Protobuf encoding
# * json:      JSON encoding (for debugging)
marshaler="protobuf"

  # MQTT integration configuration.
  [integration.mqtt]
  # Event topic template.
  event_topic_template="au915_0/gateway/{{ .GatewayID }}/event/{{ .EventType }}"

  # State topic template.
  #
  # States are sent by the gateway as retained MQTT messages (by default)
  # so that the last message will be stored by the MQTT broker. When set to
  # a blank string, this feature will be disabled. This feature is only
  # supported when using the generic authentication type.
  state_topic_template="au915_0/gateway/{{ .GatewayID }}/state/{{ .StateType }}"

  # Command topic template.
  command_topic_template="au915_0/gateway/{{ .GatewayID }}/command/#"

  # State retained.
  #
  # By default this value is set to true and states are published as retained
  # MQTT messages. Setting this to false means that states will not be retained
  # by the MQTT broker.
  state_retained=true

  # Keep alive will set the amount of time (in seconds) that the client should
  # wait before sending a PING request to the broker. This will allow the client
  # to know that a connection has not been lost with the server.
  # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
  keep_alive="30s"

  # Maximum interval that will be waited between reconnection attempts when connection is lost.
  # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
  max_reconnect_interval="1m0s"

  # Terminate on connect error.
  #
  # When set to true, instead of re-trying to connect, the ChirpStack Gateway Bridge
  # process will be terminated on a connection error.
  terminate_on_connect_error=false


  # MQTT authentication.
  [integration.mqtt.auth]
  # Type defines the MQTT authentication type to use.
  #
  # Set this to the name of one of the sections below.
  type="generic"

    # Generic MQTT authentication.
    [integration.mqtt.auth.generic]
    # MQTT servers.
    #
    # Configure one or multiple MQTT server to connect to. Each item must be in
    # the following format: scheme://host:port where scheme is tcp, ssl or ws.
    servers=[
      "tcp://redacted:1883",
    ]

    # Connect with the given username (optional)
    username="redacted"

    # Connect with the given password (optional)
    password="redacted"

    # Quality of service level
    #
    # 0: at most once
    # 1: at least once
    # 2: exactly once
    #
    # Note: an increase of this value will decrease the performance.
    # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
    qos=0

    # Clean session
    #
    # Set the "clean session" flag in the connect message when this client
    # connects to an MQTT broker. By setting this flag you are indicating
    # that no messages saved by the broker for this client should be delivered.
    clean_session=true

    # Client ID
    #
    # Set the client id to be used by this client when connecting to the MQTT
    # broker. A client id must be no longer than 23 characters. When left blank,
    # a random id will be generated. This requires clean_session=true.
    client_id="BRIDGE-AU915_0"

    # CA certificate file (optional)
    #
    # Use this when setting up a secure connection (when server uses ssl://...)
    # but the certificate used by the server is not trusted by any CA certificate
    # on the server (e.g. when self generated).
    ca_cert=""

    # mqtt TLS certificate file (optional)
    tls_cert=""

    # mqtt TLS key file (optional)
    tls_key=""


# Metrics configuration.
[metrics]

  # Metrics stored in Prometheus.
  #
  # These metrics expose information about the state of the ChirpStack Gateway Bridge
  # instance like number of messages processed, number of function calls, etc.
  [metrics.prometheus]
  # Expose Prometheus metrics endpoint.
  endpoint_enabled=false

  # The ip:port to bind the Prometheus metrics server to for serving the
  # metrics endpoint.
  bind=""

GW Service is listening in 3001 port so I made a deployment file where the pod exposes TCP port 3001:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chirpstack-gateway-bridge-au915-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: chirpstack-gb-au915
  template:
    metadata:
      labels:
        app: chirpstack-gb-au915
    spec:
      volumes:
      - name: chirpstack-gateway-bridge-au915-config
        configMap:
          name: chirpstack4-gb-config
      containers:
      - name: chirpstack-gateway-bridge-au915-cnt
        image: chirpstack/chirpstack-gateway-bridge:4
        imagePullPolicy: Always
        command: ["/usr/bin/chirpstack-gateway-bridge"]
        args: ["-c", "/etc/chirpstack-gateway-bridge/chirpstack-gateway-bridge-au915.toml"]
        ports:
        - containerPort: 3001
          name: basics
        volumeMounts:
        - name: chirpstack-gateway-bridge-au915-config
          mountPath: /etc/chirpstack-gateway-bridge
        resources:
          requests:
            cpu: 500m
            memory: 500Mi
          limits:
            cpu: 500m
            memory: 500Mi
---
apiVersion: v1
kind: Service
metadata:
  name: chirpstack-gateway-bridge-au915-service
spec:
  type: ClusterIP
  selector:
    app: chirpstack-gb-au915
  ports:
  - name: basics
    protocol: TCP
    port: 3001
    targetPort: 3001

This way of deploying services is a common practice in our company and we have many services working.
Following this scheme, I configured the Ingress to route incoming traffic (80 and 443 ports) to the corresponding service:

- host: lns.chamanagro.ar
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: chirpstack-gateway-bridge-au915-service
                port:
                  number: 3001

Now the issue is that I can´t make my indoor RAK gateway work using BasicsStation.
This are the GatewayBridge service logs:

time="2024-10-02T14:12:09.694144395Z" level=info msg="backend/basicstation: router-info request received" gateway_id=ac1f09fffe0647fa remote_addr="10.36.1.8:41618" router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa"
time="2024-10-02T14:12:21.321187525Z" level=info msg="backend/basicstation: router-info request received" gateway_id=ac1f09fffe0647fa remote_addr="10.36.1.8:40980" router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa"
time="2024-10-02T14:12:33.053466759Z" level=info msg="backend/basicstation: router-info request received" gateway_id=ac1f09fffe0647fa remote_addr="10.36.1.8:42972" router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa"
time="2024-10-02T14:12:45.010390166Z" level=info msg="backend/basicstation: router-info request received" gateway_id=ac1f09fffe0647fa remote_addr="10.36.1.8:55796" router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa"
time="2024-10-02T14:12:57.674558423Z" level=info msg="backend/basicstation: router-info request received" gateway_id=ac1f09fffe0647fa remote_addr="10.36.1.8:47186" router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa"

And the gateway logs:

Wed Oct  2 14:13:57 2024 user.err basicstation[2419]: [AIO:ERRO] [5] WS upgrade failed with HTTP status code: 400

And the gateway configuration:

image
image716×262 9.36 KB

NOTE:
The “trust” certificate is NOT the server certificate because it changes every 60 days… I used the CA certificate.

The error seems to be in the “upgrade” header but I think I am missing something. You can also notice that in the GatewayBridge logs in the “router_uri” field the scheme of the URI is “ws” and not “wss”. Thats wrong but it may have something to do with the fact that the gw bridge is configured without any TLS. (The TLS termination HAS to be done at the reverse proxy level because of Let’sEnctrypt renewal process)

This post is a mess but I am lost.

Thanks in advance

2

I’ve tried using WS scheme without TLS and the results are similar:

Wed Oct  2 17:03:52 2024 user.info basicstation[13434]: [TCE:INFO] Connecting to INFOS: ws://lns.chamanagro.ar:80
Wed Oct  2 17:03:52 2024 user.err basicstation[13434]: [AIO:ERRO] [5] WS upgrade failed with HTTP status code: 308
Wed Oct  2 17:03:52 2024 user.info basicstation[13434]: [TCE:INFO] INFOS reconnect backoff 30s (retry 3)
3

I had serious difficulty using Basics Station TLS with the reverse-proxy Traefik as well, eventually just gave up.

Sorry I can’t offer any help but if you figure this out I would be very interested.

4

My issue may be different, because if I try not to use TLS I still get a “upgrade” header error.

I have several WebSocket services running behind this same reverse proxy and they don’t have any issue regarding the upgrade headers (which is part of the websocket connection)

5

This error comes from the router-uri returned by the endpoint /router-info of the chirpstack-gateway-bridge. As you use a reverse-proxy, it needs to connect through wss:// from Internet but the bridge works with ws:// inside your network (it doesn’t know it uses TLS). You can see it in your chirpstack-gateway-bridge logs: router_uri="ws://lns.chamanagro.ar:443/gateway/ac1f09fffe0647fa".

What it does is that your gateway successfully established a connection with your bridge but when it gets the router-info with this uri, it will try to continue the communication by using it. And since, it will not work with ws:// you will get an error.

I had the same problem and it can only be resolved by modifying the way the scheme is handled by the bridge. I proposed a pull-request to the official repository. Hopefully, it will get accepted and pushed soon. You can try and say if it also resolve your problem. I had exactly the same thing as you but with traefik.

1 Like
6

Thank you @bastienvty for this PR

EDIT:

I’ve made a fork and tested your changes… it worked as intended! Thanks!

1 Like