Problem with Versions > 4.3.2

Hi,
we have an Version 4.2.x installation that we wanted to upgrade (it was never a 3.x). We have upgraded to 4.3.2 which works fine but any version later than that do not work. I just see this log output:

> 2024-04-09T14:22:56.946787Z  INFO chirpstack::cmd::root: Starting ChirpStack LoRaWAN Network Server version="4.4.0" docs="https://www.chirpstack.io/"
> 2024-04-09T14:22:56.946819Z  INFO chirpstack::storage: Setting up PostgreSQL connection pool
> 2024-04-09T14:22:56.961413Z  INFO chirpstack::storage: Applying schema migrations
> 2024-04-09T14:22:56.962437Z  INFO chirpstack::storage: Setting up Redis client
> 2024-04-09T14:22:56.986908Z  INFO chirpstack::region: Setting up regions
> 2024-04-09T14:22:56.986934Z  INFO chirpstack::backend::joinserver: Setting up Join Server clients
> 2024-04-09T14:22:56.986938Z  INFO chirpstack::backend::roaming: Setting up roaming clients
> 2024-04-09T14:22:56.986942Z  INFO chirpstack::adr: Setting up adr algorithms
> 2024-04-09T14:22:56.986968Z  INFO chirpstack::integration: Setting up global integrations
> 2024-04-09T14:22:56.986973Z  INFO chirpstack::integration::redis: Initializing Redis integration
> 2024-04-09T14:22:56.986981Z  INFO chirpstack::integration::mqtt: Initializing MQTT integration
> 2024-04-09T14:22:56.989974Z  INFO chirpstack::integration::mqtt: Connecting to MQTT broker server_uri=tcp://chirpstack-mosquitto-mosquitto:1883/ client_id=9954eaa186efb525 clean_session=false
> 2024-04-09T14:22:57.100855Z  INFO chirpstack::integration::mqtt: Connected to MQTT broker
> 2024-04-09T14:22:57.100987Z  INFO chirpstack::gateway::backend: Setting up gateway backends for the different regions
> 2024-04-09T14:22:57.101007Z  INFO chirpstack::downlink: Setting up Class-B/C scheduler loop
> 2024-04-09T14:22:57.101013Z  INFO chirpstack::downlink: Setting up multicast scheduler loop
> 2024-04-09T14:22:57.101011Z  INFO chirpstack::integration::mqtt: Starting MQTT consumer loop
> 2024-04-09T14:22:57.101014Z  INFO chirpstack::integration::mqtt: Subscribing to command topic command_topic=application/+/device/+/command/+
> 2024-04-09T14:22:57.101020Z  INFO chirpstack::api: Setting up API interface bind=0.0.0.0:8080
> 2024-04-09T14:22:57.101049Z  INFO chirpstack::api::backend: Setting up backend interfaces API bind=0.0.0.0:8090
> 2024-04-09T14:22:57.101100Z  INFO chirpstack::api::monitoring: Setting up monitoring endpoint bind=0.0.0.0:8081

No error, the server just does not process any traffic. Looking at the change-log I cannot spot anything that would explain this.

Would love to get some hint on what could be wrong.

BG, Thomas

Have you upgraded chirpstack-gateway-bridge as well?

Need to doublecheck all toml files as the toml files may not keep your changes.

I have upgraded 3-4 times from v4.0 to 4.7 and it still works after some minor comfiguration changes.

Yes, I’m even running latest GW bridge and latest API Service. I just cannot upgrade the CS service itself. The problem is, I cannot find any config changes that are due. If I read the documentation for 4.7 it looks to me like I have everything in place. The topic names have not changed.

Here is my config for reference:

    # Logging.
    [logging]

      # Log level.
      #
      # Options are: trace, debug, info, warn error.
      level="info"

    # PostgreSQL configuration.
    [postgresql]

      # PostgreSQL DSN.
      #
      # Format example: postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>/<DATABASE>?sslmode=<SSLMODE>.
      #
      # SSL mode options:
      #  * disable - no SSL
      #  * require - Always SSL (skip verification)
      #  * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
      #  * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
      # dsn="postgresql://chirpstack:chirpstack@localhost/chirpstack?sslmode=disable"
      dsn="$POSTGRESQL_DSN"

      # Max open connections.
      #
      # This sets the max. number of open connections that are allowed in the
      # PostgreSQL connection pool.
      max_open_connections=10

      # Min idle connections.
      #
      # This sets the min. number of idle connections in the PostgreSQL connection
      # pool (0 = equal to max_open_connections).
      min_idle_connections=0

    # Redis configuration.
    [redis]

      # Server address or addresses.
      #
      # Set multiple addresses when connecting to a cluster.
      servers=["{{ join "\",\"" .Values.chirpstack.redis.servers }}"]

      # Redis Cluster.
      #
      # Set this to true when the provided URLs are pointing to a Redis Cluster
      # instance.
      cluster=false

      # Key prefix.
      #
      # A key prefix can be used to avoid key collisions when multiple deployments
      # are using the same Redis database and it is not possible to separate
      # keys by database index (e.g. when using Redis Cluster, which does not
      # support multiple databases).
      key_prefix=""

      # Max open connections.
      #
      # This sets the max. number of open connections that are allowed in the
      # Redis connection pool.
      max_open_connections=100

      # Min idle connections.
      #
      # This sets the min. number of idle connections in the Redis connection
      # pool (0 = equal to max_open_connections).
      min_idle_connections=0


    # API interface configuration.
    [api]

      # interface:port to bind the API interface to.
      bind="0.0.0.0:8080"

      # Secret.
      #
      # This secret is used for generating login and API tokens, make sure this
      # is never exposed. Changing this secret will invalidate all login and API
      # tokens. The following command can be used to generate a random secret:
      #   openssl rand -base64 32
      secret=""


    # Global gateway configuration.
    # Please note that backend configuration can be found in the per-region
    # configuration.
    [gateway]

      # CA certificate and key file (optional).
      #
      # If setting the CA certificate and key file options, ChirpStack
      # will generate client certificates which can be used by the gateway for
      # authentication and authorization. The Common Name of the certificate will
      # be set to the Gateway ID.
      ca_key=""
      ca_cert=""

      # Certificate lifetime.
      #
      # This defines how long (after generating) the certificate remains valid.
      client_cert_lifetime="11months 30days 3h 50m 24s"


    # Network related configuration.
    [network]

      # Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203).
      net_id="000000"

      # Enabled regions.
      #
      # Multiple regions can be enabled simultaneously. Each region must match
      # the 'name' parameter of the region configuration in '[[regions]]'.
      enabled_regions=[
        "eu433",
        "eu868",
        "us915_0",
      ]

      # Device session expiration.
      #
      # The TTL value defines the time after which a device-session expires
      # after no activity.
      device_session_ttl="1month 13h 26m 24s"

      # Time to wait for uplink de-duplication.
      #
      # This is the time that ChirpStack will wait for other gateways to receive
      # the same uplink frame. Please note that this value affects the
      # roundtrip time. The total roundtrip time (which includes network latency)
      # must be less than the (first) receive-window.
      deduplication_delay="200ms"

      # Mac-commands disabled.
      mac_commands_disabled=false


      # Scheduler settings.
      [network.scheduler]

        # Scheduler interval.
        #
        # The interval in which the downlink scheduler for multicast, Class-B and
        # Class-C runs.
        interval="1s"

        # Class-A lock duration.
        #
        # This defines the lock duration between receiving a Class-A uplink and
        # the next scheduler-run for a device. The purpose of this lock is to
        # avoid collisions between Class-A and Class-B/C downlinks.
        class_a_lock_duration="5s"

        # Class-C lock duration.
        #
        # This defines the lock duration between scheduling two Class-C downlink
        # payloads for the same device. The purpose of this lock is to avoid
        # overlap between scheduling Class-C downlinks and / or spreading the
        # downlink capacity load on the gateway.
        class_c_lock_duration="5s"

        # Multicast Class-C use GPS time.
        #
        # Use GPS time for scheduling multicast class-c downlinks. If this is enabled
        # and the downlink must be send by multiple gateways to cover all devices
        # within the multicast-group, these downlinks will be sent at exactly the same
        # time by these gateways. If disabled, ChirpStack will use the configured
        # margin. Only enable this features when all gateways have GNSS support.
        multicast_class_c_use_gps_time=false

        # Multicast Class-C margin.
        #
        # This defines the minimum margin between scheduling multiple multicast downlinks
        # (within the same multicast-group). This value must be equal or greater than the
        # scheduler interval.
        multicast_class_c_margin="5s"

        # Multicast Class-B margin.
        #
        # This defines the minimum margin between scheduling multiple multicast downlinks
        # (within the same multicast-group). This value must be equal or greater than the
        # scheduler interval.
        multicast_class_b_margin="5s"


    # Monitoring related configuration.
    [monitoring]

      # interface:port to bind the monitoring endpoint to (optional).
      #
      # /health  - Returns 200 in case the healthchecks have passed.
      # /metrics - Returns metrics which can be scraped by Prometheus.
      #
      # If not set, this endpoint will be disabled.
      bind="0.0.0.0:8081"


      # Meta-log max history.
      #
      # This defines the max number of meta records that will be persisted in Redis Streams.
      # Setting this value to 0 disables this feature.
      meta_log_max_history=10

      # Gateway frame-log max history.
      #
      # This defines the max number of frame-log records that will be persisted in Redis Streams.
      # This stream contains the uplink and downlink frames of all gateways.
      # Setting this value to 0 disables this feature.
      gateway_frame_log_max_history=10

      # Device frame-log max history.
      #
      # This defines the max number of frame-log records that will be persisted in Redis Streams.
      # This stream contains the uplink and downlink frames of all devices.
      # Setting this value to 0 disables this feature.
      device_frame_log_max_history=10

      # Device event-log max history.
      #
      # This defines the max number of event-log records that will be persisted in Redis Streams.
      # This stream contains the events of all devices.
      # Setting this value to 0 disables this feature.
      device_event_log_max_history=10

      # Per gateway frame-log max history.
      #
      # Equal to the gateway_frame_log_max_history, but for each gateway a new Redis Stream
      # is created.
      # Setting this value to 0 disables this feature.
      per_gateway_frame_log_max_history=10

      # Per gateway frame-log TTL.
      #
      # This defines the TTL of the Redis Stream key.
      per_gateway_frame_log_ttl="1month 13h 26m 24s"

      # Per device frame-log max history.
      #
      # Equal to the device_frame_log_max_history, but for each device a new Redis Stream
      # is created.
      # Setting this value to 0 disables this feature.
      per_device_frame_log_max_history=10

      # Per device frame-log TTL.
      #
      # This defines the TTL of the Redis Stream key.
      per_device_frame_log_ttl="1month 13h 26m 24s"

      # Per device event-log max history.
      #
      # Equal to the device_event_log_max_history, but for each device a new Redis Stream
      # is created.
      # Setting this value to 0 disables this feature.
      per_device_event_log_max_history=10

      # Per device event-log TTL.
      #
      # This defines the TTL of the Redis Stream key.
      per_device_event_log_ttl="1month 13h 26m 24s"


    # Global integration related configuration.
    [integration]

      # Enabled integrations (global).
      enabled = [
        "mqtt"
      ]

      # MQTT integration configuration.
      [integration.mqtt]

        # Event topic template.
        event_topic={{`"application/{{application_id}}/device/{{dev_eui}}/event/{{event}}"`}}

        # Command topic.
        #
        # This is the topic on which the MQTT subscribes for receiving (enqueue) commands.
        command_topic={{`"application/{{application_id}}/device/{{dev_eui}}/command/{{command}}"`}}

        # Use JSON encoding instead of Protobuf (binary).
        json=false

        # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
        server="tcp://{{ .Values.chirpstack.integration.mqtt.server.host }}:1883/"

        # Connect with the given username (optional)
        username=""

        # Connect with the given password (optional)
        password=""

        # Quality of service level
        #
        # 0: at most once
        # 1: at least once
        # 2: exactly once
        #
        # Note: an increase of this value will decrease the performance.
        # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
        qos=0

        # Clean session
        #
        # Set the "clean session" flag in the connect message when this client
        # connects to an MQTT broker. By setting this flag you are indicating
        # that no messages saved by the broker for this client should be delivered.
        clean_session=false

        # Client ID
        #
        # Set the client id to be used by this client when connecting to the MQTT
        # broker. A client id must be no longer than 23 characters. If left blank,
        # a random id will be generated by ChirpStack.
        client_id=""

        # CA certificate file (optional)
        #
        # Use this when setting up a secure connection (when server uses ssl://...)
        # but the certificate used by the server is not trusted by any CA certificate
        # on the server (e.g. when self generated).
        ca_cert=""

        # TLS certificate file (optional)
        tls_cert=""

        # TLS key file (optional)
        tls_key=""

I do miss one log item in this logs, the subscribe to the gateway events, e.g. you should see something like:

2024-04-11T08:54:59.127469Z  INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=eu868 server_uri=tcp://localhost:1883 clean_session=false client_id=1d62d6dc120aac49
2024-04-11T08:54:59.127931Z  INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
2024-04-11T08:54:59.129795Z  INFO chirpstack::gateway::backend::mqtt: Subscribing to gateway event topic region_id=eu868 event_topic=$share/chirpstack/eu868/gateway/+/event/

So it looks like there is a config issue with your regions, because of that ChirpStack does not receive any gateway traffic.

1 Like

The thing is I have not changed anything in the config and I just Upgrade from 4.3.2 to 4.4.0 and it stops working. Was there any change in the config schema?

@brocaar I think I’ve found the reason. The docker image > v4.4.0 seems to muss all [[regions]] files in the /etc folder. Is that intended?

The problem is clearly that all the config files are missing in container images > 4.3.2:

Version 4.3.2 container image:

Version 4.7.0 container image:

This was removed in this commit:

was this intended?

The reason might have been that originally, all database and mqtt broker hosts configs where using env variables (e.g. server="$MQTT_BROKER_HOST"), which did not work very well and was reverted. There is not really any value in including configuration that doesn’t work (e.g. server="tcp://localhost:1883" does not work within Docker as the container itself does not provide a MQTT broker). So probably that is why it was taken out and the config was added to the chirpstack-docker repo.

1 Like

ok, many thanks. A note in the Change log would have been helpful however.