Concentratord hangs after receiving config command

Hi there.
I’ve set up a RPi + RAK2245 US915 LoRa gateway connected to a Network Server + Application Server docker instance. Once it receives the config command from the Network Server, it freezes until I manually reboot it. Here are the RPi Syslog (Concentratord + Gateway Bridge), the network-server.toml and the gateway-bridge.toml files.

Thank you so much in advance, I have no clue of what’s going on (monit-summary is reporting rpi3, gateway-bridge and concentratord as OK).

Syslog:
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: ‘chirpstack-gateway-bridge’ restart on user request
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: Monit daemon with PID 423 awakened
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: Awakened by User defined signal 1
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: ‘chirpstack-gateway-bridge’ trying to restart
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: ‘chirpstack-gateway-bridge’ stop: ‘/etc/init.d/chirpstack-gateway-bridge stop’
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1362]: time=“2021-04-20T10:47:29Z” level=info msg=“signal received” signal=terminated
> Apr 20 10:47:29 raspberrypi3 user.warn chirpstack-gateway-bridge[1362]: time=“2021-04-20T10:47:29Z” level=warning msg=“shutting down server”
> Apr 20 10:47:29 raspberrypi3 user.info monit[423]: ‘chirpstack-gateway-bridge’ start: ‘/etc/init.d/chirpstack-gateway-bridge start’
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“starting ChirpStack Gateway Bridge” docs=“Introduction - ChirpStack open-source LoRaWAN<sup>®</sup> Network Server” version=3.9.2
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“backend/concentratord: setting up backend” command_url=“ipc:///tmp/concentratord_command” event_url=“ipc:///tmp/concentratord_event”
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“backend/concentratord: connected to event socket” event_url=“ipc:///tmp/concentratord_event”
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“backend/concentratord: connected to command socket” command_url=“ipc:///tmp/concentratord_event”
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“integration/mqtt: connected to mqtt broker”
> Apr 20 10:47:29 raspberrypi3 user.debug chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=debug msg=“integration/mqtt: set gateway subscription called” gateway_id=b827ebfffeb7e06c subscribe=true
> Apr 20 10:47:29 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:29Z” level=info msg=“integration/mqtt: subscribing to topic” qos=0 topic=“gateway/b827ebfffeb7e06c/command/#”
> Apr 20 10:47:30 raspberrypi3 user.info monit[423]: ‘chirpstack-gateway-bridge’ restart action done
> Apr 20 10:47:30 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915650.507635513s
> Apr 20 10:47:31 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915651.507965325s
> Apr 20 10:47:32 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Get GPS epoch error, error: gps time reference not available
> Apr 20 10:47:32 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915652.50839143s
> Apr 20 10:47:33 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915653.508630581s
> Apr 20 10:47:34 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915654.509052158s
> Apr 20 10:47:35 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915655.509473992s
> Apr 20 10:47:36 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915656.509901446s
> Apr 20 10:47:37 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915657.510319679s
> Apr 20 10:47:38 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Get GPS epoch error, error: gps time reference not available
> Apr 20 10:47:38 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915658.51078545s
> Apr 20 10:47:39 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915659.511021136s
> Apr 20 10:47:40 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915660.511445488s
> Apr 20 10:47:41 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915661.511849258s
> Apr 20 10:47:42 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915662.512248275s
> Apr 20 10:47:43 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915663.512637017s
> Apr 20 10:47:44 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Get GPS epoch error, error: gps time reference not available
> Apr 20 10:47:44 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915664.513023297s
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Get gps coordinates error, error: gps_ref_valid = false
> Apr 20 10:47:45 raspberrypi3 user.info chirpstack-concentratord-sx1301[1383]: Publishing stats event, stats_id: fa2efa9d-f7fd-4ca3-b1a1-0243ff02d37e, rx_received: 0, rx_received_ok: 0, tx_received: 0, tx_emitted: 0
> Apr 20 10:47:45 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:45Z” level=info msg=“backend/concentratord: stats event received” stats_id=fa2efa9d-f7fd-4ca3-b1a1-0243ff02d37e
> Apr 20 10:47:45 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:45Z” level=info msg=“integration/mqtt: publishing event” event=stats qos=0 stats_id=fa2efa9d-f7fd-4ca3-b1a1-0243ff02d37e topic=gateway/b827ebfffeb7e06c/event/stats
> Apr 20 10:47:45 raspberrypi3 user.warn chirpstack-concentratord-sx1301[1383]: GPS time reference is not valid, age: 1618915665.513263897s
> Apr 20 10:47:45 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:45Z” level=info msg=“integration/mqtt: gateway configuration received” topic=gateway/b827ebfffeb7e06c/command/config
> Apr 20 10:47:45 raspberrypi3 user.info chirpstack-gateway-bridge[1415]: time=“2021-04-20T10:47:45Z” level=info msg=“backend/concentratord: forwarding configuration command” version=b579f67a-c680-4cbc-8304-b34f96fb5266-r1618911576
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Timesync loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Beacon loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: GPS validation loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Stats loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: JIT loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Uplink loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Received stop signal, signal: Configuration
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: Command loop ended
> Apr 20 10:47:45 raspberrypi3 user.debug chirpstack-concentratord-sx1301[1383]: socket dropped

And here is the network-server.toml:

[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4

# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false

# gRPC default resolver scheme.
#
# Set this to "dns" for enabling dns round-robin load balancing.
grpc_default_resolver_scheme="passthrough"


# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=chirpstack_ns dbname=chirpstack_ns sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
dsn="postgres://chirpstack_ns:chirpstack_ns@chirpstack_postgres/chirpstack_ns?sslmode=disable"

# Automatically apply database migrations.
#
# It is possible to apply the database-migrations by hand
# (see https://github.com/brocaar/chirpstack-network-server/tree/master/migrations)
# or let ChirpStack Application Server migrate to the latest state automatically, by using
# this setting. Make sure that you always make a backup when upgrading ChirpStack
# Application Server and / or applying migrations.
automigrate=true

# Max open connections.
#
# This sets the max. number of open connections that are allowed in the
# PostgreSQL connection pool (0 = unlimited).
max_open_connections=0

# Max idle connections.
#
# This sets the max. number of idle connections in the PostgreSQL connection
# pool (0 = no idle connections are retained).
max_idle_connections=2


# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]

# Server address or addresses.
#
# Set multiple addresses when connecting to a cluster.
servers=[
  "chirpstack_redis:6379",
]

# Password.
#
# Set the password when connecting to Redis requires password authentication.
password=""

# Database index.
#
# By default, this can be a number between 0-15.
database=0

# Redis Cluster.
#
# Set this to true when the provided URLs are pointing to a Redis Cluster
# instance.
cluster=false

# Master name.
#
# Set the master name when the provided URLs are pointing to a Redis Sentinel
# instance.
master_name=""

# Connection pool size.
#
# Default (when set to 0) is 10 connections per every CPU.
pool_size=0

# TLS enabled.
#
# Note: this will enable TLS, but it will not validate the certificate
# used by the server.
tls_enabled=false

# Key prefix.
#
# A key prefix can be used to avoid key collisions when multiple regions
# are using the same Redis database and it is not possible to separate
# keys by database index (e.g. when using Redis Cluster, which does not
# support multiple databases).
key_prefix=""


# Network-server settings.
[network_server]
# Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203)
net_id="000000"

# Time to wait for uplink de-duplication.
#
# This is the time that ChirpStack Network Server will wait for other gateways to receive
# the same uplink frame. Valid units are 'ms' or 's'.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means ChirpStack Network Server will be
# unable to respond to the device within its receive-window.
deduplication_delay="200ms"

# Device session expiration.
#
# The TTL value defines the time after which a device-session expires
# after no activity. Valid units are 'ms', 's', 'm', 'h'. Note that these
# values can be combined, e.g. '24h30m15s'.
device_session_ttl="744h0m0s"

# Get downlink data delay.
#
# This is the time that ChirpStack Network Server waits between forwarding data to the
# application-server and reading data from the queue. A higher value
# means that the application-server has more time to schedule a downlink
# queue item which can be processed within the same uplink / downlink
# transaction.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means ChirpStack Network Server will be
# unable to respond to the device within its receive-window.
get_downlink_data_delay="100ms"


  # LoRaWAN regional band configuration.
  #
  # Note that you might want to consult the LoRaWAN Regional Parameters
  # specification for valid values that apply to your region.
  # See: https://www.lora-alliance.org/lorawan-for-developers
  [network_server.band]
  # LoRaWAN band to use.
  #
  # Valid values are:
  # * AS923
  # * AU915
  # * CN470
  # * CN779
  # * EU433
  # * EU868
  # * IN865
  # * KR920
  # * RU864
  # * US915
  name="US915"

  # Enforce 400ms dwell time.
  #
  # Some regions require the configuration of the dwell time, which will
  # limit the time-on-air to 400ms. Please refer to the LoRaWAN Regional
  # Parameters specification for more information.
  #
  # When configured and required in the configured region, ChirpStack Network Server will
  # use the TxParamSetup mac-command to communicate this to the devices.
  uplink_dwell_time_400ms=false
  downlink_dwell_time_400ms=false

  # Uplink max. EIRP.
  #
  # This defines the maximum allowed device EIRP which must be configured
  # for some regions. Please refer the LoRaWAN Regional Parameters specification
  # for more information. Set this to -1 to use the default value for this
  # region.
  #
  # When required in the configured region, ChirpStack Network Server will use the
  # TxParamSetup mac-command to communicate this to the devices.
  # For regions where the TxParamSetup mac-command is not implemented, this
  # setting is ignored.
  uplink_max_eirp=-1

  # Enforce repeater compatibility.
  #
  # Most band configurations define the max payload size for both an optional
  # repeater encapsulation layer as for setups where a repeater will never
  # be used. The latter case increases the max payload size for some data-rates.
  # In case a repeater might used, set this flag to true.
  repeater_compatible=false


  # LoRaWAN network related settings.
  [network_server.network_settings]
  # Installation margin (dB) used by the ADR engine.
  #
  # A higher number means that the network-server will keep more margin,
  # resulting in a lower data-rate but decreasing the chance that the
  # device gets disconnected because it is unable to reach one of the
  # surrounded gateways.
  installation_margin=10

  # RX window (Class-A).
  #
  # Set this to:
  # 0: RX1 / RX2
  # 1: RX1 only
  # 2: RX2 only
  rx_window=0

  # Class A RX1 delay
  #
  # 0=1sec, 1=1sec, ... 15=15sec. A higher value means ChirpStack Network Server has more
  # time to respond to the device as the delay between the uplink and the
  # first receive-window will be increased.
  rx1_delay=1

  # RX1 data-rate offset
  #
  # Please consult the LoRaWAN Regional Parameters specification for valid
  # options of the configured network_server.band.name.
  rx1_dr_offset=0

  # RX2 data-rate
  #
  # When set to -1, the default RX2 data-rate will be used for the configured
  # LoRaWAN band.
  #
  # Please consult the LoRaWAN Regional Parameters specification for valid
  # options of the configured network_server.band.name.
  rx2_dr=-1

  # RX2 frequency
  #
  # When set to -1, the default RX2 frequency will be used.
  #
  # Please consult the LoRaWAN Regional Parameters specification for valid
  # options of the configured network_server.band.name.
  rx2_frequency=-1

  # Prefer RX2 on RX1 data-rate less than.
  #
  # Prefer RX2 over RX1 based on the RX1 data-rate. When the RX1 data-rate
  # is smaller than the configured value, then the Network Server will
  # first try to schedule the downlink for RX2, failing that (e.g. the gateway
  # has already a payload scheduled at the RX2 timing) it will try RX1.
  rx2_prefer_on_rx1_dr_lt=0

  # Prefer RX2 on link budget.
  #
  # When the link-budget is better for RX2 than for RX1, the Network Server will first
  # try to schedule the downlink in RX2, failing that it will try RX1.
  rx2_prefer_on_link_budget=false

  # Prefer gateways for downlink with given uplink (SNR) margin.
  #
  # When receiving an uplink (by multiple gateways), the Network Server will
  # prefer the gateways that have at least the configured margin for the uplink
  # SNR when sending a downlink. Margin:
  #   uplink SNR - required SNR for spreading factor
  #
  #  * In case multiple gateways match, the Network Server will select a random
  #    gateway from the match.
  #  * In case non of the gateways have the desired margin or the uplink
  #    modulation was not LoRa, then the gateway with the best SNR (or RSSI
  #    in case of FSK) will be selected when sending a downlink.
  gateway_prefer_min_margin=10

  # Downlink TX Power (dBm)
  #
  # When set to -1, the downlink TX Power from the configured band will
  # be used.
  #
  # Please consult the LoRaWAN Regional Parameters and local regulations
  # for valid and legal options. Note that the configured TX Power must be
  # supported by your gateway(s).
  downlink_tx_power=-1

  # Disable mac-commands
  #
  # When set to true, ChirpStack Network Server will not handle and / or schedule any
  # mac-commands. However, it is still possible for an external controller
  # to handle and / or schedule mac-commands. This is intended for testing
  # only.
  disable_mac_commands=false

  # Disable ADR
  #
  # When set, this globally disables ADR.
  disable_adr=true

  # Max mac-command error count.
  #
  # When a mac-command is nACKed for more than the configured value, then the
  # ChirpStack Network Server will stop sending this mac-command to the device.
  # This setting prevents that the Network Server will keep sending mac-commands
  # on every downlink in case of a malfunctioning device.
  max_mac_command_error_count=3

  # Enable only a given sub-set of channels
  #
  # Use this when ony a sub-set of the by default enabled channels are being
  # used. For example when only using the first 8 channels of the US band.
  # Note: when left blank, all channels will be enabled.
  #
  # For the US band, there are 64 125kHz channels (0-63) with 8 500kHz
  # channels (65-71) with frequencies in the middle of each
  # sub-band of 125kHz channels.
  # Most US LoRa gateways recieve on only one sub-band which consists of
  # 8 125kHz channels and 1 500 kHz channel
  #
  # Example: (sub-band 1)
  # enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7, 64]
  # Exmaple: (sub-band 2)
  # enabled_uplink_channels=[8, 9, 10, 11, 12, 13, 14, 15, 65]
  enabled_uplink_channels=[8, 9, 10, 11, 12, 13, 14, 15]

  # ADR plugins.
  #
  # By default, the 'default' ADR algorithm is available. The number of available
  # ADR algorithms can be extended through plugins. This setting can be configured
  # to a list of one or multiple plugins.
  adr_plugins=[]

  # Class B settings
  [network_server.network_settings.class_b]
  # Ping-slot data-rate.
  ping_slot_dr=0

  # Ping-slot frequency (Hz)
  #
  # Set this to 0 to use the default frequency plan for the configured region
  # (which could be frequency hopping).
  ping_slot_frequency=0


  # Rejoin-request settings
  #
  # When enabled, ChirpStack Network Server will request the device to send a rejoin-request
  # every time when one of the 2 conditions below is met (frame count or time).
  [network_server.network_settings.rejoin_request]
  # Request device to periodically send rejoin-requests
  enabled=false

  # The device must send a rejoin-request type 0 at least every 2^(max_count_n + 4)
  # uplink messages. Valid values are 0 to 15.
  max_count_n=0

  # The device must send a rejoin-request type 0 at least every 2^(max_time_n + 10)
  # seconds. Valid values are 0 to 15.
  #
  # 0  = roughly 17 minutes
  # 15 = about 1 year
  max_time_n=0

  # Scheduler settings
  #
  # These settings affect the multicast, Class-B and Class-C downlink queue
  # scheduler.
  [network_server.scheduler]
  # Scheduler interval
  #
  # The interval in which the downlink scheduler for multicast, Class-B and
  # Class-C runs.
  scheduler_interval="1s"

    # Class-C settings.
    [network_server.scheduler.class_c]
    # Downlink lock duration
    #
    # Contains the duration to lock the downlink Class-C transmissions
    # after a preceeding downlink tx (per device).
    downlink_lock_duration="2s"

    # Multicast gateway delay.
    #
    # In case of a multi-gateway multicast downlink, this delay will added to
    # the transmission time of each downlink to avoid collisions between overlapping
    # gateways.
    multicast_gateway_delay="2s"


  # Network-server API
  #
  # This is the network-server API that is used by ChirpStack Application Server or other
  # custom components interacting with ChirpStack Network Server.
  [network_server.api]
  # ip:port to bind the api server
  bind="0.0.0.0:8000"

  # ca certificate used by the api server (optional)
  ca_cert=""

  # tls certificate used by the api server (optional)
  tls_cert=""

  # tls key used by the api server (optional)
  tls_key=""


  # Gateway settings.
  [network_server.gateway]
  # CA certificate and key file (optional).
  #
  # When setting the CA certificate and key file options, ChirpStack Network Server
  # will generate client certificates which can be used by the gateway for
  # authentication and authorization. The Common Name of the certificate will
  # be set to the Gateway ID.
  ca_cert=""
  ca_key=""

  # Certificate lifetime.
  #
  # This defines how long (after generating) the certificate remains valid.
  client_cert_lifetime="8760h0m0s"

  # Force gateways as private.
  #
  # This overrides the behavior of the gws_private flag in the service-profile
  # when this setting is set to true. When set to true, the gateway and device
  # must be under the same service-profile (thus a gateway-profile must be
  # assigned to the gateway).
  force_gws_private=false


  # Backend defines the gateway backend settings.
  #
  # The gateway backend handles the communication with the gateway(s) part of
  # the LoRaWAN network.
  [network_server.gateway.backend]
    # Backend
    #
    # This defines the backend to use for the communication with the gateways.
    # Use the section name of one of the following gateway backends.
    # Valid options are:
    #  * mqtt
    #  * amqp
    #  * gcp_pub_sub
    #  * azure_iot_hub
    type="mqtt"

    # Multi-downlink feature flag.
    #
    # This controls the new multi downlink feature, in which the Chirpstack
    # Network Server will send the downlink parameters for all possible
    # receive windows to the ChirpStack Gateway Bridge, avoiding an additional
    # roundtrip in case of a retry (e.g second receive window).
    #
    # Valid options are:
    #  * hybrid     (default)
    #  * multi_only (will become the default in next major release)
    #  * legacy
    #
    # hybrid: Will send a downlink command in both the new (multi) format
    # as the old format to the ChirpStack Gateway Bridge. Use this when
    # not all ChirpStack Gateway Bridge instances are v3.9+.
    #
    # multi_only: Will send a downlink command only in the new (multi) format.
    # This will not work with ChirpStack Gateway Bridge versions less than v3.9.
    #
    # legacy: Will send a downlink command only in the old format.
    multi_downlink_feature="hybrid"


    # MQTT gateway backend settings.
    #
    # This is the backend communicating with the LoRa gateways over a MQTT broker.
    [network_server.gateway.backend.mqtt]
    # MQTT topic templates for the different MQTT topics.
    #
    # The meaning of these topics are documented at:
    # https://www.chirpstack.io/gateway-bridge/
    #
    # The default values match the default expected configuration of the
    # ChirpStack Gateway Bridge MQTT backend. Therefore only change these values when
    # absolutely needed.

    # Event topic template.
    event_topic="gateway/+/event/+"

    # Command topic template.
    #
    # Use:
    #   * "{{ .GatewayID }}" as an substitution for the LoRa gateway ID
    #   * "{{ .CommandType }}" as an substitution for the command type
    command_topic_template="gateway/{{ .GatewayID }}/command/{{ .CommandType }}"

    # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
    server="tcp://mosquitto:1883"

    # Connect with the given username (optional)
    username="chirpstackNs"

    # Connect with the given password (optional)
    password=""

    # Maximum interval that will be waited between reconnection attempts when connection is lost.
    # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
    max_reconnect_interval="1m0s"

    # Quality of service level
    #
    # 0: at most once
    # 1: at least once
    # 2: exactly once
    #
    # Note: an increase of this value will decrease the performance.
    # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
    qos=0

    # Clean session
    #
    # Set the "clean session" flag in the connect message when this client
    # connects to an MQTT broker. By setting this flag you are indicating
    # that no messages saved by the broker for this client should be delivered.
    clean_session=true

    # Client ID
    #
    # Set the client id to be used by this client when connecting to the MQTT
    # broker. A client id must be no longer than 23 characters. When left blank,
    # a random id will be generated. This requires clean_session=true.
    client_id=""

    # CA certificate file (optional)
    #
    # Use this when setting up a secure connection (when server uses ssl://...)
    # but the certificate used by the server is not trusted by any CA certificate
    # on the server (e.g. when self generated).
    ca_cert=""

    # TLS certificate file (optional)
    tls_cert=""

    # TLS key file (optional)
    tls_key=""


    # AMQP / RabbitMQ.
    #
    # Use this backend when the ChirpStack Gateway Bridge is configured to connect
    # to RabbitMQ using the MQTT plugin. See for more details about this plugin:
    # https://www.rabbitmq.com/mqtt.html
    [network_server.gateway.backend.amqp]
    # Server URL.
    #
    # See for a specification of all the possible options:
    # https://www.rabbitmq.com/uri-spec.html
    url="amqp://guest:guest@localhost:5672"

    # Event queue name.
    #
    # This queue will be created when it does not yet exist and is used to
    # queue the events received from the gateway.
    event_queue_name="gateway-events"

    # Event routing key.
    #
    # This is the routing-key used for creating the queue binding.
    event_routing_key="gateway.*.event.*"

    # Command routing key template.
    #
    # This is the command routing-key template used when publishing gateway
    # commands.
    command_routing_key_template="gateway.{{ .GatewayID }}.command.{{ .CommandType }}"


    # Google Cloud Pub/Sub backend.
    #
    # Use this backend when the ChirpStack Gateway Bridge is configured to connect
    # to the Google Cloud IoT Core MQTT broker (which integrates with Pub/Sub).
    [network_server.gateway.backend.gcp_pub_sub]
    # Path to the IAM service-account credentials file.
    #
    # Note: this service-account must have the following Pub/Sub roles:
    #  * Pub/Sub Editor
    credentials_file=""

    # Google Cloud project id.
    project_id=""

    # Uplink Pub/Sub topic name (to which Cloud IoT Core publishes).
    uplink_topic_name=""

    # Downlink Pub/Sub topic name (for publishing downlink frames).
    downlink_topic_name=""

    # Uplink retention duration.
    #
    # The retention duration that ChirpStack Network Server will set on the uplink subscription.
    uplink_retention_duration="24h0m0s"


    # Azure IoT Hub backend.
    #
    # Use this backend when the ChirpStack Gateway Bridge is configured to connect
    # to the Azure IoT Hub MQTT broker.
    [network_server.gateway.backend.azure_iot_hub]

    # Events connection string.
    #
    # This connection string must point to the Service Bus Queue to which the
    # IoT Hub is forwarding the (uplink) gateway events.
    events_connection_string=""

    # Commands connection string.
    #
    # This connection string must point to the IoT Hub and is used by ChirpStack Network Server
    # for sending commands to the gateways.
    commands_connection_string=""


  # Monitoring settings.
  #
  # Note that this replaces the metrics configuration. If a metrics section is
  # found in the configuration, then it will fall back to that and the monitoring
  # section is ignored.
  [monitoring]

  # IP:port to bind the monitoring endpoint to.
  #
  # When left blank, the monitoring endpoint will be disabled.
  bind=""

  # Prometheus metrics endpoint.
  #
  # When set true, Prometheus metrics will be served at '/metrics'.
  prometheus_endpoint=false

  # Prometheus API timing histogram.
  #
  # By setting this to true, the API request timing histogram will be enabled.
  # See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
  prometheus_api_timing_histogram=false

  # Health check endpoint.
  #
  # When set to true, the healthcheck endpoint will be served at '/health'.
  # When requesting, this endpoint will perform the following actions to
  # determine the health of this service:
  #   * Ping PostgreSQL database
  #   * Ping Redis database
  healthcheck_endpoint=false


# Join-server settings.
[join_server]
# Resolve JoinEUI (experimental).
# Default join-server settings.
#
# When set to true, ChirpStack Network Server will use the JoinEUI to resolve the join-server
# for the given JoinEUI. ChirpStack Network Server will fallback on the default join-server
# when resolving the JoinEUI fails.
resolve_join_eui=false

# Resolve domain suffix.
#
# This configures the domain suffix used for resolving the join-server.
resolve_domain_suffix=".joineuis.lora-alliance.org"


  # Per Join Server configuration.
  #
  # Example:
  # [[join_server.servers]]
  # # JoinEUI.
  # #
  # # The JoinEUI of the joinserver to to use the certificates for.
  # join_eui="0102030405060708"

  # # Server (optional).
  # #
  # # The endpoint to the Join Server. If set, the DNS lookup will not be used
  # # for the JoinEUI associated with this server.
  # server="https://example.com:1234/join/endpoint"

  # # CA certificate (optional).
  # #
  # # Set this to validate the join-server server certificate (e.g. when the
  # # certificate was self-signed).
  # ca_cert="/path/to/ca.pem"

  # # TLS client-certificate (optional).
  # #
  # # Set this to enable client-certificate authentication with the join-server.
  # tls_cert="/path/to/tls_cert.pem"

  # # TLS client-certificate key (optional).
  # #
  # # Set this to enable client-certificate authentication with the join-server.
  # tls_key="/path/to/tls_key.pem"


  # Default join-server settings.
  #
  # This join-server will be used when resolving the JoinEUI is set to false
  # or as a fallback when resolving the JoinEUI fails.
  [join_server.default]
  # Default server endpoint.
  #
  # This API is provided by ChirpStack Application Server.
  server="http://chirpstack_application_server:8003"

  # ca certificate used by the default join-server client (optional)
  ca_cert=""

  # tls certificate used by the default join-server client (optional)
  tls_cert=""

  # tls key used by the default join-server client (optional)
  tls_key=""

  # Network-controller configuration.
  [network_controller]
  # hostname:port of the network-controller api server (optional)
  server=""

  # ca certificate used by the network-controller client (optional)
  ca_cert=""

  # tls certificate used by the network-controller client (optional)
  tls_cert=""

  # tls key used by the network-controller client (optional)
  tls_key=""


# Roaming settings (experimental).
[roaming]

# Resolve NetID domain suffix.
#
# This configures the domain suffix used for resolving a Network Server
# using its NetID.
resolve_netid_domain_suffix=".netids.lora-alliance.org"

  # Roaming API settings.
  [roaming.api]
  # Interface to bind the API to (ip:port).
  bind=""

  # CA certificate (optional).
  #
  # When configured, this is used for client-certificate validation.
  ca_cert=""

  # TLS certificate (optional).
  #
  # When configured, this is used to secure the API interface using TLS.
  # This must be configured together with the tls_key.
  tls_cert=""

  # TLS key (optional).
  tls_key=""

  # Default roaming server.
  #
  # When this is configured and non of the configured servers are matching the
  # NetID, then the default roaming server will be used. The same configuration
  # parameters apply as to each roaming server, except that no NetID needs to
  # be set.
  [roaming.default]
  enabled=false
  server=""
  async=false
  async_timeout="0s"
  passive_roaming=false
  passive_roaming_lifetime="0s"
  passive_roaming_kek_label=""
  ca_cert=""
  tls_cert=""
  tls_key=""

I have the same issue. I tried both Gateway Bridge v3.9.2 and the newest release v3.10.0. I suspect this is due to the newest version of Network Server, as I became aware of this behaviour after upgrading.

This is the log of Gateway Bridge v3.9.2.

INFO[0107] backend/concentratord: connected to event socket  event_url="ipc:///tmp/concentratord_event"
INFO[0107] backend/concentratord: connected to command socket  command_url="ipc:///tmp/concentratord_event"
DEBU[0128] backend/concentratord: ignoring uplink event, CRC is not valid  crc_status=BAD_CRC uplink_id=9c8247c3-73df-4bfd-94b9-5609716cf8cd
INFO[0137] backend/concentratord: stats event received   stats_id=a3e33972-d66b-47b8-8554-2d36b4f68c76
INFO[0137] integration/mqtt: publishing event            event=stats qos=0 stats_id=a3e33972-d66b-47b8-8554-2d36b4f68c76 topic=gateway/b827ebfffe573f8a/event/stats
INFO[0137] integration/mqtt: gateway configuration received  topic=gateway/b827ebfffe573f8a/command/config
INFO[0137] backend/concentratord: forwarding configuration command  version=ca4aff89-939e-40b6-af97-a3e9719198c8-r1618855941
ERRO[0137] backend/concentratord: receive event message error  error=EOF
ERRO[0139] backend/concentratord: event socket dial error  error="dial event api url error: zmq4: could not dial to \"ipc:///tmp/concentratord_event\": dial unix /tmp/concentratord_event: connect: connection refused"
ERRO[0143] backend/concentratord: event socket dial error  error="dial event api url error: zmq4: could not dial to \"ipc:///tmp/concentratord_event\": dial unix /tmp/concentratord_event: connect: connection refused"
ERRO[0146] backend/concentratord: event socket dial error  error="dial event api url error: zmq4: could not dial to \"ipc:///tmp/concentratord_event\": dial unix /tmp/concentratord_event: connect: connection refused"
INFO[0147] backend/concentratord: connected to event socket  event_url="ipc:///tmp/concentratord_event"
INFO[0147] backend/concentratord: connected to command socket  command_url="ipc:///tmp/concentratord_event"
INFO[0177] backend/concentratord: stats event received   stats_id=c12eea80-71d7-4647-9289-29d927cd01da
INFO[0177] integration/mqtt: publishing event            event=stats qos=0 stats_id=c12eea80-71d7-4647-9289-29d927cd01da topic=gateway/b827ebfffe573f8a/event/stats
INFO[0177] integration/mqtt: gateway configuration received  topic=gateway/b827ebfffe573f8a/command/config
INFO[0177] backend/concentratord: forwarding configuration command  version=ca4aff89-939e-40b6-af97-a3e9719198c8-r1618855941
ERRO[0177] backend/concentratord: receive event message error  error=EOF
FATA[0180] backend/concentratord: send configuration command error  error="receive command request reply error: EOF"

Could you share the gateway-profile which is assigned to the gateway?

Hi! Here is my gateway profile. I intend to use the second sub-band of US915.

I was not able to reproduce your issue using the latest version of Concentratord. Potentially this could be related to the bug that was fixed in Concentratord v3.0.3?

https://www.chirpstack.io/concentratord/changelog/#v303.

Gateway Bridge also has a similar error. Here are the logs.

GreenapsisGW:~$ sudo chirpstack-gateway-bridge 
INFO[0000] starting ChirpStack Gateway Bridge            docs="https://www.chirpstack.io/gateway-bridge/" version=3.9.2
INFO[0000] backend/concentratord: setting up backend     command_url="ipc:///tmp/concentratord_command" event_url="ipc:///tmp/concentratord_event"
INFO[0000] backend/concentratord: connected to event socket  event_url="ipc:///tmp/concentratord_event"
INFO[0000] backend/concentratord: connected to command socket  command_url="ipc:///tmp/concentratord_event"
INFO[0000] integration/mqtt: connected to mqtt broker   
DEBU[0000] integration/mqtt: set gateway subscription called  gateway_id=b827ebfffe573f8a subscribe=true
INFO[0000] integration/mqtt: subscribing to topic        qos=0 topic="gateway/b827ebfffe573f8a/command/#"
INFO[0020] backend/concentratord: stats event received   stats_id=0966aa21-a3ec-4871-abf6-9098b36258a8
INFO[0020] integration/mqtt: publishing event            event=stats qos=0 stats_id=0966aa21-a3ec-4871-abf6-9098b36258a8 topic=gateway/b827ebfffe573f8a/event/stats
INFO[0020] integration/mqtt: gateway configuration received  topic=gateway/b827ebfffe573f8a/command/config
INFO[0020] backend/concentratord: forwarding configuration command  version=ca4aff89-939e-40b6-af97-a3e9719198c8-r1619538308
ERRO[0020] backend/concentratord: receive event message error  error=EOF
FATA[0023] backend/concentratord: send configuration command error  error="receive command request reply error: EOF"

Could you specify “similar”?

As I mentioned, there has been a bugfix in Concentratord 3.0.3 which incorrectly rejected some configuration due to the wrong channel_max. Maybe that is the cause of your issue?

I’m sorry, my last reply is redundant as I already had uploaded the Gateway Bridge logs in an earlier reply. I don’t know what I was thinking :sweat_smile:. How can I confirm that channel_max is correctly configured in my yocto build? As this error happened with both concentratord v3.0.3 and v3.10.0 (just tested it yesterday).

I believe the source of my issue is related to Gateway Bridge or Network Server, as concentratord works fine when Gateway Bridge is not running. When I start Gateway bridge and Concentratord, the latter keeps resetting when Gateway Bridge receives the config command, exactly as @danicardonaibz describes (also, the logs my logs are identical to his).

EDIT: Both Concentratord and Gateway Bridge work fine when the Gateway Profile is removed in App Server.

Then what probably happens is that the configuration sent to the concentratord is invalid, causing the concentratord to restart and reload the config from the concentratord toml file after which the NS will re-send the config, after which the concentratord will restart and so on…