Downlink packets get dropped in the gateway

I tried basicstation and Semtech UDP packet-forwarder as the gateway bridge backend. I am running into a problem when I send a downlink packet, it does not reach the node. I can see the packets in the gateway bridge logs

 time="2022-08-13T11:37:21.312829076Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=429ccf59ce491977 uplink_id=819812df-9e08-4044-bf79-a81c563df589
time="2022-08-13T11:37:21.31337407Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=gateway/429ccf59ce491977/event/up uplink_id=819812df-9e08-4044-bf79-a81c563df589
time="2022-08-13T11:37:22.23341386Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=0485f3e0-136b-4f75-a8fb-ae865750547d gateway_id=429ccf59ce491977
time="2022-08-13T11:37:22.233841924Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=0485f3e0-136b-4f75-a8fb-ae865750547d gateway_id=429ccf59ce491977
time="2022-08-13T11:37:31.300795751Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=14ae2079-153c-4ef6-9a7f-f0898d2783fa topic=gateway/429ccf59ce491977/event/stats
time="2022-08-13T11:37:31.954484853Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=429ccf59ce491977 uplink_id=c35e77a6-892b-4112-9aee-9dbc5ce6e34e
time="2022-08-13T11:37:31.954956127Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=gateway/429ccf59ce491977/event/up uplink_id=c35e77a6-892b-4112-9aee-9dbc5ce6e34e
time="2022-08-13T11:37:32.874727641Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=9afe50cd-e30d-4fc1-82a8-85e063e46520 gateway_id=429ccf59ce491977
time="2022-08-13T11:37:32.875311656Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=9afe50cd-e30d-4fc1-82a8-85e063e46520 gateway_id=429ccf59ce491977

I also see this in basicstation the gateway

2022-08-13 11:28:05.692 [SYN:INFO] MCU/SX130X drift stats: min: +0.5ppm  q50: +8.1ppm  q80: +11.9ppm  max: +13.8ppm - threshold q90: +12.4ppm
2022-08-13 11:28:05.693 [SYN:INFO] Mean MCU drift vs SX130X#0: 6.5ppm
2022-08-13 11:28:12.124 [SYN:ERRO] Failed to convert xtime to gpstime - last PPS sync too old: 40m35s
2022-08-13 11:28:14.664 [SYN:ERRO] Failed to convert xtime to gpstime - last PPS sync too old: 40m38s
2022-08-13 11:28:15.592 [S2E:WARN] Received 'dnmsg' before 'router_config' - dropped
2022-08-13 11:28:25.255 [SYN:ERRO] Failed to convert xtime to gpstime - last PPS sync too old: 40m48s
2022-08-13 11:28:26.186 [S2E:WARN] Received 'dnmsg' before 'router_config' - dropped
2022-08-13 11:28:35.442 [SYN:ERRO] Failed to convert xtime to gpstime - last PPS sync too old: 40m59s
2022-08-13 11:28:36.410 [SYN:ERRO] Failed to convert xtime to gpstime - last PPS sync too old: 41m0s
2022-08-13 11:28:37.346 [S2E:WARN] Received 'dnmsg' before 'router_config' - dropped

The message Received 'dnmsg' before 'router_config' - dropped is keep popping up as the downlink message persists in the downlink queue.

When I point the gateway to ttn server, I am able to send a downlink without any issues. So I am pretty sure that I have not configured the chirpstack-gateway-bridge correctly.

The gateway-bridge.toml file for the reference

# See https://www.chirpstack.io/gateway-bridge/install/config/ for a full
# configuration example and documentation.

[backend]
type="basic_station"

  # Semtech UDP packet-forwarder backend.
  [backend.semtech_udp]

  # ip:port to bind the UDP listener to
  #
  # Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
  # This is the listener to which the packet-forwarder forwards its data
  # so make sure the 'serv_port_up' and 'serv_port_down' from your
  # packet-forwarder matches this port.
  udp_bind = "0.0.0.0:1700"

  # Skip the CRC status-check of received packets
  #
  # This is only has effect when the packet-forwarder is configured to forward
  # LoRa frames with CRC errors.
  skip_crc_check = false

  # Fake RX timestamp.
  #
  # Fake the RX time when the gateway does not have GPS, in which case
  # the time would otherwise be unset.
  fake_rx_time=false

    # Basic Station backend.
  [backend.basic_station]

  # ip:port to bind the Websocket listener to.
  bind="0.0.0.0:3001"

  # TLS certificate and key files.
  #
  # When set, the websocket listener will use TLS to secure the connections
  # between the gateways and ChirpStack Gateway Bridge (optional).
  tls_cert=""
  tls_key=""

  # TLS CA certificate.
  #
  # When configured, ChirpStack Gateway Bridge will validate that the client
  # certificate of the gateway has been signed by this CA certificate.
  ca_cert=""

  # Stats interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge forwards
  # the uplink / downlink statistics.
  stats_interval="30s"

  # Ping interval.
  ping_interval="1m0s"

  # Timesync interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge sends
  # a timesync request to the gateway. Setting this to 0 disables sending
  # timesync requests.
  timesync_interval="1h0m0s"

  # Read timeout.
  #
  # This interval must be greater than the configured ping interval.
  read_timeout="1m5s"

  # Write timeout.
  write_timeout="1s"

  # Region.
  #
  # Please refer to the LoRaWAN Regional Parameters specification
  # for the complete list of common region names.
  region="AS923"

  # Minimal frequency (Hz).
  frequency_min=915000000

  # Maximum frequency (Hz).
  frequency_max=928000000

  # Concentrator configuration.
  #
  # This section contains the configuration for the SX1301 concentrator chips.
  # Example:
   [[backend.basic_station.concentrators]]
  
     # Multi-SF channel configuration.
     [backend.basic_station.concentrators.multi_sf]
  
     # Frequencies (Hz).
     frequencies=[
       923200000,
       923400000,
       922200000,
       922400000,
       922600000,
       922800000,
       923000000,
       922000000,
     ]
     # LoRa STD channel.
     [backend.basic_station.concentrators.lora_std]
  
     # Frequency (Hz).
     frequency=923000000
  
     # Bandwidth (Hz).
     bandwidth=250000
  
     # Spreading factor.
     spreading_factor=7
  
     # FSK channel.
     [backend.basic_station.concentrators.fsk]
  
     # Frequency (Hz).
     frequency=922000000
[integration]
marshaler="json"

[integration.mqtt.auth.generic]
servers=["tcp://mosquitto:1883"]
username=""
password=""

I would recommend you to look into the initial logs of the Basics Station. On connect, ChirpStack Gateway Bridge will send the configuration to the Basics Station. My assumption is that because of config error (in the ChirpStack Gateway Bridge configuration), this is not processed correctly and will log a warning or error at the beginning of the Basics Station startup.

I checked the logs.

2022-08-17 13:07:16.056 [SYS:INFO] Logging     : /var/log/basicstation.log (maxsize=10000000, rotate=3)
2022-08-17 13:07:16.057 [SYS:INFO] Station Ver : 2.0.6(rpi/std) 2022-08-13 08:06:22
2022-08-17 13:07:16.057 [SYS:INFO] Package Ver : (null)
2022-08-17 13:07:16.057 [SYS:INFO] mbedTLS Ver : 2.28.0
2022-08-17 13:07:16.057 [SYS:INFO] proto EUI   : 429c:cf59:ce49:1977	(station.conf)
2022-08-17 13:07:16.057 [SYS:INFO] prefix EUI  : ::1	(builtin)
2022-08-17 13:07:16.057 [SYS:INFO] Station EUI : 429c:cf59:ce49:1977
2022-08-17 13:07:16.057 [SYS:INFO] Station home: /etc/basicstation/	(--home)
2022-08-17 13:07:16.057 [SYS:INFO] Station temp: /var/tmp/	(builtin)
2022-08-17 13:07:16.057 [SYS:WARN] Station in NO-CUPS mode
2022-08-17 13:07:16.259 [TCE:INFO] Starting TC engine
2022-08-17 13:07:16.259 [TCE:INFO] Connecting to INFOS: ws://microsec-lcms:3001
2022-08-17 13:07:16.278 [TCE:INFO] Infos: 429c:cf59:ce49:1977 429c:cf59:ce49:1977 ws://microsec-lcms:3001/gateway/429ccf59ce491977
2022-08-17 13:07:16.278 [AIO:DEBU] [4] ws_close reason=1000
2022-08-17 13:07:16.278 [AIO:DEBU] Echoing close - reason=1000
2022-08-17 13:07:16.279 [AIO:DEBU] [4] Connection closed unexpectedly
2022-08-17 13:07:16.279 [AIO:DEBU] [4] WS connection shutdown...
2022-08-17 13:07:16.279 [TCE:VERB] Connecting to MUXS...
2022-08-17 13:07:16.288 [TCE:VERB] Connected to MUXS.
2022-08-17 13:07:16.295 [S2E:WARN] Unrecognized region: AS923 - ignored
2022-08-17 13:07:16.296 [SYS:VERB] /opt/basicstation/bin/reset_gw.sh: Forked, waiting...
2022-08-17 13:07:16.297 [SYS:DEBU] execvp argv[0]: </opt/basicstation/bin/reset_gw.sh>
2022-08-17 13:07:16.297 [SYS:DEBU]        argv[1]: </dev/spidev0.0>
2022-08-17 13:07:16.618 [SYS:INFO] Process /opt/basicstation/bin/reset_gw.sh (pid=28027) completed
2022-08-17 13:07:16.619 [RAL:INFO] Lora gateway library version: Version: 5.0.1;
2022-08-17 13:07:16.638 [RAL:INFO] [LGW lgw1] clksrc=1 lorawan_public=1
2022-08-17 13:07:16.639 [RAL:INFO]  RX/TX RF0:    922.2MHz rssi_offset=-166.0 type=2 tx_notch_freq=0
2022-08-17 13:07:16.639 [RAL:INFO]  RX    RF1:    923.2MHz rssi_offset=-166.0 type=2 tx_notch_freq=0
2022-08-17 13:07:16.639 [RAL:INFO]  [mSF]   0:    922.0MHz rf=0 freq=-200.0 datarate=0
2022-08-17 13:07:16.639 [RAL:INFO]  [mSF]   1:    922.2MHz rf=0 freq=  +0.0 datarate=0
2022-08-17 13:07:16.639 [RAL:INFO]  [mSF]   2:    922.4MHz rf=0 freq=+200.0 datarate=0
2022-08-17 13:07:16.639 [RAL:INFO]  [mSF]   3:    922.6MHz rf=0 freq=+400.0 datarate=0
2022-08-17 13:07:16.640 [RAL:INFO]  [mSF]   4:    922.8MHz rf=1 freq=-400.0 datarate=0
2022-08-17 13:07:16.640 [RAL:INFO]  [mSF]   5:    923.0MHz rf=1 freq=-200.0 datarate=0
2022-08-17 13:07:16.640 [RAL:INFO]  [mSF]   6:    923.2MHz rf=1 freq=  +0.0 datarate=0
2022-08-17 13:07:16.640 [RAL:INFO]  [mSF]   7:    923.4MHz rf=1 freq=+200.0 datarate=0
2022-08-17 13:07:16.640 [RAL:INFO]  [STD]   8:    922.1MHz rf=0 freq=-100.0 datarate=2 bw=2 
2022-08-17 13:07:16.640 [RAL:INFO]  [FSK]   9:    922.2MHz rf=0 freq=  +0.0 datarate=0 bw=0 sync_word=0/0
2022-08-17 13:07:16.641 [RAL:INFO] SX130x LBT not enabled
2022-08-17 13:07:16.641 [RAL:INFO] Station device: /dev/spidev0.0 (PPS capture disabled)
2022-08-17 13:07:16.641 [HAL:INFO] [lgw_spi_open:101] Setting SPI speed to 8000000
2022-08-17 13:07:16.641 [HAL:INFO] [lgw_spi_open:135] Note: SPI port opened and configured ok
2022-08-17 13:07:19.576 [RAL:INFO] Concentrator started (2s934ms)
2022-08-17 13:07:19.576 [S2E:INFO] Configuring for region: AS923 -- 920.0MHz..923.4MHz
2022-08-17 13:07:19.576 [S2E:INFO]   DR0  SF12/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR1  SF11/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR2  SF10/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR3  SF9/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR4  SF8/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR5  SF7/BW125 
2022-08-17 13:07:19.576 [S2E:INFO]   DR6  SF7/BW250 
2022-08-17 13:07:19.576 [S2E:INFO]   DR7  FSK 
2022-08-17 13:07:19.576 [S2E:INFO]   DR8  undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR9  undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR10 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR11 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR12 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR13 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR14 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   DR15 undefined
2022-08-17 13:07:19.576 [S2E:INFO]   TX power: 14.0 dBm EIRP
2022-08-17 13:07:19.576 [S2E:INFO]   JoinEui list: 0 entries
2022-08-17 13:07:19.576 [S2E:INFO]   NetID filter: FFFFFFFF-FFFFFFFF-FFFFFFFF-FFFFFFFF
2022-08-17 13:07:19.576 [S2E:INFO]   Dev/test settings: nocca=0 nodc=0 nodwell=0
2022-08-17 13:07:45.761 [S2E:VERB] RX 923.4MHz DR2 SF10/BW125 snr=2.2 rssi=-116 xtime=0x21000001B3FADC - jreq MHdr=00 JoinEui=::11 DevEui=::11 DevNonce=6320 MIC=-675685112
2022-08-17 13:07:45.770 [SYN:INFO] Server time sync: xtime=0x21000001B3FADC gpstime=0x4C7114474E5FD ppsOffset=637252 gpsOffset=0x4C7114474E5FD
2022-08-17 13:07:46.164 [S2E:WARN] Received 'dnmsg' before 'router_config' - dropped
2022-08-17 13:07:57.380 [SYN:WARN] No PPS pulse for ~12 secs
2022-08-17 13:08:01.581 [SYN:INFO] MCU/SX130X drift stats: min: +76.7ppm  q50: +80.5ppm  q80: +82.4ppm  max: +83.3ppm - threshold q90: +82.9ppm
2022-08-17 13:08:01.581 [SYN:INFO] Mean MCU drift vs SX130X#0: 80.4ppm

I can see the following issues

  • 2022-08-17 13:07:16.295 [S2E:WARN] Unrecognized region: AS923 - ignored.
  • 2022-08-17 13:07:16.640 [RAL:INFO] [FSK] 9: 922.2MHz rf=0 freq= +0.0 datarate=0 bw=0 sync_word=0/0 - Actual FSK in the Gateway bridge is 921800000. It is a mismatch

Any suggestions?

Was this ever resolved?
I’m seeing this same issue against Chirpstack version 3.16.4
Sep 8 19:55:38 RG1xx29D9CE user.notice lora: 2022-09-08 19:55:38.479 [S2E:WARN] Received ‘dnmsg’ before ‘router_config’ - dropped
(I haven’t figured out how to install 4.0 yet to try it)

Jim