Gateway : PULL_DATA were not ACKed

Hi, my gateway shown last seen at never. When i register the gateway in the TTN, the gateway was connected perfectly. However, when i tried to connect to the chirpstack. The Data ack was not shown. I don’t know what is the problem. I have installed all the three component on the server and i have risinghf mini kit gateway. I guess the packet forwarder was working fine since i can connect it with ttn.

Here the packet forwarder log file:
*** Beacon Packet Forwarder for Lora Gateway ***
Version: 3.0.0
*** Lora concentrator HAL library version info ***
Version: 4.0.1;
***
INFO: Little endian host
INFO: found global configuration file global_conf.json, parsing it
INFO: global_conf.json does contain a JSON object named SX1301_conf, parsing SX1                                                                                                                                                             301 parameters
INFO: lorawan_public 1, clksrc 1
INFO: antenna_gain 0 dBi
INFO: Configuring TX LUT with 16 indexes
INFO: radio 0 enabled (type SX1257), center frequency 867500000, RSSI offset -16                                                                                                                                                             6.000000, tx enabled 1
INFO: radio 1 enabled (type SX1257), center frequency 868500000, RSSI offset -16                                                                                                                                                             6.000000, tx enabled 0
INFO: Lora multi-SF channel 0>  radio 1, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 1>  radio 1, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 2>  radio 1, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 3>  radio 0, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 4>  radio 0, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 5>  radio 0, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 6>  radio 0, IF 200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 7>  radio 0, IF 400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora std channel> radio 1, IF -200000 Hz, 250000 Hz bw, SF 7
INFO: FSK channel> radio 1, IF 300000 Hz, 125000 Hz bw, 50000 bps datarate
INFO: global_conf.json does contain a JSON object named gateway_conf, parsing ga                                                                                                                                                             teway parameters
INFO: gateway MAC address is configured to B827EBFFFE5BA6AE
INFO: server hostname or IP address is configured to "10.207.165.17"
INFO: upstream port is configured to "1700"
INFO: downstream port is configured to "1700"
INFO: downstream keep-alive interval is configured to 10 seconds
INFO: statistics display interval is configured to 30 seconds
INFO: upstream PUSH_DATA time-out is configured to 100 ms
INFO: packets received with a valid CRC will be forwarded
INFO: packets received with a CRC error will NOT be forwarded
INFO: packets received with no CRC will NOT be forwarded
INFO: Auto-quit after 3 non-acknowledged PULL_DATA
INFO: found local configuration file local_conf.json, parsing it
INFO: redefined parameters will overwrite global parameters
INFO: local_conf.json does not contain a JSON object named SX1301_conf
INFO: local_conf.json does contain a JSON object named gateway_conf, parsing gat                                                                                                                                                             eway parameters
INFO: gateway MAC address is configured to B827EBFFFE5BA6AE
INFO: packets received with a valid CRC will be forwarded
INFO: packets received with a CRC error will NOT be forwarded
INFO: packets received with no CRC will NOT be forwarded
INFO: [main] concentrator started, packet can now be received

INFO: Disabling GPS mode for concentrator's counter...
INFO: host/sx1301 time offset=(1659083193s:700574µs) - drift=1751765214µs
INFO: Enabling GPS mode for concentrator's counter.


##### 2022-07-29 08:27:06 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 0
# CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 0 (0 bytes)
# PUSH_DATA datagrams sent: 0 (0 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (0.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
# BEACON queued: 0
# BEACON sent so far: 0
# BEACON rejected: 0
### [JIT] ###
src/jitqueue.c:511:jit_print_queue(): INFO: [jit] queue is empty
### [GPS] ###
# GPS sync is disabled
##### END #####

JSON up: {"stat":{"time":"2022-07-29 08:27:06 GMT","rxnb":0,"rxok":0,"rxfw":0,"a                                                                                                                                                             ckr":0.0,"dwnb":0,"txnb":0}}
INFO: [down] the last 3 PULL_DATA were not ACKed, exiting application

INFO: End of downstream thread

INFO: End of upstream thread

##### 2022-07-29 08:27:36 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 0
# CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 0 (0 bytes)
# PUSH_DATA datagrams sent: 1 (111 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 0 (0.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
# BEACON queued: 0
# BEACON sent so far: 0
# BEACON rejected: 0
### [JIT] ###
src/jitqueue.c:511:jit_print_queue(): INFO: [jit] queue is empty
### [GPS] ###
# GPS sync is disabled
##### END #####
INFO: concentrator stopped successfully
INFO: Exiting packet forwarder program

**Here my chirpstack-gateway-bridge.toml :**

[general]
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4

# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false


# Filters.
#
# These can be used to filter LoRaWAN frames to reduce bandwith usage between
# the gateway and ChirpStack Gateway Bridge. Depending the used backend, filtering
# will be performed by the Packet Forwarder or ChirpStack Gateway Bridge.
[filters]

# NetIDs filters.
#
# The configured NetIDs will be used to filter uplink data frames.
# When left blank, no filtering will be performed on NetIDs.
#
# Example:
# net_ids=[
#   "000000",
#   "000001",
# ]
net_ids=[
]

# JoinEUI filters.
#
# The configured JoinEUI ranges will be used to filter join-requests.
# When left blank, no filtering will be performed on JoinEUIs.
#
# Example:
# join_euis=[
#   ["0000000000000000", "00000000000000ff"],
#   ["000000000000ff00", "000000000000ffff"],
# ]
join_euis=[
]


# Gateway backend configuration.
[backend]

# Backend type.
#
# Valid options are:
#   * semtech_udp
#   * concentratord
#   * basic_station
type="semtech_udp"

# Backend type.
#
# Valid options are:
#   * semtech_udp
#   * concentratord
#   * basic_station
type="semtech_udp"


  # Semtech UDP packet-forwarder backend.
  [backend.semtech_udp]

  # ip:port to bind the UDP listener to
  #
  # Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
  # This is the listener to which the packet-forwarder forwards its data
  # so make sure the 'serv_port_up' and 'serv_port_down' from your
  # packet-forwarder matches this port.
  udp_bind = "0.0.0.0:1700"

  # Skip the CRC status-check of received packets
  #
  # This is only has effect when the packet-forwarder is configured to forward
  # LoRa frames with CRC errors.
  skip_crc_check = true

  # Fake RX timestamp.
  #
  # Fake the RX time when the gateway does not have GPS, in which case
  # the time would otherwise be unset.
  fake_rx_time=false


  # ChirpStack Concentratord backend.
  [backend.concentratord]

  # Check for CRC OK.
  crc_check=true

  # Event API URL.
  event_url="ipc:///tmp/concentratord_event"

  # Command API URL.
  command_url="ipc:///tmp/concentratord_command"


  # Basic Station backend.
  [backend.basic_station]

  # ip:port to bind the Websocket listener to.
  bind=":3001"

  # TLS certificate and key files.
  #
 # TLS certificate and key files.
  #
  # When set, the websocket listener will use TLS to secure the connections
  # between the gateways and ChirpStack Gateway Bridge (optional).
  tls_cert=""
  tls_key=""

  # TLS CA certificate.
  #
  # When configured, ChirpStack Gateway Bridge will validate that the client
  # certificate of the gateway has been signed by this CA certificate.
  ca_cert=""

  # Stats interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge forwards
  # the uplink / downlink statistics.
  stats_interval="30s"

  # Ping interval.
  ping_interval="1m0s"

  # Timesync interval.
  #
  # This defines the interval in which the ChirpStack Gateway Bridge sends
  # a timesync request to the gateway. Setting this to 0 disables sending
  # timesync requests.
  timesync_interval="1h0m0s"

  # Read timeout.
  #
  # This interval must be greater than the configured ping interval.
  read_timeout="1m5s"

  # Write timeout.
  write_timeout="1s"

  # Region.
  #
  # Please refer to the LoRaWAN Regional Parameters specification
  # for the complete list of common region names.
  region="EU868"

  # Minimal frequency (Hz).
  frequency_min=863000000

  # Maximum frequency (Hz).
  frequency_max=870000000

  # Concentrator configuration.
  #
  # This section contains the configuration for the SX1301 concentrator chips.
  # Example:
  # [[backend.basic_station.concentrators]]
  #
  #   # Multi-SF channel configuration.
  #   [backend.basic_station.concentrators.multi_sf]
  #
  #   # Frequencies (Hz).
  #   frequencies=[
  #     868100000,
  #     868300000,
  #     868500000,
  #     867100000,
  #     867300000,
  #     867500000,
  #     867700000,
  #     867900000,
  #   ]
  #
  #   # LoRa STD channel.
  #   [backend.basic_station.concentrators.lora_std]
  #
  #   # Frequency (Hz).
  #   frequency=868300000
  #
  #   # Bandwidth (Hz).
  #   bandwidth=250000
  #
  #   # Spreading factor.
  #   spreading_factor=7
  #
  #   # FSK channel.
  #   [backend.basic_station.concentrators.fsk]
  #
  #   # Frequency (Hz).
  #   frequency=868800000


# Integration configuration.
[integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf:  Protobuf encoding
# * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')
marshaler="protobuf"

  # MQTT integration configuration.
  [integration.mqtt]
  # Event topic template.
  event_topic_template="gateway/{{ .GatewayID }}/event/{{ .EventType }}"

  # State topic template.
  #
  # States are sent by the gateway as retained MQTT messages (by default)
  #
  # States are sent by the gateway as retained MQTT messages (by default)
  # so that the last message will be stored by the MQTT broker. When set to
  # a blank string, this feature will be disabled. This feature is only
  # supported when using the generic authentication type.
  state_topic_template="gateway/{{ .GatewayID }}/state/{{ .StateType }}"

  # Command topic template.
  command_topic_template="gateway/{{ .GatewayID }}/command/#"

  # State retained.
  #
  # By default this value is set to true and states are published as retained
  # MQTT messages. Setting this to false means that states will not be retained
  # by the MQTT broker.
  state_retained=true

  # Keep alive will set the amount of time (in seconds) that the client should
  # wait before sending a PING request to the broker. This will allow the client
  # to know that a connection has not been lost with the server.
  # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
  keep_alive="30s"

  # Maximum interval that will be waited between reconnection attempts when connection is lost.
  # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
  max_reconnect_interval="1m0s"

  # Terminate on connect error.
  #
  # When set to true, instead of re-trying to connect, the ChirpStack Gateway Bridge
  # process will be terminated on a connection error.
  terminate_on_connect_error=false


  # MQTT authentication.
  [integration.mqtt.auth]
  # Type defines the MQTT authentication type to use.
  #
  # Set this to the name of one of the sections below.
  type="generic"

    # Generic MQTT authentication.
    [integration.mqtt.auth.generic]
    # MQTT servers.
    #
    # Configure one or multiple MQTT server to connect to. Each item must be in
    # the following format: scheme://host:port where scheme is tcp, ssl or ws.
    servers=[
      "tcp://10.207.165.17:1883",
    ]

    # Connect with the given username (optional)
    username=""

    # Connect with the given password (optional)
    username=""

    # Connect with the given password (optional)
    password=""

    # Quality of service level
    #
    # 0: at most once
    # 1: at least once
    # 2: exactly once
    #
    # Note: an increase of this value will decrease the performance.
    # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
    qos=0

    # Clean session
    #
    # Set the "clean session" flag in the connect message when this client
    # connects to an MQTT broker. By setting this flag you are indicating
    # that no messages saved by the broker for this client should be delivered.
    clean_session=true

    # Client ID
    #
    # Set the client id to be used by this client when connecting to the MQTT
    # broker. A client id must be no longer than 23 characters. When left blank,
    # a random id will be generated. This requires clean_session=true.
    client_id=""

    # CA certificate file (optional)
    #
    # Use this when setting up a secure connection (when server uses ssl://...)
    # but the certificate used by the server is not trusted by any CA certificate
    # on the server (e.g. when self generated).
    ca_cert=""

    # mqtt TLS certificate file (optional)
    tls_cert=""

    # mqtt TLS key file (optional)
    tls_key=""


    # Google Cloud Platform Cloud IoT Core authentication.
    #
    # Please note that when using this authentication type, the MQTT topics
    # will be automatically set to match the MQTT topics as expected by
    # Cloud IoT Core.
    [integration.mqtt.auth.gcp_cloud_iot_core]
    # MQTT server.
    server="ssl://mqtt.googleapis.com:8883"

    # Google Cloud IoT Core Device id.
    device_id=""
 # Google Cloud project id.
    project_id=""

    # Google Cloud region.
    cloud_region=""

    # Google Cloud IoT registry id.
    registry_id=""

    # JWT token expiration time.
    jwt_expiration="24h0m0s"

    # JWT token key-file.
    #
    # Example command to generate a key-pair:
    #  $ ssh-keygen -t rsa -b 4096 -f private-key.pem
    #  $ openssl rsa -in private-key.pem -pubout -outform PEM -out public-key.pem
    #
    # Then point the setting below to the private-key.pem and associate the
    # public-key.pem with this device / gateway in Google Cloud IoT Core.
    jwt_key_file=""


    # Azure IoT Hub
    #
    # This setting will preset uplink and downlink topics that will only
    # work with Azure IoT Hub service.
    [integration.mqtt.auth.azure_iot_hub]

    # Device connection string (symmetric key authentication).
    #
    # This connection string can be retrieved from the Azure IoT Hub device
    # details when using the symmetric key authentication type.
    device_connection_string=""

    # Token expiration (symmetric key authentication).
    #
    # ChirpStack Gateway Bridge will generate a SAS token with the given expiration.
    # After the token has expired, it will generate a new one and trigger a
    # re-connect (only for symmetric key authentication).
    sas_token_expiration="24h0m0s"

    # Device ID (X.509 authentication).
    #
    # This will be automatically set when a device connection string is given.
    # It must be set for X.509 authentication.
    device_id=""

    # IoT Hub hostname (X.509 authentication).
    #
    # This will be automatically set when a device connection string is given.
    # It must be set for X.509 authentication.
    # Example: iot-hub-name.azure-devices.net
    hostname=""
    # Client certificates (X.509 authentication).
    #
    # Configure the tls_cert (certificate file) and tls_key (private-key file)
    # when the device is configured with X.509 authentication.
    tls_cert=""
    tls_key=""


# Metrics configuration.
[metrics]

  # Metrics stored in Prometheus.
  #
  # These metrics expose information about the state of the ChirpStack Gateway Bridge
  # instance like number of messages processed, number of function calls, etc.
  [metrics.prometheus]
  # Expose Prometheus metrics endpoint.
  endpoint_enabled=false

  # The ip:port to bind the Prometheus metrics server to for serving the
  # metrics endpoint.
  bind=""


# Gateway meta-data.
#
# The meta-data will be added to every stats message sent by the ChirpStack Gateway
# Bridge.
[meta_data]

  # Static.
  #
  # Static key (string) / value (string) meta-data.
  [meta_data.static]
  # Example:
  # serial_number="A1B21234"




  # Dynamic meta-data.
  #
  # Dynamic meta-data is retrieved by executing external commands.
  # This makes it possible to for example execute an external command to
  # read the gateway temperature.
  [meta_data.dynamic]

  # Execution interval of the commands.
  execution_interval="1m0s"

  # Max. execution duration.
  max_execution_duration="1s"

  # Split delimiter.
  #
  # When the output of a command returns multiple lines, ChirpStack Gateway Bridge
  # assumes multiple values are returned. In this case it will split by the given delimiter
  # to obtain the key / value of each row. The key will be prefixed with the name of the
  # configured command.
  split_delimiter="="


  # Commands to execute.
  #
  # The value of the stdout will be used as the key value (string).
  # In case the command failed, it is ignored. In case the same key is defined
  # both as static and dynamic, the dynamic value has priority (as long as the)
  # command does not fail.
  [meta_data.dynamic.commands]
  # Example:
  # temperature="/opt/gateway-temperature/gateway-temperature.sh"


# Executable commands.
#
# The configured commands can be triggered by sending a message to the
# ChirpStack Gateway Bridge.
[commands]
  # Example:
  # [commands.commands.reboot]
  # max_execution_duration="1s"
  # command="/usr/bin/reboot"

i run journalctl -u chirpstack-gateway-bridge -f -n 50:

i run journalctl -u chirpstack-network-server -f -n 50:

i run journalctl -u chirpstack-application-server -f -n 50:

Please help me, i’ve been working on this for 3 month but no progress at all.

This indicates that none of the data sent by the packet-forwarder was acknowledged (by the ChirpStack Gateway Bridge). This probably means that either the data doesn’t reach the ChirpStack Gateway Bridge (network related), The ChirpStack Gateway Bridge data doesn’t reach the packet-forwarder or ChirpStack Gateway Bridge isn’t listening on the expected port.

INFO: [up] PUSH_ACK received in 0 ms
INFO: [down] PULL_ACK received in 0 ms
INFO: [down] PULL_ACK received in 0 ms
INFO: [down] PULL_ACK received in 0 ms

is it same with this? why the push and pull received in 0 ms. Is it because the same possible reason you state above? because after i enable port the this shown in the log file and the gateway was connected to the chirpstack but no live frame was coming in the dashboard.

The Semtech UDP Packet Forwarder creates two UDP sockets, one for uplink and one for downlink. Depending your configuration, it could be that one works (and you see ACKs) and the other one fails.

In which configuration? Does it in the global_conf.json file?

See an example here: