Hi, I am using the gateway with your sensors with an embedded ChirpStack server and MQTT forwarder. It operates correctly when the network it is connected to has Internet access. When connected to the network without Internet access, the ChirpStack dashboard shows that the gateway is offline. I can see Lorawan traffic in statistics, but no message is forwarded on MQTT. How can I enable offline operation?
To clarify, you are running Chirpstack on the gateway itself? Did you install it yourself? I believe Chirpstack Gateway OS does not support SenseCap gateways.
If all the components are on one system, perhaps the issue is just with your MQTT forwarders’ server configuration, if you set the MQTT forwarder to forward events to the IP of your Chirpstack server (IE the IP of the whole device), it will attempt to reach out to the internet and fail. Instead you should use localhost or similar, that way the MQTT forwarder will not try to reach out to WAN to forward the events and it should work.
Hi,
SenseCap gateway comes with a Chirpstack already installed, it can be accessed on a different port. You can see it here, under point 5.
I believe MQTT forwardes is not an issue, it works fine when the network has access to the Internet. When I unplug Internet cable from the switch, it doesn’t work anymore. Similar issue was described here: ChirpstackOS - as standalone with no internet
As you can see in the photos below, when I use the network without Internet access I am receiving lorawan frames, but nothing gets sent as MQTT (nothing appears in the device data tab as well)
[images here, blocked because I am a new user]
However, when I connect Internet to the network, everything starts working correctly and I am able to receive MQTT messages
[images here, blocked because I am a new user]
Forwarder config:
There are only two spots in the Device → Gateway → Chirpstack link that could require internet, the MQTT forwarder connecting to the MQTT broker, or Chirpstack connecting to the MQTT broker. (Technically the two database connections could possibly use internet as well but if that was the issue it would throw all sorts of errors). If you can SSH into your gateway, subscribe to the MQTT broker (while it is offline) to see if the events are making it to the broker or not. If they are not then it is certainly an issue with the MQTT forwarder, or an issue with the gateways internal routing for the MQTT forwarder.
Your photo got blocked, could you repost it. I feel this point is insightful to your actual issue. When you say your are recieving LoRaWAN frames, is that in the Chirpstack UI or in the gateways logging (i.e has it been sent to the MQTT broker yet).
The config you shared is not your MQTT forwarder config, it is the MQTT bridge that exports data to a different MQTT broker - it is not the backhaul for Chirpstack.
Ah okay this is interesting, and you are correct that the MQTT forwarder is likely not the issue here. It seems Sensecap is running Chirpstack V3, I am used to V4 where the application and network servers are merged into one.
From looking at the V3 Architecture, the application and network servers communicate over gRPC
It would seem like your issue here is that the network server is receiving the packet over MQTT (hence the lorawan frames), but then the application server fails to get that info from the network server over gRPC without internet (hence no decrypted data).
You should dig into your application server config then. Maybe share that config file? Likely the issue is that it is not routing that gRPC connection internally. Hopefully a change in the address there will fix it, or else it might be something more complex like DNS resolving poorly through a proxy like the thread you shared previously.
When I use Wireshark to capture Ethernet packets I can see repeated ARP requests whois 8.8.8.8, 223.5.5.5 and 114.114.114.114 sent from the gateway. It’s not there with Internet access.
There are 3 configs:
Please see it attached below
Gateway Bridge Configuration
[general]
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4
# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false
# Filters.
#
# These can be used to filter LoRaWAN frames to reduce bandwith usage between
# the gateway and ChirpStack Gateway Bridge. Depending the used backend, filtering
# will be performed by the Packet Forwarder or ChirpStack Gateway Bridge.
[filters]
# NetIDs filters.
#
# The configured NetIDs will be used to filter uplink data frames.
# When left blank, no filtering will be performed on NetIDs.
#
# Example:
# net_ids=[
# "000000",
# "000001",
# ]
net_ids=[
]
# JoinEUI filters.
#
# The configured JoinEUI ranges will be used to filter join-requests.
# When left blank, no filtering will be performed on JoinEUIs.
#
# Example:
# join_euis=[
# ["0000000000000000", "00000000000000ff"],
# ["000000000000ff00", "000000000000ffff"],
# ]
join_euis=[
]
# Gateway backend configuration.
[backend]
# Backend type.
#
# Valid options are:
# * semtech_udp
# * concentratord
# * basic_station
type="semtech_udp"
# Semtech UDP packet-forwarder backend.
[backend.semtech_udp]
# ip:port to bind the UDP listener to
#
# Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
# This is the listener to which the packet-forwarder forwards its data
# so make sure the 'serv_port_up' and 'serv_port_down' from your
# packet-forwarder matches this port.
udp_bind = "0.0.0.0:1700"
# Skip the CRC status-check of received packets
#
# This is only has effect when the packet-forwarder is configured to forward
# LoRa frames with CRC errors.
skip_crc_check = false
# Fake RX timestamp.
#
# Fake the RX time when the gateway does not have GPS, in which case
# the time would otherwise be unset.
fake_rx_time=false
# Integration configuration.
[integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf: Protobuf encoding
# * json: JSON encoding (easier for debugging, but less compact than 'protobuf')
marshaler="protobuf"
# MQTT integration configuration.
[integration.mqtt]
# Event topic template.
event_topic_template="gateway/{{ .GatewayID }}/event/{{ .EventType }}"
# Command topic template.
command_topic_template="gateway/{{ .GatewayID }}/command/#"
# Maximum interval that will be waited between reconnection attempts when connection is lost.
# Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
max_reconnect_interval="1m0s"
# Terminate on connect error.
#
# When set to true, instead of re-trying to connect, the ChirpStack Gateway Bridge
# process will be terminated on a connection error.
terminate_on_connect_error=false
# MQTT authentication.
[integration.mqtt.auth]
# Type defines the MQTT authentication type to use.
#
# Set this to the name of one of the sections below.
type="generic"
# Generic MQTT authentication.
[integration.mqtt.auth.generic]
# MQTT servers.
#
# Configure one or multiple MQTT server to connect to. Each item must be in
# the following format: scheme://host:port where scheme is tcp, ssl or ws.
servers=[
"tcp://127.0.0.1:1883",
]
# Connect with the given username (optional)
username=""
# Connect with the given password (optional)
password=""
# Quality of service level
#
# 0: at most once
# 1: at least once
# 2: exactly once
#
# Note: an increase of this value will decrease the performance.
# For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
qos=0
# Clean session
#
# Set the "clean session" flag in the connect message when this client
# connects to an MQTT broker. By setting this flag you are indicating
# that no messages saved by the broker for this client should be delivered.
clean_session=true
# Client ID
#
# Set the client id to be used by this client when connecting to the MQTT
# broker. A client id must be no longer than 23 characters. When left blank,
# a random id will be generated. This requires clean_session=true.
client_id=""
# CA certificate file (optional)
#
# Use this when setting up a secure connection (when server uses ssl://...)
# but the certificate used by the server is not trusted by any CA certificate
# on the server (e.g. when self generated).
ca_cert=""
# mqtt TLS certificate file (optional)
tls_cert=""
# mqtt TLS key file (optional)
tls_key=""
# Metrics configuration.
[metrics]
# Metrics stored in Prometheus.
#
# These metrics expose information about the state of the ChirpStack Gateway Bridge
# instance like number of messages processed, number of function calls, etc.
[metrics.prometheus]
# Expose Prometheus metrics endpoint.
endpoint_enabled=false
# The ip:port to bind the Prometheus metrics server to for serving the
# metrics endpoint.
bind=""
# Gateway meta-data.
#
# The meta-data will be added to every stats message sent by the ChirpStack Gateway
# Bridge.
[meta_data]
# Static.
#
# Static key (string) / value (string) meta-data.
[meta_data.static]
# Example:
# serial_number="A1B21234"
# Dynamic meta-data.
#
# Dynamic meta-data is retrieved by executing external commands.
# This makes it possible to for example execute an external command to
# read the gateway temperature.
[meta_data.dynamic]
# Execution interval of the commands.
execution_interval="1m0s"
# Max. execution duration.
max_execution_duration="1s"
# Split delimiter.
#
# When the output of a command returns multiple lines, ChirpStack Gateway Bridge
# assumes multiple values are returned. In this case it will split by the given delimiter
# to obtain the key / value of each row. The key will be prefixed with the name of the
# configured command.
split_delimiter="="
Network Server Configuration
[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4
# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false
# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=chirpstack_ns dbname=chirpstack_ns sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
dsn="postgres://chirpstack_ns:dbpassword@127.0.0.1/chirpstack_ns?sslmode=disable"
# Automatically apply database migrations.
#
# It is possible to apply the database-migrations by hand
# (see https://github.com/brocaar/chirpstack-network-server/tree/master/migrations)
# or let ChirpStack Application Server migrate to the latest state automatically, by using
# this setting. Make sure that you always make a backup when upgrading ChirpStack
# Application Server and / or applying migrations.
automigrate=true
# Max open connections.
#
# This sets the max. number of open connections that are allowed in the
# PostgreSQL connection pool (0 = unlimited).
max_open_connections=0
# Max idle connections.
#
# This sets the max. number of idle connections in the PostgreSQL connection
# pool (0 = no idle connections are retained).
max_idle_connections=2
# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# Server address or addresses.
#
# Set multiple addresses when connecting to a cluster.
servers=[
"127.0.0.1:6379",
]
# Password.
#
# Set the password when connecting to Redis requires password authentication.
password=""
# Database index.
#
# By default, this can be a number between 0-15.
database=0
# Redis Cluster.
#
# Set this to true when the provided URLs are pointing to a Redis Cluster
# instance.
cluster=false
# Master name.
#
# Set the master name when the provided URLs are pointing to a Redis Sentinel
# instance.
master_name=""
# Connection pool size.
#
# Default (when set to 0) is 10 connections per every CPU.
pool_size=0
# Network-server settings.
[network_server]
# Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203)
net_id="000000"
# Time to wait for uplink de-duplication.
#
# This is the time that ChirpStack Network Server will wait for other gateways to receive
# the same uplink frame. Valid units are 'ms' or 's'.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means ChirpStack Network Server will be
# unable to respond to the device within its receive-window.
deduplication_delay="200ms"
# Device session expiration.
#
# The TTL value defines the time after which a device-session expires
# after no activity. Valid units are 'ms', 's', 'm', 'h'. Note that these
# values can be combined, e.g. '24h30m15s'.
device_session_ttl="744h0m0s"
# Get downlink data delay.
#
# This is the time that ChirpStack Network Server waits between forwarding data to the
# application-server and reading data from the queue. A higher value
# means that the application-server has more time to schedule a downlink
# queue item which can be processed within the same uplink / downlink
# transaction.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means ChirpStack Network Server will be
# unable to respond to the device within its receive-window.
get_downlink_data_delay="100ms"
# LoRaWAN regional band configuration.
#
# Note that you might want to consult the LoRaWAN Regional Parameters
# specification for valid values that apply to your region.
# See: https://www.lora-alliance.org/lorawan-for-developers
[network_server.band]
# LoRaWAN band to use.
#
# Valid values are:
# * AS923
# * AU915
# * CN470
# * CN779
# * EU433
# * EU868
# * IN865
# * KR920
# * RU864
# * US915
name="EU868"
# Enforce 400ms dwell time.
#
# Some regions require the configuration of the dwell time, which will
# limit the time-on-air to 400ms. Please refer to the LoRaWAN Regional
# Parameters specification for more information.
#
# When configured and required in the configured region, ChirpStack Network Server will
# use the TxParamSetup mac-command to communicate this to the devices.
uplink_dwell_time_400ms=false
downlink_dwell_time_400ms=false
# Uplink max. EIRP.
#
# This defines the maximum allowed device EIRP which must be configured
# for some regions. Please refer the LoRaWAN Regional Parameters specification
# for more information. Set this to -1 to use the default value for this
# region.
#
# When required in the configured region, ChirpStack Network Server will use the
# TxParamSetup mac-command to communicate this to the devices.
# For regions where the TxParamSetup mac-command is not implemented, this
# setting is ignored.
uplink_max_eirp=-1
# Enforce repeater compatibility.
#
# Most band configurations define the max payload size for both an optional
# repeater encapsulation layer as for setups where a repeater will never
# be used. The latter case increases the max payload size for some data-rates.
# In case a repeater might used, set this flag to true.
repeater_compatible=false
# LoRaWAN network related settings.
[network_server.network_settings]
# Installation margin (dB) used by the ADR engine.
#
# A higher number means that the network-server will keep more margin,
# resulting in a lower data-rate but decreasing the chance that the
# device gets disconnected because it is unable to reach one of the
# surrounded gateways.
installation_margin=10
# RX window (Class-A).
#
# Set this to:
# 0: RX1 / RX2
# 1: RX1 only
# 2: RX2 only
rx_window=0
# Class A RX1 delay
#
# 0=1sec, 1=1sec, ... 15=15sec. A higher value means ChirpStack Network Server has more
# time to respond to the device as the delay between the uplink and the
# first receive-window will be increased.
rx1_delay=1
# RX1 data-rate offset
#
# Please consult the LoRaWAN Regional Parameters specification for valid
# options of the configured network_server.band.name.
rx1_dr_offset=0
# RX2 data-rate
#
# When set to -1, the default RX2 data-rate will be used for the configured
# LoRaWAN band.
#
# Please consult the LoRaWAN Regional Parameters specification for valid
# options of the configured network_server.band.name.
rx2_dr=-1
# RX2 frequency
#
# When set to -1, the default RX2 frequency will be used.
#
# Please consult the LoRaWAN Regional Parameters specification for valid
# options of the configured network_server.band.name.
rx2_frequency=-1
# Prefer RX2 on RX1 data-rate less than.
#
# Prefer RX2 over RX1 based on the RX1 data-rate. When the RX1 data-rate
# is smaller than the configured value, then the Network Server will
# first try to schedule the downlink for RX2, failing that (e.g. the gateway
# has already a payload scheduled at the RX2 timing) it will try RX1.
rx2_prefer_on_rx1_dr_lt=0
# Prefer RX2 on link budget.
#
# When the link-budget is better for RX2 than for RX1, the Network Server will first
# try to schedule the downlink in RX2, failing that it will try RX1.
rx2_prefer_on_link_budget=false
# Prefer gateways for downlink with given uplink (SNR) margin.
#
# When receiving an uplink (by multiple gateways), the Network Server will
# prefer the gateways that have at least the configured margin for the uplink
# SNR when sending a downlink. Margin:
# uplink SNR - required SNR for spreading factor
#
# * In case multiple gateways match, the Network Server will select a random
# gateway from the match.
# * In case non of the gateways have the desired margin or the uplink
# modulation was not LoRa, then the gateway with the best SNR (or RSSI
# in case of FSK) will be selected when sending a downlink.
gateway_prefer_min_margin=10
# Downlink TX Power (dBm)
#
# When set to -1, the downlink TX Power from the configured band will
# be used.
#
# Please consult the LoRaWAN Regional Parameters and local regulations
# for valid and legal options. Note that the configured TX Power must be
# supported by your gateway(s).
downlink_tx_power=-1
# Disable mac-commands
#
# When set to true, ChirpStack Network Server will not handle and / or schedule any
# mac-commands. However, it is still possible for an external controller
# to handle and / or schedule mac-commands. This is intended for testing
# only.
disable_mac_commands=false
# Disable ADR
#
# When set, this globally disables ADR.
disable_adr=false
# Max mac-command error count.
#
# When a mac-command is nACKed for more than the configured value, then the
# ChirpStack Network Server will stop sending this mac-command to the device.
# This setting prevents that the Network Server will keep sending mac-commands
# on every downlink in case of a malfunctioning device.
max_mac_command_error_count=3
# Enable only a given sub-set of channels
#
# Use this when ony a sub-set of the by default enabled channels are being
# used. For example when only using the first 8 channels of the US band.
# Note: when left blank, all channels will be enabled.
#
# For the US band, there are 64 125kHz channels (0-63) with 8 500kHz
# channels (65-71) with frequencies in the middle of each
# sub-band of 125kHz channels.
# Most US LoRa gateways recieve on only one sub-band which consists of
# 8 125kHz channels and 1 500 kHz channel
#
# Example: (sub-band 1)
# enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7, 64]
# Exmaple: (sub-band 2)
# enabled_uplink_channels=[8, 9, 10, 11, 12, 13, 14, 15, 65]
enabled_uplink_channels=[]
# Extra channel configuration.
#
# Use this for LoRaWAN regions where it is possible to extend the by default
# available channels with additional channels (e.g. the EU band).
# The first 5 channels will be configured as part of the OTAA join-response
# (using the CFList field).
# The other channels (or channel / data-rate changes) will be (re)configured
# using the NewChannelReq mac-command.
#
# Example:
# [[network_server.network_settings.extra_channels]]
# frequency=867100000
# min_dr=0
# max_dr=5
# [[network_server.network_settings.extra_channels]]
# frequency=867300000
# min_dr=0
# max_dr=5
# [[network_server.network_settings.extra_channels]]
# frequency=867500000
# min_dr=0
# max_dr=5
# [[network_server.network_settings.extra_channels]]
# frequency=867700000
# min_dr=0
# max_dr=5
# [[network_server.network_settings.extra_channels]]
# frequency=867900000
# min_dr=0
# max_dr=5
# Class B settings
[network_server.network_settings.class_b]
# Ping-slot data-rate.
ping_slot_dr=0
# Ping-slot frequency (Hz)
#
# Set this to 0 to use the default frequency plan for the configured region
# (which could be frequency hopping).
ping_slot_frequency=0
# Rejoin-request settings
#
# When enabled, ChirpStack Network Server will request the device to send a rejoin-request
# every time when one of the 2 conditions below is met (frame count or time).
[network_server.network_settings.rejoin_request]
# Request device to periodically send rejoin-requests
enabled=false
# The device must send a rejoin-request type 0 at least every 2^(max_count_n + 4)
# uplink messages. Valid values are 0 to 15.
max_count_n=0
# The device must send a rejoin-request type 0 at least every 2^(max_time_n + 10)
# seconds. Valid values are 0 to 15.
#
# 0 = roughly 17 minutes
# 15 = about 1 year
max_time_n=0
# Scheduler settings
#
# These settings affect the multicast, Class-B and Class-C downlink queue
# scheduler.
[network_server.scheduler]
# Scheduler interval
#
# The interval in which the downlink scheduler for multicast, Class-B and
# Class-C runs.
scheduler_interval="1s"
# Class-C settings.
[network_server.scheduler.class_c]
# Downlink lock duration
#
# Contains the duration to lock the downlink Class-C transmissions
# after a preceeding downlink tx (per device).
downlink_lock_duration="2s"
# Multicast gateway delay.
#
# In case of a multi-gateway multicast downlink, this delay will added to
# the transmission time of each downlink to avoid collisions between overlapping
# gateways.
multicast_gateway_delay="2s"
# Network-server API
#
# This is the network-server API that is used by ChirpStack Application Server or other
# custom components interacting with ChirpStack Network Server.
[network_server.api]
# ip:port to bind the api server
bind="0.0.0.0:8005"
# ca certificate used by the api server (optional)
ca_cert=""
# tls certificate used by the api server (optional)
tls_cert=""
# tls key used by the api server (optional)
tls_key=""
# Gateway settings.
[network_server.gateway]
# CA certificate and key file (optional).
#
# When setting the CA certificate and key file options, ChirpStack Network Server
# will generate client certificates which can be used by the gateway for
# authentication and authorization. The Common Name of the certificate will
# be set to the Gateway ID.
ca_cert=""
ca_key=""
# Certificate lifetime.
#
# This defines how long (after generating) the certificate remains valid.
client_cert_lifetime="8760h0m0s"
# Backend defines the gateway backend settings.
#
# The gateway backend handles the communication with the gateway(s) part of
# the LoRaWAN network.
[network_server.gateway.backend]
# Backend
#
# This defines the backend to use for the communication with the gateways.
# Use the section name of one of the following gateway backends.
# Valid options are:
# * mqtt
# * amqp
# * gcp_pub_sub
# * azure_iot_hub
type="mqtt"
# Multi-downlink feature flag.
#
# This controls the new multi downlink feature, in which the Chirpstack
# Network Server will send the downlink parameters for all possible
# receive windows to the ChirpStack Gateway Bridge, avoiding an additional
# roundtrip in case of a retry (e.g second receive window).
#
# Valid options are:
# * hybrid (default)
# * multi_only (will become the default in next major release)
# * legacy
#
# hybrid: Will send a downlink command in both the new (multi) format
# as the old format to the ChirpStack Gateway Bridge. Use this when
# not all ChirpStack Gateway Bridge instances are v3.9+.
#
# multi_only: Will send a downlink command only in the new (multi) format.
# This will not work with ChirpStack Gateway Bridge versions less than v3.9.
#
# legacy: Will send a downlink command only in the old format.
multi_downlink_feature="hybrid"
# MQTT gateway backend settings.
#
# This is the backend communicating with the LoRa gateways over a MQTT broker.
[network_server.gateway.backend.mqtt]
# MQTT topic templates for the different MQTT topics.
#
# The meaning of these topics are documented at:
# https://www.chirpstack.io/gateway-bridge/
#
# The default values match the default expected configuration of the
# ChirpStack Gateway Bridge MQTT backend. Therefore only change these values when
# absolutely needed.
# Event topic template.
event_topic="gateway/+/event/+"
# Command topic template.
#
# Use:
# * "{{ .GatewayID }}" as an substitution for the LoRa gateway ID
# * "{{ .CommandType }}" as an substitution for the command type
command_topic_template="gateway/{{ .GatewayID }}/command/{{ .CommandType }}"
# MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
server="tcp://127.0.0.1:1883"
# Connect with the given username (optional)
username=""
# Connect with the given password (optional)
password=""
# Maximum interval that will be waited between reconnection attempts when connection is lost.
# Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
max_reconnect_interval="1m0s"
# Quality of service level
#
# 0: at most once
# 1: at least once
# 2: exactly once
#
# Note: an increase of this value will decrease the performance.
# For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
qos=0
# Clean session
#
# Set the "clean session" flag in the connect message when this client
# connects to an MQTT broker. By setting this flag you are indicating
# that no messages saved by the broker for this client should be delivered.
clean_session=true
# Monitoring settings.
#
# Note that this replaces the metrics configuration. If a metrics section is
# found in the configuration, then it will fall back to that and the monitoring
# section is ignored.
[monitoring]
# IP:port to bind the monitoring endpoint to.
#
# When left blank, the monitoring endpoint will be disabled.
bind=""
# Prometheus metrics endpoint.
#
# When set true, Prometheus metrics will be served at '/metrics'.
prometheus_endpoint=false
# Prometheus API timing histogram.
#
# By setting this to true, the API request timing histogram will be enabled.
# See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
prometheus_api_timing_histogram=false
# Health check endpoint.
#
# When set to true, the healthcheck endpoint will be served at '/health'.
# When requesting, this endpoint will perform the following actions to
# determine the health of this service:
# * Ping PostgreSQL database
# * Ping Redis database
healthcheck_endpoint=false
# Join-server settings.
[join_server]
# Resolve JoinEUI (experimental).
# Default join-server settings.
#
# When set to true, ChirpStack Network Server will use the JoinEUI to resolve the join-server
# for the given JoinEUI. ChirpStack Network Server will fallback on the default join-server
# when resolving the JoinEUI fails.
resolve_join_eui=false
# Resolve domain suffix.
#
# This configures the domain suffix used for resolving the join-server.
resolve_domain_suffix=".joineuis.lora-alliance.org"
# Per Join Server configuration.
#
# Example:
# [[join_server.servers]]
# # JoinEUI.
# #
# # The JoinEUI of the joinserver to to use the certificates for.
# join_eui="0102030405060708"
# # Server (optional).
# #
# # The endpoint to the Join Server. If set, the DNS lookup will not be used
# # for the JoinEUI associated with this server.
server="https://example.com:1234/join/endpoint"
# # CA certificate (optional).
# #
# # Set this to validate the join-server server certificate (e.g. when the
# # certificate was self-signed).
# ca_cert="/path/to/ca.pem"
# # TLS client-certificate (optional).
# #
# # Set this to enable client-certificate authentication with the join-server.
# tls_cert="/path/to/tls_cert.pem"
# # TLS client-certificate key (optional).
# #
# # Set this to enable client-certificate authentication with the join-server.
# tls_key="/path/to/tls_key.pem"
# Default join-server settings.
#
# This join-server will be used when resolving the JoinEUI is set to false
# or as a fallback when resolving the JoinEUI fails.
[join_server.default]
# Default server endpoint.
#
# This API is provided by ChirpStack Application Server.
server="http://127.0.0.1:8003"
# ca certificate used by the default join-server client (optional)
ca_cert=""
# tls certificate used by the default join-server client (optional)
tls_cert=""
# tls key used by the default join-server client (optional)
tls_key=""
# Join-server KEK set.
#
# These KEKs (Key Encryption Keys) are used to decrypt the network related
# session-keys received from the join-server on a (re)join-accept.
# Please refer to the LoRaWAN Backend Interface specification
# 'Key Transport Security' section for more information.
#
# Example (the [[join_server.kek.set]] can be repeated):
# [[join_server.kek.set]]
# # KEK label.
# label="000000"
# # Key Encryption Key.
# kek="01020304050607080102030405060708"
# Network-controller configuration.
[network_controller]
# hostname:port of the network-controller api server (optional)
server=""
# ca certificate used by the network-controller client (optional)
ca_cert=""
# tls certificate used by the network-controller client (optional)
tls_cert=""
# tls key used by the network-controller client (optional)
tls_key=""
Application Server Configuration
[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4
# Log to syslog.
#
# When set to true, log messages are being written to syslog.
log_to_syslog=false
# The number of times passwords must be hashed. A higher number is safer as
# an attack takes more time to perform.
password_hash_iterations=100000
# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=chirpstack_as dbname=chirpstack_as sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
dsn="postgres://chirpstack_as:dbpassword@localhost/chirpstack_as?sslmode=disable"
# Automatically apply database migrations.
#
# It is possible to apply the database-migrations by hand
# (see https://github.com/brocaar/chirpstack-application-server/tree/master/migrations)
# or let ChirpStack Application Server migrate to the latest state automatically, by using
# this setting. Make sure that you always make a backup when upgrading Lora
# App Server and / or applying migrations.
automigrate=true
# Max open connections.
#
# This sets the max. number of open connections that are allowed in the
# PostgreSQL connection pool (0 = unlimited).
max_open_connections=0
# Max idle connections.
#
# This sets the max. number of idle connections in the PostgreSQL connection
# pool (0 = no idle connections are retained).
max_idle_connections=2
# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# Server address or addresses.
#
# Set multiple addresses when connecting to a cluster.
servers=[
"localhost:6379",
]
# Password.
#
# Set the password when connecting to Redis requires password authentication.
password=""
# Database index.
#
# By default, this can be a number between 0-15.
database=0
# Redis Cluster.
#
# Set this to true when the provided URLs are pointing to a Redis Cluster
# instance.
cluster=false
# Master name.
#
# Set the master name when the provided URLs are pointing to a Redis Sentinel
# instance.
master_name=""
# Connection pool size.
#
# Default (when set to 0) is 10 connections per every CPU.
pool_size=0
# Application-server settings.
[application_server]
# Application-server identifier.
#
# Random UUID defining the id of the application-server installation (used by
# ChirpStack Network Server as routing-profile id).
# For now it is recommended to not change this id.
id="6d5db27e-4ce2-4b2b-b5d7-91f069397978"
# User authentication
[application_server.user_authentication]
# OpenID Connect.
[application_server.user_authentication.openid_connect]
# Enable OpenID Connect authentication.
#
# Enabling this option replaces password authentication.
enabled=false
# Registration enabled.
#
# Enabling this will automatically register the user when it is not yet present
# in the ChirpStack Application Server database. There is no
# registration form as the user information is automatically received using the
# OpenID Connect provided information.
# The user will not be associated with any organization, but in order to
# facilitate the automatic onboarding of users, it is possible to configure a
# registration callback URL (next config option).
registration_enabled=false
# Registration callback URL.
#
# This (optional) endpoint will be called on the registration of the user and
# can implement the association of the user with an organization, create a new
# organization, ...
# ChirpStack Application Server will make a HTTP POST call to this endpoint,
# with the URL parameter user_id.
registration_callback_url=""
# Provider URL.
# This is the URL of the OpenID Connect provider.
provider_url=""
# Client ID.
client_id=""
# Client secret.
client_secret=""
# Redirect URL.
#
# This must contain the ChirpStack Application Server web-interface hostname
# with '/auth/oidc/callback' path, e.g. https://example.com/auth/oidc/callback.
redirect_url=""
# Login label.
#
# The login label is used in the web-interface login form.
login_label=""
# JavaScript codec settings.
[application_server.codec.js]
# Maximum execution time.
max_execution_time="1000ms"
# Integration configures the data integration.
#
# This is the data integration which is available for all applications,
# besides the extra integrations that can be added on a per-application
# basis.
[application_server.integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf: Protobuf encoding
# * json: JSON encoding (easier for debugging, but less compact than 'protobuf')
# * json_v3: v3 JSON (will be removed in the next major release)
marshaler="json"
# Enabled integrations.
#
# Enabled integrations are enabled for all applications. Multiple
# integrations can be configured.
# Do not forget to configure the related configuration section below for
# the enabled integrations. Integrations that can be enabled are:
# * mqtt - MQTT broker
# * amqp - AMQP / RabbitMQ
# * aws_sns - AWS Simple Notification Service (SNS)
# * azure_service_bus - Azure Service-Bus
# * gcp_pub_sub - Google Cloud Pub/Sub
# * kafka - Kafka distributed streaming platform
# * postgresql - PostgreSQL database
enabled=["mqtt"]
# MQTT integration backend.
[application_server.integration.mqtt]
# Event topic template.
event_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/event/{{ .EventType }}"
# Command topic template.
command_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/command/{{ .CommandType }}"
# Retain events.
#
# The MQTT broker will store the last publised message, when retain events is set
# to true. When a MQTT client connects and subscribes, it will always receive the
# last published message.
retain_events=false
# MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
server="tcp://localhost:1883"
# Connect with the given username (optional)
username=""
# Connect with the given password (optional)
password=""
# Maximum interval that will be waited between reconnection attempts when connection is lost.
# Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
max_reconnect_interval="1m0s"
# Quality of service level
#
# 0: at most once
# 1: at least once
# 2: exactly once
#
# Note: an increase of this value will decrease the performance.
# For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
qos=0
# Clean session
#
# Set the "clean session" flag in the connect message when this client
# connects to an MQTT broker. By setting this flag you are indicating
# that no messages saved by the broker for this client should be delivered.
clean_session=true
# Client ID
#
# Set the client id to be used by this client when connecting to the MQTT
# broker. A client id must be no longer than 23 characters. When left blank,
# a random id will be generated. This requires clean_session=true.
client_id=""
# CA certificate file (optional)
#
# Use this when setting up a secure connection (when server uses ssl://...)
# but the certificate used by the server is not trusted by any CA certificate
# on the server (e.g. when self generated).
ca_cert=""
# TLS certificate file (optional)
tls_cert=""
# TLS key file (optional)
tls_key=""
# AMQP / RabbitMQ.
[application_server.integration.amqp]
# Server URL.
#
# See for a specification of all the possible options:
# https://www.rabbitmq.com/uri-spec.html
url="amqp://guest:guest@localhost:5672"
# Event routing key template.
#
# This is the event routing-key template used when publishing device
# events.
event_routing_key_template="application.{{ .ApplicationID }}.device.{{ .DevEUI }}.event.{{ .EventType }}"
# AWS Simple Notification Service (SNS)
[application_server.integration.aws_sns]
# AWS region.
#
# Example: "eu-west-1".
# See also: https://docs.aws.amazon.com/general/latest/gr/rande.html.
aws_region=""
# AWS Access Key ID.
aws_access_key_id=""
# AWS Secret Access Key.
aws_secret_access_key=""
# Topic ARN (SNS).
topic_arn=""
# Azure Service-Bus integration.
[application_server.integration.azure_service_bus]
# Connection string.
#
# The connection string can be found / created in the Azure console under
# Settings -> Shared access policies. The policy must contain Manage & Send.
connection_string=""
# Publish mode.
#
# Select either "topic", or "queue".
publish_mode=""
# Publish name.
#
# The name of the topic or queue.
publish_name=""
# Google Cloud Pub/Sub integration.
[application_server.integration.gcp_pub_sub]
# Path to the IAM service-account credentials file.
#
# Note: this service-account must have the following Pub/Sub roles:
# * Pub/Sub Editor
credentials_file=""
# Google Cloud project id.
project_id=""
# Pub/Sub topic name.
topic_name=""
# Kafka integration.
[application_server.integration.kafka]
# Brokers, e.g.: localhost:9092.
brokers=["localhost:9092"]
# Topic for events.
topic="chirpstack_as"
# Template for keys included in Kafka messages. If empty, no key is included.
# Kafka uses the key for distributing messages over partitions. You can use
# this to ensure some subset of messages end up in the same partition, so
# they can be consumed in-order. And Kafka can use the key for data retention
# decisions. A header "event" with the event type is included in each
# message. There is no need to parse it from the key.
event_key_template="application.{{ .ApplicationID }}.device.{{ .DevEUI }}.event.{{ .EventType }}"
# PostgreSQL database integration.
[application_server.integration.postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
dsn=""
# This sets the max. number of open connections that are allowed in the
# PostgreSQL connection pool (0 = unlimited).
max_open_connections=0
# Max idle connections.
#
# This sets the max. number of idle connections in the PostgreSQL connection
# pool (0 = no idle connections are retained).
max_idle_connections=2
# Settings for the "internal api"
#
# This is the API used by ChirpStack Network Server to communicate with ChirpStack Application Server
# and should not be exposed to the end-user.
[application_server.api]
# ip:port to bind the api server
bind="0.0.0.0:8001"
# ca certificate used by the api server (optional)
ca_cert=""
# tls certificate used by the api server (optional)
tls_cert=""
# tls key used by the api server (optional)
tls_key=""
# Public ip:port of the application-server API.
#
# This is used by ChirpStack Network Server to connect to ChirpStack Application Server. When running
# ChirpStack Application Server on a different host than ChirpStack Network Server, make sure to set
# this to the host:ip on which ChirpStack Network Server can reach ChirpStack Application Server.
# The port must be equal to the port configured by the 'bind' flag
# above.
public_host="localhost:8001"
# Settings for the "external api"
#
# This is the API and web-interface exposed to the end-user.
[application_server.external_api]
# ip:port to bind the (user facing) http server to (web-interface and REST / gRPC api)
bind="0.0.0.0:8080"
# http server TLS certificate (optional)
tls_cert=""
# http server TLS key (optional)
tls_key=""
# JWT secret used for api authentication / authorization
# You could generate this by executing 'openssl rand -base64 32' for example
jwt_secret="gR/PKjBWkNUOsF7iAXA966VFljQnjMPDhVDXh8RR1ok="
# Join-server configuration.
#
# ChirpStack Application Server implements a (subset) of the join-api specified by the
# LoRaWAN Backend Interfaces specification. This API is used by ChirpStack Network Server
# to handle join-requests.
[join_server]
# ip:port to bind the join-server api interface to
bind="0.0.0.0:8003"
# CA certificate (optional).
#
# When set, the server requires a client-certificate and will validate this
# certificate on incoming requests.
ca_cert=""
# TLS server-certificate (optional).
#
# Set this to enable TLS.
tls_cert=""
# TLS server-certificate key (optional).
#
# Set this to enable TLS.
tls_key=""
# Key Encryption Key (KEK) configuration.
#
# The KEK mechanism is used to encrypt the session-keys sent from the
# join-server to the network-server.
#
# The ChirpStack Application Server join-server will use the NetID of the requesting
# network-server as the KEK label. When no such label exists in the set,
# the session-keys will be sent unencrypted (which can be fine for
# private networks).
#
# Please refer to the LoRaWAN Backend Interface specification
# 'Key Transport Security' section for more information.
[join_server.kek]
# Application-server KEK label.
#
# This defines the KEK label used to encrypt the AppSKey (note that the
# AppSKey is signaled to the NS and on the first received uplink from the
# NS to the AS).
#
# When left blank, the AppSKey will be sent unencrypted (which can be fine
# for private networks).
as_kek_label=""
# KEK set.
#
# Example (the [[join_server.kek.set]] can be repeated):
# [[join_server.kek.set]]
# # KEK label.
# label="000000"
# # Key Encryption Key.
# kek="01020304050607080102030405060708"
# Metrics collection settings.
[metrics]
# Timezone
#
# The timezone is used for correctly aggregating the metrics (e.g. per hour,
# day or month).
# Example: "Europe/Amsterdam" or "Local" for the the system's local time zone.
timezone="Local"
# Metrics stored in Redis.
#
# The following metrics are stored in Redis:
# * gateway statistics
[metrics.redis]
# Aggregation intervals
#
# The intervals on which to aggregate. Available options are:
# 'MINUTE', 'HOUR', 'DAY', 'MONTH'.
aggregation_intervals=["MINUTE", "HOUR", "DAY", "MONTH"]
# Aggregated statistics storage duration.
minute_aggregation_ttl="2h0m0s"
hour_aggregation_ttl="48h0m0s"
day_aggregation_ttl="2160h0m0s"
month_aggregation_ttl="17520h0m0s"
# Metrics stored in Prometheus.
#
# These metrics expose information about the state of the ChirpStack Network Server
# instance.
[metrics.prometheus]
# Enable Prometheus metrics endpoint.
endpoint_enabled=false
# The ip:port to bind the Prometheus metrics server to for serving the
# metrics endpoint.
bind=""
# API timing histogram.
#
# By setting this to true, the API request timing histogram will be enabled.
# See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
api_timing_histogram=false
# Monitoring settings.
#
# Note that this replaces the metrics.prometheus configuration. If a
# metrics.prometheus if found in the configuration then it will fall back
# to that and the monitoring section is ignored.
[monitoring]
# IP:port to bind the monitoring endpoint to.
#
# When left blank, the monitoring endpoint will be disabled.
bind=""
# Prometheus metrics endpoint.
#
# When set to true, Prometheus metrics will be served at '/metrics'.
prometheus_endpoint=false
# Prometheus API timing histogram.
#
# By setting this to true, the API request timing histogram will be enabled.
# See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
prometheus_api_timing_histogram=false
# Health check endpoint.
#
# When set to true, the healthcheck endpoint will be served at '/health'.
# When requesting, this endpoint will perform the following actions to
# determine the health of this service:
# * Ping PostgreSQL database
# * Ping Redis database
healthcheck_endpoint=false
This might be a better question for seeed themselves. I’m definitely getting out of my comfort zone here working on V3. A few thoughts though. Considering there is no actual address definition in the config files, I don’t think they are going to be the solution here. When going through the sensecap guide step 5. Did you just put the network server address as localhost:8005? Might seem silly but try 127.0.0.1:8005, potentially avoiding some resolving issues. As for the IPs GPT has this to say:
Yes, the IP addresses you see in the ARP requests captured with Wireshark—8.8.8.8, 223.5.5.5, and 114.114.114.114—are all publicly known DNS servers:
- 8.8.8.8: This is a widely used public DNS server provided by Google.
- 223.5.5.5: This IP address belongs to AliDNS, a DNS service operated by Alibaba in China.
- 114.114.114.114: This IP address belongs to 114DNS, a public DNS service popular in China.
Slightly concerning that your gateway is reaching out to China, but it is interesting that they are all DNS servers. Not sure if this is meaningless or the root of your problems (perhaps the fact that it is reaching out to those server implies a proxy like the other post you linked).
It is set to 0.0.0.0:8005
Oh, have you tried localhost:8005? Or 127.0.0.1:8005? This is likely your problem.
Just tried both, no difference.
Isn’t the gateway trying to resolve the EUI online?
# Join-server settings.
[join_server]
# Resolve JoinEUI (experimental).
# Default join-server settings.
#
# When set to true, ChirpStack Network Server will use the JoinEUI to resolve the join-server
# for the given JoinEUI. ChirpStack Network Server will fallback on the default join-server
# when resolving the JoinEUI fails.
resolve_join_eui=false
# Resolve domain suffix.
#
# This configures the domain suffix used for resolving the join-server.
resolve_domain_suffix=".joineuis.lora-alliance.org"
# Per Join Server configuration.
#
# Example:
# [[join_server.servers]]
# # JoinEUI.
# #
# # The JoinEUI of the joinserver to to use the certificates for.
# join_eui="0102030405060708"
# # Server (optional).
# #
# # The endpoint to the Join Server. If set, the DNS lookup will not be used
# # for the JoinEUI associated with this server.
server="https://example.com:1234/join/endpoint"
Damn. Well sorry man I’ve exhausted all my ideas, probably best to bring it up with Seeed support for now.
I believe resolve_join_eui=false mean it will not try to resolve that externally. In V3 the application server handles joins unless specified otherwise:
Are you pointing your gateway to the IP address of your chirpstack server or to domain name? If to a domain name is it resolvable by the DNS server that is set on your gateway?
Considering the data from the MQTT forwarder still makes it to the MQTT broker and the network server (i.e still getting logged in LoRaWAN frames) when the gateway is offline I don’t think this is the issue. The issue is almost certainly with the NS ↔ AS connection.
Hi Liam,
I am a Software Engineer from Seeed. Thanks for helping to investigate the issue. We have chirpstack-network-server v3.10.0 and chirpstack-application-server v3.12.0 on our SenseCAP LoRaWAN Outdoor Gateway.
This issue only happens when our gateway boots up without the Internet. Here are the logs from the application server but I haven’t found any useful clues yet. You may want to have a look at them.
1 Like
Thanks for the log info, I scanned through them and also didn’t see anything that immediately jumped out. A few things sparked my curiosity though:
Again I’ve only worked with Chirpstack V4 but I’m shocked at the amount of SQL calls in these logs especially considering it’s all in about a minute. Typically in V4 this only happens when someone is clicking through the webUI. Can I get a bit more info on the device while these logs were running? Does the gateway have end devices uplinking during that period or was it a factory gateway with no devices connected? Were you using the UI at all during these logs? And to confirm these logs are from a device that was not connected to the internet?
The other curiosity is the lack of gRPC calls, I figured the application server would need to poll the network server frequently (at the very least with each UI click) and you would see plenty of grpc logs and virtually no SQL logs, but it would likely make more sense the other way around: the network server receives an event and uses the application servers grpc api to pass the info to it, and on the application server side we only see the SQL calls that are a consequence of those gRPC commands. If that’s the case the logs from the network server might be more important.
The only thing that really stood out to me in the logs were these lines:
DEBU[0057] Failed to write response: io: read/write on closed pipe
INFO[0057] finished streaming call with code OK
grpc.code=OK grpc.method=StreamEventLogs grpc.service=api.DeviceService
grpc.start_time="2024-11-18T06:09:18Z" grpc.time_ms=1529.572
peer.address="127.0.0.1:45630" span.kind=server system=grpc
But even they are not very suspect. I think the failed to write response was just from the prior SQL call and not the following grpc call. Although it is interesting that the Application server is making successful grpc calls, my question is to what API? The network sever or its own API? If you could check what service is bound to port 45630 it would help, because if that is the network servers port then clearly the application server can communicate with the network server and the issue must lie with the network server communicating to the application server. (Or perhaps it could still be a database issue depending on what the application server stores itself in SQL, although I’d expect errors if that were the case)
I believe it must be the network servers as the application server starts a bind on 8080:
INFO[0001] api/external: starting api server bind="0.0.0.0:8080" tls-cert= tls-key=
Q1: Does the gateway have end devices uplinking during that period or was it a factory gateway with no devices connected?
A1: There are dozens of end devices around the gateway. Basically, the gateway is receiving lorawan packages every second. However not all of the end devices are registered on this gateway. Only 1 end node(eui: 2cf7f1c042900212) gets registered.
Q2: Were you using the UI at all during these logs?
A2: No, I thought I didn’t open the UI at that time.
Q3: And to confirm these logs are from a device that was not connected to the internet?
A3: Yes, the gateway is connected to a router without Internet at that time.
Thanks again for the analysis, I will post the network server logs and applications server logs in contrast.
Update:
I have just checked the network server logs and found this:
INFO[0025] gateway/mqtt: uplink frame received gateway_id=2cf7f110413000c2 uplink_id=c3355429-db3d-44ab-868e-a0458074a043
DEBU[0025] dns: SRV record lookup error: lookup _grpclb._tcp.localhost on [::1]:53: dial udp [::1]:53: operation was canceled
DEBU[0025] ccResolverWrapper: sending update to cc: {[{127.0.0.1:8001 <nil> 0 <nil>} {[::1]:8001 <nil> 0 <nil>}] <nil> <nil>}
DEBU[0025] running multicast scheduler batch ctx_id=35f406d0-11b7-4ab8-a036-e33f4b1c091d
ERRO[0025] uplink: processing uplink frame error ctx_id=25fd32f9-da06-4cb9-b30d-2f8f0963868c error="get application-server client error: create application-server api client error: dial application-server api error: context deadline exceeded"
Is the DNS lookup failure the root cause?