I have a dockerised chirpstack setup on an EC2 instance. I have a US915 network server and a EU868 Network server.
I have one application server they both feed into and one postgresql application with two databases for each network on it. Once DB for the application server.
They use the same bridge at the moment.
The US915 server is working perfectly, the EU868 server is not. We would like the EU868 to use ack for the devices attached to it. The devices send on subband 2.
The process we are seeing is that the devices sent their load, 8 packets in a chain. The gateway live lorawan frames on chirpstack shows they are coming in. When we go to the device itself we see all 8 packets in the Lorawan Frames. Under Device Data we see only some of them. There seems to be no pattern in which of the packets are dropped.
After digging into the logs on each network server I see that the US915 is behaving like normal, no thrown errors but the EU868 is running this message as fast as it can
log":"time=\"2022-01-18T21:21:34.523964033Z\" level=debug msg=\"running multicast scheduler batch\" ctx_id=3eaf6faf-7bf9-4fd8-ab90-e4d287d4d5d3\n","stream":"stderr","time":"2022-01-18T21:21:34.524156739Z"}
{"log":"time=\"2022-01-18T21:21:34.52583728Z\" level=debug msg=\"sql query executed\" args=\"[100 2022-01-18 21:21:34.525090807 +0000 UTC m=+2.478267316]\" duration=\"733.209µs\" query=\"\\n\\t\\tselect\\n\\t\\t\\t*\\n\\t\\tfrom\\n\\t\\t\\tmulticast_queue\\n\\t\\twhere\\n\\t\\t\\tschedule_at \u003c= $2\\n\\t\\torder by\\n\\t\\t\\tid\\n\\t\\tlimit $1\\n\\t\\tfor update skip locked\\n\\t\"\n","stream":"stderr","time":"2022-01-18T21:21:34.52594857Z"}
{"log":"time=\"2022-01-18T21:21:34.532875614Z\" level=debug msg=\"running class-b / class-c scheduler batch\" ctx_id=4fb75342-4ba9-479f-8776-54c2d3782f9d\n","stream":"stderr","time":"2022-01-18T21:21:34.532999043Z"}
{"log":"time=\"2022-01-18T21:21:34.535219157Z\" level=debug msg=\"sql query executed\" args=\"[100 368493h21m54.533761353s 2022-01-18 21:21:34.533764862 +0000 UTC m=+2.486941272]\" duration=1.442587ms query=\"\\n select\\n d.*\\n from\\n device d\\n where\\n\\t\\t\\td.mode in ('B', 'C')\\n -- we want devices with queue items\\n and exists (\\n select\\n 1\\n from\\n device_queue dq\\n where\\n dq.dev_eui = d.dev_eui\\n and (\\n\\t\\t\\t\\t\\t\\td.mode = 'C'\\n \\tor (\\n\\t\\t\\t\\t\\t\\t\\td.mode = 'B'\\n \\t\\tand dq.emit_at_time_since_gps_epoch \u003c= $2\\n \\t)\\n )\\n )\\n\\t\\t\\t-- exclude device which have one of the following below\\n and not exists (\\n select\\n 1\\n from\\n device_queue dq\\n where\\n dq.dev_eui = d.dev_eui\\n\\t\\t\\t\\t\\tand (\\n\\t\\t\\t\\t\\t\\t-- pending queue-item with timeout_after in the future\\n\\t\\t\\t\\t\\t\\t(dq.is_pending = true and dq.timeout_after \u003e $3)\\n\\n\\t\\t\\t\\t\\t\\t-- or retry_after set to a timestamp in the future\\n\\t\\t\\t\\t\\t\\tor (dq.retry_after is not null and dq.retry_after \u003e $3)\\n\\t\\t\\t\\t\\t)\\n )\\n order by\\n d.dev_eui\\n limit $1\\n for update of d skip locked\"\n","stream":"stderr","time":"2022-01-18T21:21:34.535426962Z"}
googling the error has not resulted in any new knowledge. Has this happened to anyone or does anyone know why it is happening to the EU one? Below is my network toml
# This file is for the EU868 band. See the examples/ folder for more
# configuration examples.
[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=5
[postgresql]
dsn="postgres://chirpstack_ns2:chirpstack_ns2@postgresql/chirpstack_ns2?sslmode=disable"
[redis]
url="redis://redis:6379"
[network_server]
net_id="000000"
[network_server.band]
name="EU868"
[network_server.network_settings]
rx_window=0
rx1_delay=1
rx1_dr_offset=0
rx2_dr=-1
rx2_frequency=-1
#disable_adr=false
disable_adr=true
# disable_mac_commands=true
# Enable only a given sub-set of channels
#
# Use this when ony a sub-set of the by default enabled channels are being
# used. For example when only using the first 8 channels of the US band.
# Note: when left blank, all channels will be enabled.
#
# Example:
# enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7]
enabled_uplink_channels=[]
[[network_server.network_settings.extra_channels]]
frequency=867100000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867300000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867500000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867700000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867900000
min_dr=0
max_dr=5
[network_server.gateway.backend.mqtt]
server="tcp://mosquitto:1883"
[join_server.default]
server="http://chirpstack-application-server:8003"
# Monitoring settings.
#
# Note that this replaces the metrics configuration. If a metrics section is
# found in the configuration, then it will fall back to that and the monitoring
# section is ignored.
[monitoring]
# IP:port to bind the monitoring endpoint to.
#
# When left blank, the monitoring endpoint will be disabled.
bind=""
# Prometheus metrics endpoint.
#
[[network_server.network_settings.extra_channels]]
frequency=867100000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867300000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867500000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867700000
min_dr=0
max_dr=5
[[network_server.network_settings.extra_channels]]
frequency=867900000
min_dr=0
max_dr=5
[network_server.gateway.backend.mqtt]
server="tcp://mosquitto:1883"
[join_server.default]
server="http://chirpstack-application-server:8003"
# Monitoring settings.
#
# Note that this replaces the metrics configuration. If a metrics section is
# found in the configuration, then it will fall back to that and the monitoring
# section is ignored.
[monitoring]
# IP:port to bind the monitoring endpoint to.
#
# When left blank, the monitoring endpoint will be disabled.
bind=""
# Prometheus metrics endpoint.
#
# When set true, Prometheus metrics will be served at '/metrics'.
prometheus_endpoint=false
# Prometheus API timing histogram.
#
# By setting this to true, the API request timing histogram will be enabled.
# See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
prometheus_api_timing_histogram=false
# Health check endpoint.
#
# When set to true, the healthcheck endpoint will be served at '/health'.
# When requesting, this endpoint will perform the following actions to
# determine the health of this service:
# * Ping PostgreSQL database
# * Ping Redis database
healthcheck_endpoint=false
# Device frame-log max history.
#
# When set to a value > 0, ChirpStack Network Server will log all uplink and
# downlink frames associated to a device as a Redis stream for the
# consumption by external applications (e.g. for monitoring purposes) Note
# that only uplinks passing the MIC and frame-counter tests will be published.
# The configured value will be used as approximate amount of frames which
# will be kept in the stream.
#
# The following Redis key is used:
# 'lora:ns:device:stream:frame'
device_frame_log_max_history=100
# Per device frame-log max history.
#
# Equal to the device_frame_log_max_history, but it has the device DevEUI
# in the Redis key. This feature is used by the ChirpStack Application Server
# web-interface.
per_device_frame_log_max_history=100
# Gateway frame-log max history.
#
# When set to a value > 0, ChirpStack Network Server will log all uplink and
# downlink frames received by the gateways as a Redis stream for the
# consumption by external applications (e.g. for monitoring purposes). Note
# that all data reported by the gateways will be published.
# The configured value will be used as approximate amount of frames which
# will be kept in the stream.
#
prometheus_endpoint=false
# Prometheus API timing histogram.
#
# By setting this to true, the API request timing histogram will be enabled.
# See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms
prometheus_api_timing_histogram=false
# Health check endpoint.
#
# When set to true, the healthcheck endpoint will be served at '/health'.
# When requesting, this endpoint will perform the following actions to
# determine the health of this service:
# * Ping PostgreSQL database
# * Ping Redis database
healthcheck_endpoint=false
# Device frame-log max history.
#
# When set to a value > 0, ChirpStack Network Server will log all uplink and
# downlink frames associated to a device as a Redis stream for the
# consumption by external applications (e.g. for monitoring purposes) Note
# that only uplinks passing the MIC and frame-counter tests will be published.
# The configured value will be used as approximate amount of frames which
# will be kept in the stream.
#
# The following Redis key is used:
# 'lora:ns:device:stream:frame'
device_frame_log_max_history=100
# Per device frame-log max history.
#
# Equal to the device_frame_log_max_history, but it has the device DevEUI
# in the Redis key. This feature is used by the ChirpStack Application Server
# web-interface.
per_device_frame_log_max_history=100
# Gateway frame-log max history.
#
# When set to a value > 0, ChirpStack Network Server will log all uplink and
# downlink frames received by the gateways as a Redis stream for the
# consumption by external applications (e.g. for monitoring purposes). Note
# that all data reported by the gateways will be published.
# The configured value will be used as approximate amount of frames which
# will be kept in the stream.
#
# The following Redis key is used:
# 'lora:ns:gw:stream:frame'
gateway_frame_log_max_history=100
# Per gateway frame-log max history.
#
# Equal to the gateway_frame_log_max_history, but it has the GatewayID
# in the Redis key. This feature is used by the ChirpStack Application Server
# web-interface.
per_gateway_frame_log_max_history=100