Downlink scheduler error

Hi,

I am trying downlink with “Enqueue” option from console and seeing this error.

chirpstack_1                       | 2022-11-17T17:22:26.793450Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=region_name  does not exist in REGIONS
chirpstack_1                       | 2022-11-17T17:22:28.810152Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=region_name  does not exist in REGIONS
chirpstack_1                       | 2022-11-17T17:22:30.828207Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=region_name  does not exist in REGIONS
chirpstack_1                       | 2022-11-17T17:22:32.849561Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=region_name  does not exist in REGIONS

I have deployed Chirpstack version 4 using docker-compose today. This is the docker-compose.yml

version: "3"

services:
  chirpstack:
    image: chirpstack/chirpstack:4.0.4
    command: -c /etc/chirpstack
    restart: unless-stopped
    volumes:
      - ./configuration/chirpstack:/etc/chirpstack
      - ./lorawan-devices:/opt/lorawan-devices
    depends_on:
      - postgres
      - mosquitto
      - redis
    environment:
      - MQTT_BROKER_HOST=mosquitto
      - REDIS_HOST=redis
      - POSTGRESQL_HOST=postgres
    ports:
      - 8080:8080

  chirpstack-gateway-bridge-eu868:
    image: chirpstack/chirpstack-gateway-bridge:4.0.1
    restart: unless-stopped
    ports:
      - 1700:1700/udp
    volumes:
      - ./configuration/chirpstack-gateway-bridge:/etc/chirpstack-gateway-bridge
    depends_on: 
      - mosquitto

  chirpstack-rest-api:
    image: chirpstack/chirpstack-rest-api:4.0.4
    restart: unless-stopped
    command: --server chirpstack:8080 --bind 0.0.0.0:8090 --insecure
    ports:
      - 8090:8090
    depends_on:
      - chirpstack

  postgres:
    image: postgres:14-alpine
    restart: unless-stopped
    volumes:
      - ./configuration/postgresql/initdb:/docker-entrypoint-initdb.d
      - postgresqldata:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=root

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    volumes:
      - redisdata:/data

  mosquitto:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - 1883:1883
    volumes: 
      - ./configuration/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf

volumes:
  postgresqldata:
  redisdata:

I have verified the “chirpstack.toml” and dont see any errors there. This is the “chirpstack.toml” file

# Logging.
[logging]

  # Log level.
  #
  # Options are: trace, debug, info, warn error.
  level="info"


# PostgreSQL configuration.
[postgresql]

  # PostgreSQL DSN.
  #
  # Format example: postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>/<DATABASE>?sslmode=<SSLMODE>.
  #
  # SSL mode options:
  #  * disable - no SSL
  #  * require - Always SSL (skip verification)
  #  * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
  #  * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
  dsn="postgres://chirpstack:chirpstack@$POSTGRESQL_HOST/chirpstack?sslmode=disable"

  # Max open connections.
  #
  # This sets the max. number of open connections that are allowed in the
  # PostgreSQL connection pool.
  max_open_connections=10

  # Min idle connections.
  #
  # This sets the min. number of idle connections in the PostgreSQL connection
  # pool (0 = equal to max_open_connections).
  min_idle_connections=0


# Redis configuration.
[redis]

  # Server address or addresses.
  #
  # Set multiple addresses when connecting to a cluster.
  servers=[
    "redis://$REDIS_HOST/",
  ]

  # TLS enabled.
  tls_enabled=false

  # Redis Cluster.
  #
  # Set this to true when the provided URLs are pointing to a Redis Cluster
  # instance.
  cluster=false


# Network related configuration.
[network]

  # Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203).
  net_id="000000"

  # Enabled regions.
  #
  # Multiple regions can be enabled simultaneously. Each region must match
  # the 'name' parameter of the region configuration in '[[regions]]'.
  enabled_regions=[
    "as923",
    "as923_2",
    "as923_3",
    "as923_4",
    "au915_0",
    "cn470_10",
    "cn779",
    "eu433",
    "eu868",
    "in865",
    "ism2400",
    "kr920",
    "ru864",
    "us915_0",
    "us915_1",
  ]


# API interface configuration.
[api]

  # interface:port to bind the API interface to.
  bind="0.0.0.0:8080"

  # Secret.
  #
  # This secret is used for generating login and API tokens, make sure this
  # is never exposed. Changing this secret will invalidate all login and API
  # tokens. The following command can be used to generate a random secret:
  #   openssl rand -base64 32
  secret="you-must-replace-this"


[integration]
  enabled=["mqtt"]

  [integration.mqtt]
    server="tcp://$MQTT_BROKER_HOST:1883/"
    json=true

    # Event topic template.
    event_topic="application/{{application_id}}/device/{{dev_eui}}/event/{{event}}"

    # Command topic.
    #
    # This is the topic on which the MQTT subscribes for receiving (enqueue) commands.
    command_topic="application/{{application_id}}/device/{{dev_eui}}/command/{{command}}"

Please let me know how to resolve the error.

Thanks.

Regards,
Harsha

Hi,

I tried “Flush queue” and then tried “Enqueue” and I am seeing this error on docker compose console

chirpstack_1                       | 2022-11-17T18:02:17.875075Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-17T18:02:17.880818Z  INFO chirpstack::storage::device_queue: Device queue-item updated id=46b43a47-743d-4be6-b35a-16539d9d99b6 dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-17T18:02:17.881832Z  INFO chirpstack::storage::downlink_frame: Downlink-frame saved downlink_id=2695820889
chirpstack_1                       | 2022-11-17T18:02:17.882002Z  INFO chirpstack::gateway::backend::mqtt: Sending downlink frame gateway_id=dabbccffff2a79cb topic=in865/gateway/dabbccffff2a79cb/command/down json=true
chirpstack_1                       | 2022-11-17T18:02:19.904066Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-17T18:02:19.904829Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=Get device lock

I had earlier tried to publish on MQTT broker on this topic to initiale downlink

application/7c7746bb-bc82-41b7-9a32-211fdcb72651/device/506f98000000ac63/command/down

and this payload

{
    "devEui": "506f98000000ac63",
    "confirmed": true,
    "fPort": 7,
    "data": "abcd",
    "object": {
        "temperatureSensor": {"1": 25},
        "humiditySensor": {"1": 32}
    }
}

Regards,
Harsha

Hi,

I made an observation that in the scenario when I get this error

ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=region_name  does not exist in REGIONS

If I did an uplink the error message above went away and seeing this error message instead

INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-18T17:33:57.030945Z  INFO chirpstack::storage::device_queue: Device queue-item updated id=669c62b0-b024-44d2-80fc-6919fb376588 dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-18T17:33:57.031961Z  INFO chirpstack::storage::downlink_frame: Downlink-frame saved downlink_id=1428197808
chirpstack_1                       | 2022-11-18T17:33:57.032144Z  INFO chirpstack::gateway::backend::mqtt: Sending downlink frame gateway_id=dabbccffff2a79cb topic=in865/gateway/dabbccffff2a79cb/command/down json=true
chirpstack_1                       | 2022-11-18T17:33:59.050894Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-18T17:33:59.051590Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=Get device lock

Regards,
Harsha

What modifications did you make after cloning the chirpstack-docker repository? Are there any other errors in the logs (Redis, PostgreSQL)?

Hi,

I just modified the “chirpstack.toml” and “chirpstack-gateway-bridge.toml” and did a “docker-compose up --build”. There is no other errors in logs. I also noticed if I tried to delete the application and devices and readd the same it would not delete and old data would come back. So I set up a new LNS server and working with it and everything works but I still keep seeing this error over and over in the logs.

chirpstack_1                       | 2022-11-29T14:19:40.115447Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-29T14:19:40.116101Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=Get device lock
chirpstack_1                       | 2022-11-29T14:19:42.882627Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-29T14:19:42.883245Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=Get device lock
chirpstack_1                       | 2022-11-29T14:19:45.139295Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-29T14:19:45.338353Z  INFO chirpstack::storage::device_queue: Device queue-item updated id=a48a2ad3-742a-4885-83a2-aaf48ee77046 dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-29T14:19:45.339136Z  INFO chirpstack::storage::downlink_frame: Downlink-frame saved downlink_id=604020511
chirpstack_1                       | 2022-11-29T14:19:45.339277Z  INFO chirpstack::gateway::backend::mqtt: Sending downlink frame gateway_id=506f980000000012 topic=in865/gateway/506f980000000012/command/down json=true
chirpstack_1                       | 2022-11-29T14:19:47.358891Z  INFO chirpstack::storage::device: Aquiring device lock dev_eui=506f98000000ac63
chirpstack_1                       | 2022-11-29T14:19:47.359567Z ERROR chirpstack::downlink::scheduler: Schedule next queue-item for device failed error=Get device lock

This is how my “chirpstack.toml” config looks like

# Logging.
[logging]

  # Log level.
  #
  # Options are: trace, debug, info, warn error.
  level="info"


# PostgreSQL configuration.
[postgresql]

  # PostgreSQL DSN.
  #
  # Format example: postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>/<DATABASE>?sslmode=<SSLMODE>.
  #
  # SSL mode options:
  #  * disable - no SSL
  #  * require - Always SSL (skip verification)
  #  * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
  #  * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
  dsn="postgres://chirpstack:chirpstack@$POSTGRESQL_HOST/chirpstack?sslmode=disable"

  # Max open connections.
  #
  # This sets the max. number of open connections that are allowed in the
  # PostgreSQL connection pool.
  max_open_connections=10

  # Min idle connections.
  #
  # This sets the min. number of idle connections in the PostgreSQL connection
  # pool (0 = equal to max_open_connections).
  min_idle_connections=0


# Redis configuration.
[redis]

  # Server address or addresses.
  #
  # Set multiple addresses when connecting to a cluster.
  servers=[
    "redis://$REDIS_HOST/",
  ]

  # TLS enabled.
  tls_enabled=false

  # Redis Cluster.
  #
  # Set this to true when the provided URLs are pointing to a Redis Cluster
  # instance.
  cluster=false


# Network related configuration.
[network]

  # Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203).
  net_id="000000"

  # Enabled regions.
  #
  # Multiple regions can be enabled simultaneously. Each region must match
  # the 'name' parameter of the region configuration in '[[regions]]'.
  enabled_regions=[
    "as923",
    "as923_2",
    "as923_3",
    "as923_4",
    "au915_0",
    "cn470_10",
    "cn779",
    "eu433",
    "eu868",
    "in865",
    "ism2400",
    "kr920",
    "ru864",
    "us915_0",
    "us915_1",
  ]


# API interface configuration.
[api]

  # interface:port to bind the API interface to.
  bind="0.0.0.0:8080"

  # Secret.
  #
  # This secret is used for generating login and API tokens, make sure this
  # is never exposed. Changing this secret will invalidate all login and API
  # tokens. The following command can be used to generate a random secret:
  #   openssl rand -base64 32
  secret="you-must-replace-this"


[integration]
  enabled=["mqtt"]

  [integration.mqtt]
    server="tcp://$MQTT_BROKER_HOST:1883/"
    json=true

    # Event topic template.
    event_topic="application/{{application_id}}/device/{{dev_eui}}/event/{{event}}"

    # Command topic.
    #
    # This is the topic on which the MQTT subscribes for receiving (enqueue) commands.
    command_topic="application/{{application_id}}/device/{{dev_eui}}/command/{{command}}"

and confign in “chirpstack-gateway-bridge.toml”

# See https://www.chirpstack.io/gateway-bridge/install/config/ for a full
# configuration example and documentation.

[integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf:  Protobuf encoding
# * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')
marshaler="json"

[integration.mqtt.auth.generic]
servers=["tcp://mosquitto:1883"]
username=""
password=""

[integration.mqtt]
event_topic_template="in865/gateway/{{ .GatewayID }}/event/{{ .EventType }}"
state_topic_template="in865/gateway/{{ .GatewayID }}/state/{{ .StateType }}"
command_topic_template="in865/gateway/{{ .GatewayID }}/command/#"


[backend]

# Backend type.
#
# Valid options are:
#   * semtech_udp
#   * concentratord
#   * basic_station
type="semtech_udp"

[backend.semtech_udp]

# ip:port to bind the UDP listener to
#
# Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
# This is the listener to which the packet-forwarder forwards its data
# so make sure the 'serv_port_up' and 'serv_port_down' from your
# packet-forwarder matches this port.
udp_bind = "0.0.0.0:1700"

# Skip the CRC status-check of received packets
#
# This is only has effect when the packet-forwarder is configured to forward
# LoRa frames with CRC errors.
skip_crc_check = false

# Fake RX timestamp.
#
# Fake the RX time when the gateway does not have GPS, in which case
# the time would otherwise be unset.
fake_rx_time=false

If you can tell me what this error message “Schedule next queue-item for device failed error=Get device lock” means maybe I can do some debugging myself.

I had noticed this error for device id “506f98000000ac63” and then it began appearing for other devices also. Then after deploying new LNS I dont see this error for other devcies and see this for this device id only.

I am using this simulator and I have noticed this messes things up. But havent used this with new deployment but still see this error. I think the simulator happens to send uplink with the device id "“506f98000000ac63” for which we see the error

Thank you.

Regards,
Harsha