JoinRequest then JoinAccept but no Join message

I have Chirpstack running in production. I wanted to test some changes on a development system. I cloned the Debian production system and installed in on my development system. The development system sees JoinRequests and responds with JoinAccept, but I never see the Join message. Any suggestions?

The only thing odd that I can find is the Application session key that is generated is set to 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 after adding the new end-point.

I have now tried rebuilding the entire development system from scratch but I am still seeing the same results. I am using a Laird RG1xx with firmware Laird Linux gatwick-laird-93.8.5.21. I am using Microchip WLR089U0 running their MLS_SDK_1_0_P_5 firmware.

Here is my join sequence:
mac reset na915

mac set ch status 8 off

ok

mac set ch status 9 off

ok

mac set ch status 10 off

ok

mac set ch status 11 off

ok

mac set ch status 12 off

ok

mac set ch status 13 off

ok

mac set ch status 14 off

ok

mac set ch status 15 off

ok

mac set ch status 16 off

ok

mac set ch status 17 off

ok

mac set ch status 18 off

ok

mac set ch status 19 off

ok

mac set ch status 20 off

ok

mac set ch status 21 off

ok

mac set ch status 22 off

ok

mac set ch status 23 off

ok

mac set ch status 24 off

ok

mac set ch status 25 off

ok

mac set ch status 26 off

ok

mac set ch status 27 off

ok

mac set ch status 28 off

ok

mac set ch status 29 off

ok

mac set ch status 30 off

ok

mac set ch status 31 off

ok

mac set ch status 32 off

ok

mac set ch status 33 ok

off

mac set ch status 34 off

ok

mac set ch status 35 off

ok

mac set ch status 36 off

ok

mac set ch status 37 off

ok

mac set ch status 38 off

ok

mac set ch status 39 off

ok

mac set ch status 40 off

ok

mac set ch status 41 off

ok

mac set ch status 42 off

ok

mac set ch status 43 off

ok

mac set ch status 44 off

ok

mac set ch status 45 off

ok

mac set ch status 46 off

ok

mac set ch status 47 off

ok

mac set ch status 48 off

ok

mac set ch status 49 off

ok

mac set ch status 50 off

ok

mac set ch status 51 off

ok

mac set ch status 52 off

ok

mac set ch status 53 off

ok

mac set ch status 54 off

ok

mac set ch status 55 off

ok

mac set ch status 56 off

ok

mac set ch status 57 off

ok

mac set ch status 58 off

ok

mac set ch status 59 off

ok

mac set ch status 60 off

ok

mac set ch status 61 off

ok

mac set ch status 62 off

ok

mac set ch status 63 off

ok

mac set ch status 65 off

ok

mac set ch status 66 off

ok

mac set ch status 67 off

ok

mac set ch status 68 off

ok

mac set ch status 69 off

ok

mac set ch status 70 off

ok

mac set subband status 1 on

ok

sys get hweui

b191c6feff1bebea

ok

mac set joineui 0000000000000000

ok

mac set deveui b191c6feff1bebea

ok

mac set appkey b191c6feff1bebea0000000000000000

ok

mac set devaddr 00000000

ok

mac save

ok

mac join otaa

denied

ok

no_free_ch
# This configuration provides a Semtech UDP packet-forwarder backend and
# integrates with a MQTT broker. Many options and defaults have been omitted
# for simplicity.
#
# See https://www.chirpstack.io/gateway-bridge/install/config/ for a full
# configuration example and documentation.


# Gateway backend configuration.
[backend]
# Backend type.
type="semtech_udp"

  # Semtech UDP packet-forwarder backend.
  [backend.semtech_udp]

  # ip:port to bind the UDP listener to
  #
  # Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
  # This is the listener to which the packet-forwarder forwards its data
  # so make sure the 'serv_port_up' and 'serv_port_down' from your
  # packet-forwarder matches this port.
  udp_bind = "0.0.0.0:1700"


# Integration configuration.
[integration]
# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * protobuf:  Protobuf encoding
# * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')
marshaler="protobuf"

  # MQTT integration configuration.
  [integration.mqtt]
  # Event topic template.
  event_topic_template="gateway/{{ .GatewayID }}/event/{{ .EventType }}"

  # Command topic template.
  command_topic_template="gateway/{{ .GatewayID }}/command/#"

  # MQTT authentication.
  [integration.mqtt.auth]
  # Type defines the MQTT authentication type to use.
  #
  # Set this to the name of one of the sections below.
  type="generic"

    # Generic MQTT authentication.
    [integration.mqtt.auth.generic]
    # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
    server="tcp://127.0.0.1:1883"

    # Connect with the given username (optional)
    username=""

    # Connect with the given password (optional)
    password=""
# This configuration configures ChirpStack Network Server for the US915 band using a MQTT
# broker to communicate with the gateways. Many options and defaults have been
# omitted for simplicity.
#
# For other bands, see the ./examples/ sub-directory.
#
# See https://www.chirpstack.io/network-server/install/config/ for a full
# configuration example and documentation.

[general]
log_level=4

# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=chirpstack_ns dbname=chirpstack_ns sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
#dsn="postgres://localhost/chirpstack_ns_ns?sslmode=disable"
 dsn="postgres://chirpstack_ns:dbpassword@localhost/chirpstack_ns?sslmode=disable"


# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# Redis url (e.g. redis://user:password@hostname/0)
#
# For more information about the Redis URL format, see:
# https://www.iana.org/assignments/uri-schemes/prov/redis
url="redis://localhost:6379"


# Network-server settings.
[network_server]
# Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203)
net_id="000000"


  # LoRaWAN regional band configuration.
  #
  # Note that you might want to consult the LoRaWAN Regional Parameters
  # specification for valid values that apply to your region.
  # See: https://www.lora-alliance.org/lorawan-for-developers
  [network_server.band]
    name="US_902_928"
    #name="US915"

    # LoRaWAN network related settings.
    [network_server.network_settings]
    downlink_tx_power=18

    # Enable only a given sub-set of channels
    #
    # Use this when ony a sub-set of the by default enabled channels are being
    # used. For example when only using the first 8 channels of the US band.
    # Note: when left blank, all channels will be enabled.
    enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7, 64]
    #enabled_uplink_channels=[8, 9, 10, 11, 12, 13, 14, 15, 65]

    #rx_frequency=923300000
    #rx_frequency=923900000

    # Class B settings
    #[network_server.network_settings.class_b]
    # Ping-slot data-rate.
   # ping_slot_dr=0

    # Ping-slot frequency (Hz)
    #
    # Set this to 0 to use the default frequency plan for the configured region
    # (which could be frequency hopping).
   # ping_slot_frequency=0


  # Network-server API
  #
  # This is the network-server API that is used by ChirpStack Application Server or other
  # custom components interacting with ChirpStack Network Server.
  [network_server.api]
  # ip:port to bind the api server
  bind="0.0.0.0:8000"


  # Backend defines the gateway backend settings.
  #
  # The gateway backend handles the communication with the gateway(s) part of
  # the LoRaWAN network.
  [network_server.gateway.backend]
  # Backend
  type="mqtt"


    # MQTT gateway backend settings.
    #
    # This is the backend communicating with the LoRa gateways over a MQTT broker.
    [network_server.gateway.backend.mqtt]
    # MQTT topic templates for the different MQTT topics.
    #
    # The meaning of these topics are documented at:
    # https://www.chirpstack.io/gateway-bridge/
    #
    # The default values match the default expected configuration of the
    # ChirpStack Gateway Bridge MQTT backend. Therefore only change these values when
    # absolutely needed.

    # Event topic template.
    event_topic="gateway/+/event/+"

    # Command topic template.
    #
    # Use:
    #   * "{{ .GatewayID }}" as an substitution for the LoRa gateway ID
    #   * "{{ .CommandType }}" as an substitution for the command type
    command_topic_template="gateway/{{ .GatewayID }}/command/{{ .CommandType }}"

    # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
    server="tcp://localhost:1883"

    # Connect with the given username (optional)
    username=""

    # Connect with the given password (optional)
    password=""


# Metrics collection settings.
[metrics]
# Timezone
#
# The timezone is used for correctly aggregating the metrics (e.g. per hour,
# day or month).
# Example: "Europe/Amsterdam" or "Local" for the the system's local time zone.
timezone="Local"


# Join-server settings.
[join_server]

  # Default join-server settings.
  #
  # This join-server will be used when resolving the JoinEUI is set to false
  # or as a fallback when resolving the JoinEUI fails.
  [join_server.default]
  # hostname:port of the default join-server
  #
  # This API is provided by ChirpStack Application Server.
  server="http://localhost:8003"
# This configuration sets the required settings and configures an integration
# with a MQTT broker. Many options and defaults have been omitted for
# simplicity.
#
# See https://www.chirpstack.io/application-server/install/config/ for a full
# configuration example and documentation.

[general]
log_level=4

# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=chirpstack_as dbname=chirpstack_as sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
#dsn="postgres://localhost/chirpstack_as?sslmode=disable"
dsn="postgres://chirpstack_as:dbpassword@localhost/chirpstack_as?sslmode=disable"



# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# Redis url (e.g. redis://user:password@hostname/0)
#
# For more information about the Redis URL format, see:
# https://www.iana.org/assignments/uri-schemes/prov/redis
url="redis://localhost:6379"


# Application-server settings.
[application_server]

  # Integration configures the data integration.
  #
  # This is the data integration which is available for all applications,
  # besides the extra integrations that can be added on a per-application
  # basis.
  [application_server.integration]
  # Payload marshaler.
  #
  # This defines how the MQTT payloads are encoded. Valid options are:
  # * protobuf:  Protobuf encoding
  # * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')
  # * json_v3:   v3 JSON (will be removed in the next major release)
  marshaler="json_v3"

  # Enabled integrations.
  enabled=["mqtt"]

    # MQTT integration backend.
    [application_server.integration.mqtt]
    # Event topic template.
    event_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/event/{{ .EventType }}"

    # Command topic template.
    command_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/command/{{ .CommandType }}"

    # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
    server="tcp://localhost:1883"

    # Connect with the given username (optional)
    username=""

    # Connect with the given password (optional)
    password=""


    # Settings for the "internal api"
    #
    # This is the API used by ChirpStack Network Server to communicate with ChirpStack Application Server
    # and should not be exposed to the end-user.
    [application_server.api]
    # ip:port to bind the api server
    bind="0.0.0.0:8001"

    # Public ip:port of the application-server API.
    #
    # This is used by ChirpStack Network Server to connect to ChirpStack Application Server. When running
    # ChirpStack Application Server on a different host than ChirpStack Network Server, make sure to set
    # this to the host:ip on which ChirpStack Network Server can reach ChirpStack Application Server.
    # The port must be equal to the port configured by the 'bind' flag
    # above.
    public_host="localhost:8001"


    # Settings for the "external api"
    #
    # This is the API and web-interface exposed to the end-user.
    [application_server.external_api]
    # ip:port to bind the (user facing) http server to (web-interface and REST / gRPC api)
    bind="0.0.0.0:8080"

    # http server TLS certificate (optional)
    tls_cert=""

    # http server TLS key (optional)
    tls_key=""

    # JWT secret used for api authentication / authorization
    # You could generate this by executing 'openssl rand -base64 32' for example
    jwt_secret="6eSV3yY9PL9GIaSj1sgPJ+RCFoWTwCGDRjrbuGekFqU="


# Join-server configuration.
#
# ChirpStack Application Server implements a (subset) of the join-api specified by the
# LoRaWAN Backend Interfaces specification. This API is used by ChirpStack Network Server
# to handle join-requests.
[join_server]
# ip:port to bind the join-server api interface to
bind="0.0.0.0:8003"
#
# JavaScript codec settings.
[application_server.codec.js]
# Maximum execution time.
max_execution_time="200ms"

Just found the following in the logs:
INFO[0040] finished unary call with code NotFound ctx_id=9c48d3a0-a731-4485-bf75-b996c6ba43ba error=“rpc error: code = NotFound desc = object does not exist” grpc.code=NotFound grpc.method=GetDeviceActivation grpc.service=ns.NetworkServerService grpc.start_time=“2021-09-24T05:37:22Z” grpc.time_ms=8.389 peer.address=“127.0.0.1:53312” span.kind=server system=grpc

Does this help knowing what is wrong? It may be do to my trying to get it working.

Is the device in the production network? Some device I think is joined in “some” network. The device sends a message (after the join, there is only the DevAddr) and your network-server is trying to match that against a known devAddr to find the proper device_eui with keys, but I might be wrong.

We have one installation running in production with several end-points that configured ok.

The exact same settings are on this unit, our development unit, and it is not working.

I meant, if it’s both prod and your test network. If it’s in the database.

I do not understand your suggestion.

We had deployed a system to production in another location with several end-points and a Laird Gateway running on a Raspberry Pi (Originally we used a Boundary Devices Nitrogen running an iMX8).

I took a Raspberry Pi image of production and deployed in on a second Raspberry Pi in the office with another Laird Gateway and new end-points. The new end-points will not Join with this configuration. I have verified all keys, and still get the same issue with several different end-points.

I even rebuild the entire system from scratch and the same problem occurs.

I not suggesting anything, I’m asking you. Is it a new clean device that never joined a Network? The devaddr with only zeros does not look good either, I dont think multiple devices should have the same devAddr , but I dont know if your device documentation says otherwise. Make sure every device you want to add, is deleted from the other network. Look at the Frames the gateway is uploading.

For OTAA, devAddr is dynamically assigned in the join accept, so the one set manually before doing the join request would be overwritten before it was ever used.

Technically devAddr does not need to be unique, rather it’s the tuple of devAddr and the network session key that does. But of course for simplicity’s sake, a join server would hand out distinct addresses.

Until there’s a received join accept, no devAddr matters.

I know that, and collisions will someday happen. Should not != must not
The NS should handle the collision case just fine, I though maybe something is danggling in redis or a DB and they all point to the same devaddr and somehow the NS is getting “confused” which device it is.

I think he said he is getting a Join-Accept, but the device is not.

The development system sees JoinRequests and responds with JoinAccept, but I never see the Join message . Any suggestions?

Yes I get a Join Accept but no Accept.

That’s kind of confused.

It sounds like you’re saying a “join accept” is generated but not received at the node, or at least not processed there to the point that a message gets printed saying it has joined.

I see Join Accept message in LORAWAN FRAMES but I do not see the Join in DEVICE DATA.
The end-device shows a denied message when it performs the OTAA.

Node stacks really shouldn’t say such misleading things.

There’s no such thing as “denial” in LoRaWan, there is only “lack of success” - which is more likely a passive failure than a rejection, though both failure and silent rejection are possible.

There are no active rejection transmissions.

Could it be that your cloned Dev system is acting as a join server, but is responding to tell the device to join the production system ?