Passive Roaming (v4 Help)

I’m trying to setup a passive roaming with TTN and have gone through the steps on their site and emails with johan for signing and the certs for them.

I have generated the certs as per the instructions given but on pulling up the containers chirpstack dies with an error, yet the certs are there and populated… is it a permissions or owner thing?

2023-10-27T08:20:10.258204Z  INFO chirpstack::api::backend: Setting up backend interfaces API bind=0.0.0.0:8181
thread 'tokio-runtime-worker' panicked at 'error binding to 0.0.0.0:8181: key contains no private key', /cargo/registry/src/index.crates.io-6f17d22bba15001f/warp-0.3.5/src/server.rs:534:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: task 23 panicked

Interestingly if i replace the server.key with one thats generated for the mqtt server i do not get this error, is the key type generated for pb not compatible with the rust library used on v4?

Does the PB signing processes generate .crt ?
MQTT/mosquitto uses .pem

You can try converting .cert to .pem using openssl
I don’t know the command so will have to google it

I do use a domain level reverse proxy (Nginx Proxy Manager) with letsencrypt certs.

I created all the files on this page:

as i reverse proxy so I dont know what certificates it really wants here if i didnt generate the ones it says not too as it seems to want self signed you need to make during the process it says to skip if you use letsencrypt.

Hum just thinking out aloud backend interface on 8181 it servers TLS based on certs for Roaming JS etc. So it doesn’t need a reverse proxy. So given error is it worth converting just the .key to xxx-key.pem & try that.

Anything I have tried to convert the server.key file too just fails to convert the key or throws the same error on starting the container :frowning:

idk how i go about it right now, but it looks like I need to make all the certificates as rsa 4096 and sign them with the ca from the first step…

with pem certs not signed by the right ca… i get this.

2023-10-27T13:01:15.078230Z  INFO stats{gateway_id=b827ebfffec9b93e}: chirpstack::storage::metrics: Metrics saved name=gw:b827ebfffec9b93e aggregation=MONTH
2023-10-27T13:01:17.117876Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="au915_1" topic="au915_1/gateway/dca632fffe6ab3dd/event/up" qos=0 json=false
2023-10-27T13:01:17.320977Z  INFO up{deduplication_id=dd23b7df-d521-4886-8352-62cf35a86c6a}: chirpstack::uplink: Uplink received m_type="UnconfirmedDataUp"
2023-10-27T13:01:17.323791Z  INFO up{deduplication_id=dd23b7df-d521-4886-8352-62cf35a86c6a}:data_up:data_pr: chirpstack::uplink::data_fns: Starting passive-roaming session net_id=000013 dev_addr=260d9988
2023-10-27T13:01:17.323913Z  INFO up{deduplication_id=dd23b7df-d521-4886-8352-62cf35a86c6a}:data_up:data_pr: backend: Making request receiver_id=000013 transaction_id=1678576893 message_type=PRStartReq server=https://apac.packetbroker.io:5138 async_interface=true
2023-10-27T13:01:18.042714Z ERROR up{deduplication_id=dd23b7df-d521-4886-8352-62cf35a86c6a}:data_up:data_pr: chirpstack::uplink::data_fns: Start passive-roaming error net_id=000013 error=HTTP status client error (401 Unauthorized) for url (https://apac.packetbroker.io:5138/)

The rust code crashes if the certs are not in the same format as what’s used to generate the basicstation or mqtt tls certs in the v4 docs and to get any response the port had to be 8181 for the backend api Mosquitto TLS configuration - ChirpStack open-source LoRaWAN® Network Server documentation

i adjusted the above to generate 5 year certs - but they’re not signed by the ttn ca.

with them if a ttn packet arrives i get a 401 unauthorised so im stuck atm knowing how to generate compatible certs with the existing ttn ca using the cfssl tool.

If anyone is familiar with how to do this please let me know :pray:

2023-10-27T22:46:23.229011Z  INFO chirpstack::api::backend: Setting up backend interfaces API bind=0.0.0.0:8181
thread 'tokio-runtime-worker' panicked at 'error binding to 0.0.0.0:8181: key contains no private key', /cargo/registry/src/index.crates.io-6f17d22bba15001f/warp-0.3.5/src/server.rs:534:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: task 22 panicked

private key but truncated… so its there.

-----BEGIN EC PARAMETERS-----
...
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
...
...
...
-----END EC PRIVATE KEY-----

You should just be able to convert the certs issued by TTN CA.

openssl x509 -in cert.crt -out cert.pem

I’ve done this for a MQTT broker that needed .pem & our CA only issued .crt
So I know that works in that instance.

Unfortunately this didn’t seem to work for me.
anything generated with EC even after conversion just fails or chirp says it doesn’t exist or no private key like the error above even though the file is there and mapped.

Will have to revisit later as i’m really at a loss here as to why this is the case.

If it helps:
I used a cert/key pair, generated by letsencrypt (which uses RSA).

I bound the backend-interface to port 5138… are you shure, that 8181 isn’t already used on your system?

[backend_interfaces] 
bind="0.0.0.0:5138"
ca_cert="/etc/chirpstack/certs/userca/ca.crt"
tls_cert="/etc/chirpstack/certs/tls/tls.crt"
tls_key="/etc/chirpstack/certs/tls/tls.key"

[roaming]
#resolve_netid_domain_suffix=".netids.lora-alliance.org"
[[roaming.servers]]
net_id="000013"
async_timeout="30s"
passive_roaming_lifetime="0s"
server="https://eu.packetbroker.io:5138"
authorization_header="Key <KeyID>.<Token>"

8181 is definitely not used already :smiley:

I just generated them all in rsa 4096 and they seem to work without crashing.
I can try grab the ones in my reverse proxy container but figured this would be easier.

are they all the settings you had to set?

I have the exact same setup now and it still errors. i dont know what to do from here.

do you have a netid assigned to your instance? I have a helium netid i wonder if this is causing problems somehow but its only supposed to be a forwarder etc for ttn on 000000 to 000013.

using certs generated as RSA was key though as warp.rs just shits itself if they’re not coded that way.


how does it setup the passive roaming interface?
Is there anywhere to setup a join server and prefix?

from what i can tell it seems to join here

cory@dred:~/chirpstack-docker$ docker logs -f chirpstack-docker-chirpstack-1 | grep https://apac.packetbroker.io:5138
2023-10-31T10:14:00.708591Z  INFO setup{net_id=000013}: chirpstack::backend::roaming: Configuring roaming client passive_roaming_lifetime=0ns server=https://apac.packetbroker.io:5138 async_timeout=30s
--
2023-10-31T10:27:00.463034Z  INFO up{deduplication_id=aebde3a3-363d-46b5-a4a9-f526165a4210}:join_request: chirpstack::uplink::join: Unknown device, trying passive-roaming activation dev_eui=70b3d5499e39ba23 join_eui=70b3d57ef0005f34
2023-10-31T10:27:00.463137Z ERROR up{deduplication_id=aebde3a3-363d-46b5-a4a9-f526165a4210}: chirpstack::uplink::join: Handle join-request error error=Join Server client for join_eui 70b3d57ef0005f34 does not exist

but then any devices trying to join ttn just error still :frowning:

cory@dred:~/chirpstack-docker$ docker logs -f chirpstack-docker-chirpstack-1 | grep ERROR
2023-10-31T10:14:18.449555Z ERROR up{deduplication_id=718a27ba-ae31-4e1f-ac0a-b4013f60a000}: chirpstack::uplink::join: Handle join-request error error=Join Server client for join_eui 70b3d57ef0005f34 does not exist
2023-10-31T10:14:33.445876Z ERROR up{deduplication_id=af8d7df7-33a3-4502-ba43-cc0be6e08740}: chirpstack::uplink::join: Handle join-request error error=Join Server client for join_eui 70b3d57ef0005f34 does not exist
2023-10-31T10:14:48.447630Z ERROR up{deduplication_id=4232cea1-0aef-4bf8-8b7d-bc743decde2e}: chirpstack::uplink::join: Handle join-request error error=Join Server client for join_eui 70b3d57ef0005f34 does not exist
2023-10-31T10:15:03.442469Z ERROR up{deduplication_id=92556a3c-8979-493c-b9fe-e755ead7114c}: chirpstack::uplink::join: Handle join-request error error=Join Server client for join_eui 70b3d57ef0005f34 does not exist

As far as I know, join-requests are currently not roamed, this is a limitation. Only data frames with a foreign netid (netid from devid != own netid) will be roamed.

so you think it will only pass already joined devices at the moment?
(like devices we hear that are already joined to ttn)

We added a gw to ttn so it would join the device, once joined now its throwing a 403 on the passive roaming url. Is the forward slash being added in the backend causing this issue perhaps?

2023-10-31T11:52:03.410177Z  INFO up{deduplication_id=6a71dda4-db98-4f9f-b71e-3b64a104fe5e}:data_up:data_pr: chirpstack::uplink::data_fns: Starting passive-roaming session net_id=000013 dev_addr=260dd21c
2023-10-31T11:52:04.073046Z ERROR up{deduplication_id=6a71dda4-db98-4f9f-b71e-3b64a104fe5e}:data_up:data_pr: chirpstack::uplink::data_fns: Start passive-roaming error net_id=000013 error=HTTP status client error (403 Forbidden) for url (https://apac.packetbroker.io:5138/)
2023-10-31T11:55:43.175186Z  INFO up{deduplication_id=3919869e-e565-445b-9c50-db7dcd99d5e6}:data_up:data_pr: chirpstack::uplink::data_fns: Starting passive-roaming session net_id=000013 dev_addr=260dd21c
2023-10-31T11:55:43.926031Z ERROR up{deduplication_id=3919869e-e565-445b-9c50-db7dcd99d5e6}:data_up:data_pr: chirpstack::uplink::data_fns: Start passive-roaming error net_id=000013 error=HTTP status client error (403 Forbidden) for url (https://apac.packetbroker.io:5138/)
2023-10-31T11:59:23.452357Z  INFO up{deduplication_id=2911c98f-0ef1-4d41-9b14-f6af95397ecc}:data_up:data_pr: chirpstack::uplink::data_fns: Starting passive-roaming session net_id=000013 dev_addr=260dd21c
2023-10-31T11:59:23.840098Z ERROR up{deduplication_id=2911c98f-0ef1-4d41-9b14-f6af95397ecc}:data_up:data_pr: chirpstack::uplink::data_fns: Start passive-roaming error net_id=000013 error=HTTP status client error (403 Forbidden) for url (https://apac.packetbroker.io:5138/)

this is whats being passed in settings.

[backend_interfaces]
  bind="0.0.0.0:5138"
  ca_cert="/etc/chirpstack/certs/ca.crt"
  tls_cert="/etc/chirpstack/certs/server_full.crt"
  tls_key="/etc/chirpstack/certs/server.key"

[roaming]
  # resolve_netid_domain_suffix=".netids.lora-alliance.org"
  [[roaming.servers]]
    net_id="000013"
    async_timeout="30s"
    # passive_roaming=true
    passive_roaming_lifetime="0s"
    server="https://apac.packetbroker.io:5138"
    authorization_header="Key mykey.here"

I think that is the case see this from Orne

I think that might still be the case yeh.

Just been going back and forth with johan, the other issue is that i have a netid set on the instance that they dont want to route (00003c) so that maybe the 403 error perhaps as there may have been a misunderstanding on my part on how it works on their end - was assuming it would send all ttn stuff to them if it seen it.

another handy feature would be the ability to set netid per tenant. the multi region saved us for a bit but it seems we’re needing to run multiple instances to get around the netid’s now :frowning:

Bit closer now, but last stumbling block fingers crossed is to work out what triggers the error channel closed on trying to send a uplink.

2023-11-01T11:15:11.139453Z  INFO up{deduplication_id=20599d5b-a44e-4386-bdca-83b6ca39c93a}:data_up:data_pr: chirpstack::uplink::data_fns: Starting passive-roaming session net_id=000013 dev_addr=260dd1b7
2023-11-01T11:15:41.202635Z ERROR up{deduplication_id=20599d5b-a44e-4386-bdca-83b6ca39c93a}:data_up:data_pr: chirpstack::uplink::data_fns: Start passive-roaming error net_id=000013 error=channel closed

any ideas?

current settings…

[backend_interfaces]
  bind="0.0.0.0:5138"
  ca_cert="/etc/chirpstack/certs/ca.crt"
  tls_cert="/etc/chirpstack/certs/server_full.crt"
  tls_key="/etc/chirpstack/certs/server.key"

[roaming]
  #resolve_netid_domain_suffix=".netids.lora-alliance.org"
  [[roaming.servers]]
    net_id="000013"
    async_timeout="30s"
    passive_roaming_lifetime="0s"
    server="https://apac.packetbroker.io:5138"
    authorization_header="Key TOKEN.HERE"