[release] ChirpStack v4.7

Announcements
1

v4.7.0

Notes before you upgrade

Device-session migration

This release moves the device-session storage from Redis to PostgreSQL. After upgrading, you must execute the following command (adapted to your environment):

chirpstack -c /etc/chirpstack migrate-device-sessions-to-postgres

This command will iterate over the devices in the PostgreSQL database of which the device-session column is empty and will migrate the device-session from Redis if it exists. This will not overwrite existing device-sessions in PostgreSQL thus it is safe to re-execute this command in case needed.

OpenID Connect / OAuth2

A new authentication backend has been added for OAuth2 based providers (see below). If you are using the OpenID Connect authentication backend, you must update your configuration from:

[user_authentication.openid_connect]
enabled=true

To:

[user_authentication]
enabled="openid_connect"

PostgreSQL CA certificate

If the PostgreSQL server uses TLS, please read the note below with regards to the ca_cert configuration option.

Features

MQTT shared subscription

This by defaults connects to the MQTT broker using a shared subscription name chirpstack. Using a shared subscription, the MQTT broker will send a received uplink only to one subscriber. In case ChirpStack is deployed as a cluster, this removes the overhead caused by all instances receiving the same uplink (which would be correctly handled by the deduplication logic).

In case you have multiple ChirpStack environments (e.g. production, staging and testing) connected to the same MQTT broker, then make sure that each environment has a correct share_name configured in the region_XXXXX.toml configuration.

OAuth2 / Clerk integration

This adds support for integrating with the Clerk authentication backend (OAuth2 interface).

Improvements

Store device-sessions in PostgreSQL

This moves the device-session storage from Redis to PostgreSQL. In case of a high DevAddr re-usage (where multiple DevEUIs share the same DevAddr), the old architecture had a significant overhead, because it would perform a significant amount of Redis queries to retrieve all the potential device-sessions. It would retrieve the DevAddr → DevEUIs mapping, and then retrieve the device-session for each DevEUI. Because device-session data can be sharded (Redis Cluster), a separate query per device-session was
required.

With this improvement, a device-session column has been added to the device table in the PostgreSQL database, which also contains a DevAddr column. The big advantage is that all device-sessions for a given DevAddr can be retrieved using a single query.

This improvements also means that:

  • Device-sessions will no longer expire after device inactivity
  • Device-sessions can be restored from a PostgreSQL backup

Replace OpenSSL with Rustls

This is an internal improvement and removes the OpenSSL dependency in favor of Rustls, which is a pure Rust TLS implementation. This makes the build process easier as it is no longer needed to build a (static) OpenSSL version to link against.

Use async PostgreSQL

This is an internal improvement, and migrates away from pq-sys in favor of tokio-postgres, which is a pure Rust client implementation which works with async. This removes all task::spawn_blocking(...) blocks around SQL queries. As well, we no longer need to static link against libpq (C library, with dependency on OpenSSL).

Important note: this also adds a new ca_cert option to the [postgresql] configuration section where you can configure the CA certificate which must be used for validating the PostgreSQL server certificate (if not already provided by the host system).

Use async Redis

This is an internal improvement and updates all Redis queries to use async / await and removes all task::spawn_blocking(...) blocks around Redis queries.

Paho MQTT > rumqttc client

This is an internal improvement and replaces paho-mqtt with rumqttc. The latter crate is a pure Rust client which uses rustls for TLS instead of OpenSSL.

Other

  • Update dependencies.
  • Return Redis connection to the pool immediately after query completion.
  • Return PostgreSQL client to the pool immediately after query completion.
6 Likes
2

Thank you for the update!

Do you have an example on how to run the migration command using docker?

Regards

3

You should be able to use docker exec to run it from within the running ChirpStack container.

4

this should work from from your machine shell without the need to go into the container: docker exec <chirpstack-container-name> chirpstack -c /etc/chirpstack migrate-device-sessions-to-postgres

3 Likes
6

Thank you for the update!

I would recommend enabling custom providers for OAuth2. I had to fake my OAuth2 server provider name exclusively for Chirpstack to enable login via OAuth2, but it would be more appropriate if I could use my real provider name and simply define that user_id is sub for everything to work correctly.

Essentially, the configuration needs:

# Provider.
#
# Supported options by Chirpstack are:
#  * clerk
#
provider="CustomProvider"

# .... other options

# OAuth2 ID Claim. (Can be empty, if using supported provider)
user_id="sub"
7

Ideally, you would use OpenID Connect instead of OAuth2, because OIDC specifies the complete authentication flow including the claims that the authentication provider must provide. OAuth2 only specifies how to obtain access to resource(s) using a token, but it does not specify the content of these resource(s). OpenID Connect is based on OAuth2 and specifies the complete flow. For some reason, Clerk has decided to only implement OAuth2 (Auth0 for example can be used directly using OIDC).

Per provider there might as well be minor differences in the OAuth2 configuration, e.g. the default OAuth2 configuration (used by the Rust crate), does not work work Clerk, but works with other providers. E.g. when implementing the Clerk integration I could not get it work initially, after doing lots of debugging I found out that this was the fix: chirpstack/chirpstack/src/api/oauth2.rs at master · chirpstack/chirpstack · GitHub.

So I think it is better to define a list of known providers and based on the configured provider handle these potential minor changes automatically rather than exposing all these parameters to the end-user.

1 Like
8

Hi there :wave:,

Trying to upgrade from 4.6.0 to 4.7.0, I get the following error:

Error: Error occurred while creating a new object: invalid connection string: unknown option sslcert

My connection string is:

postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>/<DATABASE>?sslcert=/etc/chirpstack/pg-certs/client-cert.pem&sslkey=/etc/chirpstack/pg-certs/client-key.pem&sslrootcert=/etc/chirpstack/pg-certs/server-ca.pem&sslmode=require

Where/how should I enter the path to my certificate?

9

Am I the only one to have this problem or is it due to the fact that I’m the only one using client certificates :joy:

1 Like
10

Please see chirpstack/chirpstack/src/cmd/configfile.rs at master · chirpstack/chirpstack · GitHub. Some config options have changed.

11

Hi @orne,
I saw, but I still don’t have any possibility to configure a client certificate.

12

This was never an official / documented ChirpStack feature to use client-certificates for authentication / authorization… This “undocumented feature” might indeed not be available after the refactor of the database code…

13

Probably, this would need some code changes here:

14

What happens if the device sessions are not transferred? Will the existing devices eventually re-register?

15

If the device firmware is smart enough to rejoin after a period of failed communication, yes. But I’ve also seen devices in the field that are not that smart.

1 Like
16

The devices unfortunately do not rejoin. The only solution is probably to trigger the devices so that they initiate a join.