Database error with multiple ChirpStack v4 instances

[Guilleojeda]

I’m trying to set up multiple ChirpStack v4 instances, in an auto-scaling group in AWS (that means if more traffic comes, new instances will be created, and if less traffic comes, instances will be deleted).
All instances are using the same database (not just database instance, database), which is hosted in AWS RDS.

The issue I’ve encountered is that the first instance of ChirpStack will run just fine, but the second one will exit with the following error:

Applying schema migrations
, stderr: Error: duplicate key value violates unique constraint "pg_type_typname_nsp_index"

As I understand, this happens because the second instance tries to apply migrations to the database as if it were clean (i.e. it tries to perform all the necessary setup), but that’s already done by the first instance. In that case, I’d need to either skip the migrations or make them not fail (e.g. create if not exists).

However, this question from ChirpStack v3 is about the same issue, and @brocaar 's response was to create multiple databases, one for each Network Server. I’m not sure whether that still applies to ChirpStack v4, which merged the NS and the AS. Also, I’m only using a single region, a single configuration, and I’m not interested in distributing my instances across geographical locations or AWS regions.

If I created a new database for each instance, my questions are:

• Will I be able to access all the data from all the databases from the same URL?
• Will the data be replicated across all databases? How much data is this, and how should I plan for, say, 10 ChirpStack instances?
• What happens to the data when (not if, when) an instance is deleted? Will I still be able to access it? Do I have to take any extra steps to access it?

[brocaar]

@Guilleojeda

As I understand, this happens because the second instance tries to apply migrations to the database as if it were clean (i.e. it tries to perform all the necessary setup), but that’s already done by the first instance. In that case, I’d need to either skip the migrations or make them not fail (e.g. create if not exists).

There is a difference here between the v3 recommendation and your current setup. In your case the multiple ChirpStack instances act as a cluster.

There should be any issue, as the migration state is stored in the __diesel_schema_migrations table. For example, this table contains:

chirpstack=> select * from __diesel_schema_migrations;
    version     |           run_on           
----------------+----------------------------
 00000000000000 | 2022-10-04 14:33:48.232427
 20220426153628 | 2022-10-04 14:33:48.479417
 20220428071028 | 2022-10-04 14:33:48.483597
 20220511084032 | 2022-10-04 14:33:48.486645
 20220614130020 | 2022-10-04 14:33:48.520536
 20221102090533 | 2022-11-02 09:33:21.415878

Therefore, the other ChirpStack instances connecting to the same database should not try to re-run the migrations, as they will detect that these migrations have already been applied.