Problem with Lorawan to Chirpstack Latest v3 Upgrade of App Server: Dirty Database Version 1

1

Hey all,

I posted a while back about an upgrade I was taking on of some old versions:

There is documentation on v3 latest to v4, but not any on old v3 to latest v3. Understandable as it’s a bit dated now.

I managed to upgrade the loraserver 3.0.2 and lora-gateway-server 3.0.1 to the latest chirpstack v3 versions (at time of update). They are running and means gateway is now 3.14.6 and network server is 3.16.6. Due to rebrand, I did have to move some configuration files around to the new chirpstack locations, but aside from that it went well.

I am now however trying to upgrade the lora-app-server from 3.1.0 to the chirpstack-app-server 3.17.9. I have done similar to above with replacing binaries and config location. It starts, however outputs the following error:

Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.704535505Z" level=info msg="starting ChirpStack Application Server" docs="https://www.chirpstack.io/" version=3.17.9
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.70459616Z" level=info msg="storage: setting up storage package"
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.704602691Z" level=info msg="storage: setup metrics"
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.704610708Z" level=info msg="storage: setting up Redis client"
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.704634323Z" level=info msg="storage: connecting to PostgreSQL database"
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.708037247Z" level=info msg="storage: applying PostgreSQL data migrations"
Dec 20 13:07:15 chirpstack-app-server: time="2022-12-20T13:07:15.710883076Z" level=fatal msg="setup storage error: storage: migrate up error: Dirty database version 1. Fix and force version."

Forgive me as I know there are other posts regarding this, but I’m no postgres expert, and I also wasn’t expecting a ‘data migration’ at this stage. There is mention of a migration when going from v3 to v4 in the below link which isn’t what I’m doing:

https://www.chirpstack.io/docs/v3-v4-migration.html#chirpstack-v4

Can someone advise what steps need to happen here if known? I don’t want to run any v4 migration steps from the doc above unless advised, as I haven’t gotten to the moving to v4 part yet. I’m doing the initially requested part of upgrading everything to latest v3 (and as a result, converting to chirpstack from lorawan branded services).

Thanks in advance, and sorry for the long post.

Darren

2

I went to Postgres and got the following out of the log if it helps. Postgres is v9.5:

< 2022-12-20 17:05:43.673 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:43.673 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:43.673 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:43.673 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:43.959 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:43.959 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:43.960 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:43.960 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.207 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.207 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.207 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.207 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.458 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.458 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.458 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.458 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.708 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.708 CET >STATEMENT:  lock table code_migration
< 2022-12-20 17:05:44.709 CET >ERROR:  relation "code_migration" does not exist
< 2022-12-20 17:05:44.709 CET >STATEMENT:  lock table code_migration
3

The first time when you get this error, there should have been a description which part of the schema migration code failed.

A common issue has been forgetting to add one of the extensions, e.g.:

create extension hstore;

In which case one of the migrations will fail because it can not create a hstore column.

In general, the best way to upgrade from a very old v3 to the latest v3 is to refer to the changelog.

4

Thanks Brocaar, good to know. I will look into that suggestion and bear change-log in mind.

A common issue has been forgetting to add one of the extensions, e.g.:

When you say forgetting, is there somewhere I’d see this mention of adding a hstore extension? Or is that the sort of thing you’d advise me to look into change-logs to find out?

5

It’s in the docs here:

https://www.chirpstack.io/docs/chirpstack/requirements.html#create-database

If you use one of the docker-compose templates, it is included in the DB init scripts.

6

Thanks @bconway. I suppose I meant in the context of upgrading through version three, as I can’t be sure what the software will handle itself from the upgrading, versus things I need to perform myself.

I’ll start by giving the step provided a go after review of the db, and then move onto change-log review otherwise to identify anything that needs to be done manually.

7

As @bconway mentioned, it is in the docs. Historically it wasn’t there and it was mentioned in the changelog that an additional step was needed before upgrading.

Probably the first time when you tried to run the update ChirpStack, the error contained an error that it could not create a column with type hstore, because the extension did not exist, after which the migration was marked as dirty.