First of all I would like to wish everybody a Happy New year. The last year has been a great year for the ChirpStack project and many new features were added. I would like to thank the ChirpStack community for all the changes that have been contributed. Thank you!
With this post I want to give a brief overview of what I’ve been working on the past few months, and what will be released later this year.
When starting the project, I decided to have a Network Server and Application Server component. At the time this was introduced, this made a lot of sense (to me), but more and more this is becoming a bottleneck both for setup as for development. Looking at the setup part, it requires quite a lot of effort when you want to support multiple (and in some cases all) regions. Looking at the development, many (some even very simple) changes required changes to several repositories, slowing down development and making some pull-requests more complex than needed. A few months ago, I therefore decided to start merging the ChirpStack Network Server and ChirpStack Application Server. This code is not yet public, but of course will be made open-source like the current components.
Some of the changes that you are going to see as a user:
Simplified setup as there is only one service to setup. This also means that there will be just one service to install. Many recurring issues that have been reported were related to connecting the ChirpStack Application Server with the ChirpStack Network Server or database issues because both services were configured against the same database.
Simplified usage as some of the “bootstrapping” steps will be removed (like adding a network-server).
Multi-region support The new version will support multiple regions simultaneously, taking away the need to setup a ChirpStack Network Server for each region. This also paves the way for devices that can switch between regions in the future.
Many small improvements
New JavaScript engine (for the payload encoding / decoding)
Cleanup of the gRPC API (some inconsistencies removed)
Any many more…
Some of the changes that you are going to see as developer and integrator:
Single repo Where currently you have to make modifications to multiple repos to introduce a new feature, there will be one repo to deal with which should make it a lot easier to make modifications.
Test environment You will be able to use the provided Docker environment for both running the test as spinning up a ChirpStack environment for testing. Currently this is not possible as the Docker environments are only intended for running the tests and compiling the binaries.
gRPC only API Historically the REST API was added in order to support the web-interface. The REST API in the current version is a layer on top of the gRPC API that translates the REST API calls to gRPC calls internally. As the updated web-interface will be using gRPC-web, this REST API translation will be removed from ChirpStack itself. You can still add the REST to gRPC translation layer yourself as an external component.
I hope you are as excited as me about these changes!
Lots of good info here, thank you for sharing it. It sounds like an exciting update, gRPC-Web is very cool.
Regarding your last point, I think you may be underestimating how important the existing gRPC-Gateway is to many people/deployments. I only use gRPC directly for production implementations/integrations, but being able to hit the REST API with curl/Python or using the Swagger-UI directly is hugely beneficial for quick tasks, and I know a lot of people have built code on top of it. Yes, you could stand up a standalone gRPC-Gateway (or other proxy) container and point it at the right places, but then you’re back with another support mess on the forums.
None of this precludes the use of gRPC-Web, but I think you might want to consider including a gRPC-Gateway/Swagger-UI option, even if it’s off by default. Some food for thought.
I hope this fixes the merry-go-round trying to get application-server connected to Postgresql. I get continuous [level=fatal msg=“setup storage error: storage: migrate up error: Dirty database version 27. Fix and force version.”] errors everytime and everyway I install Chirpstack AS. Have not been able to find a fix that works. Several clean starts, always the same using the installation guide on Debian. Is there an error in that?
Debian Buster on RPi 3 B+
I gave up trying to get it to work on a Pi B+ since it doesn’t seem to run on arm6 at all. Ironically, the Chirpstack OS got everything running on the Pi B+ but nothing could complete a join. Too complicated for a normal human. I think you got that.
I did have TTN running until they changed to V3. No easy migration path.
Many thanks for your continuous support on this project.
I see the potential simplification benefits related to merging NS & AS (even if installation was already reather quite straightforward) but what about scalability ?
Having both services separated is helpful to optimise scaling and possibly load-balance the services (i.e.AS) independantly.
How do you see the scaling with this new architecture ?
Also, we have some scripts using the Rest API. I understabd they will need to be updated
Hi,
I like the idea of simplification as well. But I am also a bit afraid of the implications and considerations. Are you planing to create some kind of migration path from older versions? Will you drop support (improvements, fixes) for the old version and what is the time-frame here?
I am also a bit disappointed that you want to drop the REST API. We built many internal integrations on the endpoints and rely heavily on them. Please consider at least an alternative here.
Nevertheless I am still happy to hear this project will be supported by you this year as well.
I don’t think is good idea to remove rest api.
For external service I think the rest api you should maintain because Is very quick integrate and there are a lot of integrations based on this api.
I’m looking into facilitating this too, but I would like to keep this external. Currently I see two options (potentially both could be supported):
A gRPC Gateway bridge service which could run next to ChirpStack. Probably the only config would be the API endpoint of ChirpStack.
Provide a gRPC UI (GitHub - fullstorydev/grpcui: An interactive web UI for gRPC, along the lines of postman) for ChirpStack. The benefit of this would be that everything would still be pure gRPC. It could also help to implement gRPC In production as it provides a nice overview of the API options. In my understanding, this could be easier to use as it has a closer integration with the Protobuf / gRPC definitions.
The AS and NS configuration are pointing to the same database, in which case the migration history gets mixed up. This will be made easier as after the merge, there will be one database to manage.
The migration failed. Looking at migration #27, it creates trigram based indices. Did you execute the installation step create extension pg_trgm;?
Yes, the end-goal is to make the migration seamless. There will be a migration script to import data from v3 to v4. I need this too for the clients that I’m supporting commercially
I am also a bit disappointed that you want to drop the REST API.
Please note that the plan is to drop it from ChirpStack itself, not to not make it impossible to use a REST API. The current REST API is nothing more that gRPC Gateway on top of the gRPC API. This can still exist as a separate service in the next version if this is needed.
Could you define multi-tenancy? As it is already possible to create multiple organizations and assign users to an organization? As well it is already possible to make gateways private to a gateway.
One thing that I do want to change is to make the incrementing IDs UUIDs, such that in a multi-tenant environment you don’t expose the number of users, organizations and applications.
Also a good idea. And maybe consider adding the ability to use tags (adding support for tags for organizations and gateways, too) in the MQTT topics as well?
