[release] ChirpStack Application Server v3.6

ChirpStack Application Server v3.6.0

Features

Protobuf based integrations

This feature adds a marshaler configuration option to the integration configuration. The default is backwards compatible with the current JSON format. New options are protobuf (Protobuf binary) and json (Protobuf based JSON).

RPM packaging

This is the first release providing .rpm packages for CentOS and RedHat. (#383

Improvements

gRPC / Protobuf cleanup

All definitions are now imported from github.com/brocaar/chirpstack-api/go. When using the gRPC API, you must update your imports.

Azure Service-Bus integration

Implement re-connect in case of an Azure Service-Bus error. This to work partly around azure-service-bus-go/issues/149.

Support multiple HTTP endpoints

Implement support for configuring multiple endpoints in the HTTP integration.

Internal improvements

Handlink of the received uplinks has been improved to reduce the number of required database transactions.

Bugfixes

  • Payload codec is hidden on application create (it was already on edit).
1 Like

Hello, Orne.
After upgrade to v3.6 “time” field is gone from rxInfo. on 3.5.1 it works fine. Tell me please is it a bug or I need to make some changes in app server configuration ?

I see no changes to time field here. Are you saying that if you downgrade to 3.5.1 the time field returns?

Do the gateways that are forwarding the traffic using GPS time or fake time?

Gateways using internal RTC clock time, which synced via ntp and using polypacket forwarder.
And I suggest, this is not a gateway side problem. We are using many kinds of gateways i.e Lorix, Tektelic with the same issue after upgrade. For today I can’t rollback the app server.

Let me check this for you.

1 Like

Sorry about that, I made a typo when refactoring the migrations to Protobuf structs. In the conversion to the “old” JSON format (which is currently the default), I was setting the timestamp on err != nil instead of err == nil :man_facepalming: This fixes the issue (including the test):

1 Like

Thanks a lot, Orne. Will be debian/ubuntu package update, or I need to build app-server from source for today?

This will be included in the next bugfix release which I have not yet created, so until then you need to build from source. Alternatively, you could change the marshaler to json (the default is json_v3). This gives you a slightly different structure, but has the advantage that you get the rxInfo and txInfo information as-is as forwarded by the ChirpStack Gateway Bridge.

You can use this Protobuf definition for the Protobuf based json or (binary) protobuf marshalers:

(Internally everything is now implemented using the Protobuf structs, the bug you found was in the Protobuf to “old” / “json_v3” conversion where I made a typo).

I always use marshaler=“json” in gateway-bridge. but this typo “bug” appears with it too.

This is a new feature of the ChirpStack Application Server, see the changelog:

This feature adds a marshaler configuration option to the integration configuration. The default is backwards compatible with the current JSON format. New options are protobuf (Protobuf binary) and json (Protobuf based JSON).

1 Like

ChirpStack Application Server v3.6.1

Bugfixes

  • Fix setting time in rxInfo for json_v3 (default) integration marshaler.
  • Fix integrations break when codec function returns a NaN value. (#390)
2 Likes

Hi , @brocaar , I cloned the chirpstack-api and made changes in the .proto files there and after compilation the changes reflected in the respective .pb.go file also. But how do I import those changes in chirpstack-application-server’s respective .go file? I keep trying to use make api command but it says :

 can't load package: package api/api.go: malformed module path "api/api.go": missing dot in first path element
>         Makefile:46: recipe for target 'api' failed
>         make: *** [api] Error 1

How do I integrate my additional rpc methods, defined in the .proto file, in my chirpstack-application-server’s .go file now?

You need to replace the github.com/brocaar/chirpstack-api/go Go module in the go.mod (chirpstack-application-server) with the path of your modified chirpstack-api/go. You can use the replace directive for this. For documentation see: https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive.

2 Likes

Hi, @brocaar, I did as you told me, here is my go.mod (chirpstack-application-server):

module GitHub - brocaar/chirpstack-application-server: ChirpStack Application Server is an open-source LoRaWAN application-server.

go 1.13

require (
Go Programming Language  |  Google Cloud v0.44.3
GitHub - Azure/azure-service-bus-go: Golang library for Azure Service Bus -- https://aka.ms/azsb v0.9.1
GitHub - NickBall/go-aes-key-wrap: Golang implementation of the AES Key Wrap algorithm as specified in RFC 3394 v0.0.0-20170929221519-1c3aa3e4dfc5
GitHub - aws/aws-sdk-go: AWS SDK for the Go programming language. v1.19.19
github.com/brocaar/chirpstack-api/go v0.0.0-20191119153543-35c59165bf94
GitHub - brocaar/lorawan: Package lorawan provides structures and tools to read and write LoraWAN messages from and to a slice of bytes. v0.0.0-20191115102621-6095d473cf60
GitHub - dgrijalva/jwt-go: ARCHIVE - Golang implementation of JSON Web Tokens (JWT). This project is now maintained at: v3.2.0+incompatible
GitHub - eclipse/paho.mqtt.golang v1.2.0
GitHub - elazarl/go-bindata-assetfs: Serves embedded files from `jteeuwen/go-bindata` with `net/http` v1.0.0
GitHub - gobuffalo/packr: The simple and easy way to embed static files into Go binaries. v1.22.0 // indirect
GitHub - gofrs/uuid: A UUID package originally forked from github.com/satori/go.uuid v3.2.0+incompatible
GitHub - gogo/protobuf: [Deprecated] Protocol Buffers for Go with Gadgets v1.2.1
GitHub - golang/protobuf: Go support for Google's protocol buffers v1.3.2
GitHub - gomodule/redigo: Go client for Redis v2.0.0+incompatible
GitHub - gopherjs/gopherjs: A compiler from Go to JavaScript for running Go code in a browser v0.0.0-20190430165422-3e4dfb77656c // indirect
GitHub - goreleaser/goreleaser: Deliver Go binaries as fast and easily as possible v0.106.0
GitHub - goreleaser/nfpm: nFPM is Not FPM - a simple deb, rpm, apk and arch linux packager written in Go v0.11.0
GitHub - gorilla/mux: Package gorilla/mux is a powerful HTTP router and URL matcher for building Go web servers with 🦍 v1.7.1
GitHub - grpc-ecosystem/go-grpc-middleware: Golang gRPC Middlewares: interceptor chaining, auth, logging, retries and more. v1.0.0
GitHub - grpc-ecosystem/go-grpc-prometheus: Prometheus monitoring for your gRPC Go servers. v1.2.0
GitHub - grpc-ecosystem/grpc-gateway: gRPC to JSON proxy generator following the gRPC HTTP spec v1.11.3
GitHub - jacobsa/crypto: Some Go cryptography routines that are not included in the Go standard library. v0.0.0-20190317225127-9f44e2d11115 // indirect
GitHub - jmoiron/sqlx: general purpose extensions to golang's database/sql v1.2.0
GitHub - jteeuwen/go-bindata: Hard fork of jteeuwen/go-bindata because it disappeared, Please refer to issues#5 for details. v3.0.8-0.20180305030458-6025e8de665b+incompatible
GitHub - konsorten/go-windows-terminal-sequences: Enable support for Windows Terminal Colors v1.0.2 // indirect
GitHub - lib/pq: Pure Go Postgres driver for database/sql v1.2.0
GitHub - mmcloughlin/geohash: Golang geohash library v0.9.0
GitHub - pkg/errors: Simple error handling primitives v0.8.1
GitHub - golang/protobuf: Go support for Google's protocol buffers v1.3.2
GitHub - gomodule/redigo: Go client for Redis v2.0.0+incompatible
GitHub - gopherjs/gopherjs: A compiler from Go to JavaScript for running Go code in a browser v0.0.0-20190430165422-3e4dfb77656c // indirect
GitHub - goreleaser/goreleaser: Deliver Go binaries as fast and easily as possible v0.106.0
GitHub - goreleaser/nfpm: nFPM is Not FPM - a simple deb, rpm, apk and arch linux packager written in Go v0.11.0
GitHub - gorilla/mux: Package gorilla/mux is a powerful HTTP router and URL matcher for building Go web servers with 🦍 v1.7.1
GitHub - grpc-ecosystem/go-grpc-middleware: Golang gRPC Middlewares: interceptor chaining, auth, logging, retries and more. v1.0.0
GitHub - grpc-ecosystem/go-grpc-prometheus: Prometheus monitoring for your gRPC Go servers. v1.2.0
GitHub - grpc-ecosystem/grpc-gateway: gRPC to JSON proxy generator following the gRPC HTTP spec v1.11.3
GitHub - jacobsa/crypto: Some Go cryptography routines that are not included in the Go standard library. v0.0.0-20190317225127-9f44e2d11115 // indirect
GitHub - jmoiron/sqlx: general purpose extensions to golang's database/sql v1.2.0
GitHub - jteeuwen/go-bindata: Hard fork of jteeuwen/go-bindata because it disappeared, Please refer to issues#5 for details. v3.0.8-0.20180305030458-6025e8de665b+incompatible
GitHub - konsorten/go-windows-terminal-sequences: Enable support for Windows Terminal Colors v1.0.2 // indirect
GitHub - lib/pq: Pure Go Postgres driver for database/sql v1.2.0
GitHub - mmcloughlin/geohash: Golang geohash library v0.9.0
GitHub - pkg/errors: Simple error handling primitives v0.8.1
GitHub - prometheus/client_golang: Prometheus instrumentation library for Go applications v1.1.0
GitHub - robertkrimen/otto: A JavaScript interpreter in Go (golang) v0.0.0-20180617131154-15f95af6e78d
GitHub - rubenv/sql-migrate: SQL schema migration tool for Go. v0.0.0-20181213081019-5a8808c14925
GitHub - sirupsen/logrus: Structured, pluggable logging for Go. v1.4.2
GitHub - smarty/assertions: Fluent assertion-style functions used by goconvey and gunit. Can also be used in any test or application. v1.0.0 // indirect
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a
github.com/spf13/cobra v0.0.5
github.com/spf13/viper v1.4.0
github.com/stretchr/testify v1.4.0
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5
github.com/ziutek/mymysql v1.5.4 // indirect
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
golang.org/x/net v0.0.0-20190628185345-da137c7871d7
golang.org/x/tools v0.0.0-20190708203411-c8855242db9c
google.golang.org/api v0.9.0
google.golang.org/grpc v1.24.0
gopkg.in/gorp.v1 v1.7.2 // indirect
gopkg.in/sourcemap.v1 v1.0.5 // indirect
pack.ag/amqp v0.12.1 // indirect
)

replace github.com/brocaar/chirpstack-api/go => /home/network-server/src/github.com/brocaar/chirpstack-api/go

After setting this up, I again implemented make api , and got the same result as follows:

root@localsvr:/home/network-server/src/github.com/brocaar/chirpstack-application-server# make api
Generating API code from .proto files
can’t load package: package api/api.go: malformed module path “api/api.go”: missing dot in first path element
Makefile:46: recipe for target ‘api’ failed
make: *** [api] Error 1

I am guessing, this error is returned because there is no api/ directory which contains any api.go file; unlike previously. As this folder has been moved from the source. So now, how do I generate my additional API endpoint?

You should generate the api definitions in the chirpstack-api repository, not chirpstack-application-server. Anyway, lets not mix the release announcement topic with support on using Go modules. If needed, please create a separate topic :slight_smile:

1 Like