New chirpstack FUOTA Server

riuss · January 17, 2021, 8:41am

Hi,

I’m trying to use the new implementation of the fuota server but I don’t realy understand how to set it up.
I followed the explanation on the git and then started the fuota-server :

root@friloranet01:~/chirpstack-fuota-server# dist/linux_386/chirpstack-fuota-ser                   ver --config packaging/files/chirpstack-fuota-server.toml
INFO[0000] starting ChirpStack FUOTA Server              version=SNAPSHOT-fa9c0b                   9
INFO[0000] storage: connecting to PostgreSQL database
INFO[0000] storage: applying PostgreSQL schema migrations
INFO[0000] client/as: setup application-server client
INFO[0000] eventhandler: setup application-server event-handler
INFO[0000] api: starting fuota-server api server         bind="0.0.0.0:8070" ca_                   cert= tls_cert= tls_key=
INFO[0000] integration/eventhandler: starting event-handler server  bind="0.0.0.                   0:8090" marshaler=protobuf

It seems to run but I don’t understand how to access it. Is there a UI? I tried to acces the localhost:8070 and 8090 but nothing happend

Thank’s in advance for your help

Curt_Black · January 19, 2021, 7:11pm

You can find a go script in the examples/ folder of the fuota-server, which connects to that default 8070 port and creates a fuota job. The data previously found in the application server fuota deployment screen should look very familiar to the members of the fuota.Deployment object in that example script. To answer more directly, its not a UI, its a grpc endpoint. API: https://github.com/brocaar/chirpstack-api/blob/master/protobuf/fuota/fuota.proto

riuss · February 10, 2021, 11:53am

Hi,

Thank you for you reply. I am now able to start the fuota server and I successfully tested the Clock synchronization.
Now I’m trying to understand how to do a firmware update. I not familiar with Go and gRPC and it’s still difficult for me to understand how it works.
If I want to use the API with python, can I use the same functions as in the Go example and simply adapt the code like in the following example Python example?
When I look at the Go example , there is parameters about the multicast session and fragmentation session but I can’t find any information about the firmware binary I want to send over this session. What should I do to chose this file?

Thank’s in advance for your reply

Curt_Black · March 23, 2021, 7:20pm

Did you get this sorted? Sorry I can’t be much help with porting to python and using grpc, but I suspect there are some examples our there and using the downlink example you posted is a good start.

The data for your firmware binary is sent as the “Payload” member of the Deployment method. I basically did this:

payload, err := ioutil.ReadFile("deltaFw-test.bin")
if err != nil {
    panic(err)
}

Then, in the deployment object:

Payload:                    payload,

instead of the make([]byte, 100)

don’t forget “io/ioutil” in the import section

Shabaz_Shaik · June 1, 2021, 4:51am

Hi @riuss,
I am also trying to establish fuota. Till here I was also able to configure. But in the case of configuring HTTP integration, I am stuck.

What should I give at the Endpoints field. In place of IP address field, I am giving the docker container IP of the FUOTA-server. Is this the right way to do it?

How did you configure it?

riuss · June 1, 2021, 5:38am

Hi Shabaz

Since I can’t check my configuration anymore as I don’t work anymore for this project, it’s a little bit difficult to answer. What i remember is that you have to configure this file : chirpstack-fuota-server/chirpstack-fuota-server.toml at master · brocaar/chirpstack-fuota-server · GitHub to with your application server ip and token (to communicate from FUOTA Server to application server)

From the application server to the FUOTA Server it’s correct as you did, but I’m not sure about the 8070/up. I think it should also correspond to the chirpstack-fuota-server.toml definition

Sorry that I cannot help more. It would be great to have a bit more documentation for this feature, it’s not so easy to understand…

Curt_Black · June 1, 2021, 9:30am

The fuota-server is a standalone service now, as you know. The config file for said service has 2 endpoints:

The listener for uplink packets. You MUST add the integration, as you posted in your picture, so that uplink packets are passed to the fuota-server (if fuota-server is running on the same server as the app server and you didn’t change any settings then “http://localhost:8090” is what you would type in that box you posted - “Protocol Buffers” is correct for the first option). Again, security beware this is “localhost”, all security settings within the config files
The fuota endpoint for creating fota jobs. See the readme @ GitHub - brocaar/chirpstack-fuota-server: FUOTA server which can be used together with ChirpStack Application Server.. If you want to create a fota job, it is done through this grpc endpoint. An example Go script is in the repo

Shabaz_Shaik · June 1, 2021, 10:37am

Hi,
I am new to this kind of work so bear with me if I ask any silly questions.
I am having my FUOTA server in docker and the application server is in the host machine. So I will use docker container IP.
<docker_ipaddress:8090>/up is this one enough? As I have seen few other events also like ack, txack and error.

The fuota-server is a standalone service now, as you know. The config file for said service has 2 endpoints:

And out of two endpoints what is the second endpoint?
Thank you

Curt_Black · June 1, 2021, 5:23pm

The /up from that integration url can be removed, it’s not expecting a url path. The up/txack are mqtt topics that dont really apply to what we are doing here
The 2nd endpoint is a grpc listener, by default that listens on port 8070 (see config file to change that). This is how you create/deploy a fuota job. See the example Go script in the repo

Shabaz_Shaik · June 2, 2021, 5:36am

Now whenever I run the main.go I am getting these logs in futoa :

INFO[9432] storage: deployment created                   id=30fdd33a-6160-4a71-9750-60a675bfc064
INFO[9432] storage: deployment device created            deployment_id=30fdd33a-6160-4a71-9750-60a675bfc064 dev_eui=7703486190e47fde
DEBU[9432] fuota: stepCreateMulticastGroup funtion called  deployment_id=30fdd33a-6160-4a71-9750-60a675bfc064
DEBU[9432] [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing" 
INFO[9432] finished client unary call                    error="rpc error: code = NotFound desc = object does not exist" grpc.code=NotFound grpc.method=Get grpc.service=api.ApplicationService grpc.time_ms=3.117 span.kind=client system=grpc
ERRO[9432] api: fuota deployment error                   deployment_id=30fdd33a-6160-4a71-9750-60a675bfc064 error="get application error: rpc error: code = NotFound desc = object does not exist"
DEBU[9443] eventhandler: event received from application-server  data=9c dev_eui=7703486190e47fde event=up f_cnt=235 f_port=1

and main .go is getting terminated with this log:

bash-5.1# go run examples/class_c/main.go 
deployment created: 30fdd33a-6160-4a71-9750-60a675bfc064
bash-5.1#

The node is not getting any ifnormation regarding multicast.

Curt_Black · June 2, 2021, 2:36pm

Looks like your script is working – “deployment created” is the expected success. At this point, your service, and the endpoints appear to be working ok.

Further debug you’ll need to infer from the debug messages. I see “rpc error: code = NotFound desc = object does not exist” - maybe the DevEUI of the device does not exist in the application server? Not sure which object its referring to, but I see that when my DevEUI is wrong and does not exist in my app server

Shabaz_Shaik · June 3, 2021, 10:31am

Hi,
the device with devEUI is registered in the application server. This I am confirming as we are getting regular uplink packets from the device.

In few places, I want to confirm few things regarding the IP addresses.

In main.go example
conn, err := grpc.Dial("localhost:8070", dialOpts...)
As I am running this in the docker container I am giving the IP-address of docker in the place of localhost.
Is it ok? If not which Ip we should provide here?
In config file

  // API integration settings.
  [application_server.api]
  // ChirpStack Application Server API server endpoint.
  server="10.60.69.55:8080"

Here I have specified my application server IP.
I think this is also fine?

In your case how these configurations and Ip-address were?
How and where were your chirpastack network and application and FUOTA servers(means everything docker or not)?

Curt_Black · June 3, 2021, 1:55pm

This is correct. The fact that your fuota server logs show “deployment created” means this setting is correct
From what you say, it seems this setting is also correct. What about the api token? You created an api token in the app server then set that token in the fuota server config file?

I have not tried in docker, so both my fuota server and app server are localhost. However, our devops has deployed this in production using docker, with a similar setup.

Can you retrieve the application server logs when this occurs?

Shabaz_Shaik · June 3, 2021, 4:54pm

I have created the api key using the web ui apikeys

What ever the long key generated while creating this API key, the same key I copied in the API-token field of the config file.

The log on the chirpstack application server is:
Jun 03 22:18:24 HYSSHAIK chirpstack-application-server[28254]: time="2021-06-03T22:18:24+05:30" level=info msg="finished unary call with code NotFound" ctx_id=46ebe6d6-0427-4336-b61c-f00e0067c61e error="rpc error: code = NotFound desc = object does not exist" grpc.code=NotFound grpc.method=Get grpc.service=api.ApplicationService grpc.start_time="2021-06-03T22:18:24+05:30" grpc.time_ms=2.278 peer.address="172.18.0.3:32884" span.kind=server system=grpc

Shabaz_Shaik · June 4, 2021, 6:37am

Hi,
I got to know the problem for above error.
In main.go(example) I overlooked the application-ID. I have only one application. After changing it I am getting JWT related error.

Jun 04 11:33:52 HYSSHAIK chirpstack-application-server[25149]: time="2021-06-04T11:33:52+05:30" level=info msg="finished unary call with code Unauthenticated" ctx_id=47e4f577-cbee-4e31-b3e6-b1b82fd18efb error="rpc error: code = Unauthenticated desc = authentication failed: jwt parse error: signature is invalid" grpc.code=Unauthenticated grpc.method=Get grpc.service=api.ApplicationService grpc.start_time="2021-06-04T11:33:52+05:30" grpc.time_ms=19.728 peer.address="172.18.0.3:41348" span.kind=server system=grpc
Jun 04 11:33:52 HYSSHAIK chirpstack-application-server[25149]: time="2021-06-04T11:33:52+05:30" level=info msg="finished unary call with code Unauthenticated" ctx_id=47e4f577-cbee-4e31-b3e6-b1b82fd18efb error="rpc error: code = Unauthenticated desc = authentication failed: jwt parse error: signature is invalid" grpc.code=Unauthenticated grpc.method=Get grpc.service=api.ApplicationService grpc.start_time="2021-06-04T11:33:52+05:30" grpc.time_ms=19.728 peer.address="172.18.0.3:41348" span.kind=server system=grpc

Curt_Black · June 4, 2021, 7:12am

Also I have no idea what you are doing with MULTIPLE app-id/servers. That is for you to debug

Shabaz_Shaik · June 7, 2021, 3:57am

Hi @Curt_Black,
Thanks for your response and patience. I am able to figure out the problem. Now we are able to start the multicast group setup, class change request etc.

INFO[0006] fuota: add devices to multicast-group         deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0006] fuota: add device to multicast-group          deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde multicast_group_id=3e7b68c0-a6f2-400c-9b2a-3043dc7df6ce
INFO[0006] finished client unary call                    grpc.code=OK grpc.method=AddDevice grpc.service=api.MulticastGroupService grpc.time_ms=107.856 span.kind=client system=grpc
INFO[0006] fuota: starting multicast-setup for devices   deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0006] fuota: initiate multicast-setup for device    attempt=1 deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0007] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.DeviceQueueService grpc.time_ms=41.629 span.kind=client system=grpc
INFO[0007] storage: deployment log created               command=McGroupSetupReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
DEBU[0041] eventhandler: event received from application-server  data=73 dev_eui=7703486190e47fde event=up f_cnt=74 f_port=1
DEBU[0041] fuota: ignoring uplink event                  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde f_port=1
DEBU[0043] eventhandler: event received from application-server  data=0200 dev_eui=7703486190e47fde event=up f_cnt=75 f_port=200
INFO[0043] fuota: multicast-setup command received       cid=McGroupSetupReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0043] fuota: McGroupSetupAns received               deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde id_error=false mc_group_id=0
INFO[0043] storage: deployment log created               command=McGroupSetupAns deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0043] storage: deployment device updated            deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0043] fuota: multicast-setup completed successful for all devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0043] storage: deployment updated                   id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0043] fuota: starting fragmentation-session setup for devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0043] fuota: initiate fragmentation-session setup for device  attempt=1 deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0043] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.DeviceQueueService grpc.time_ms=24.182 span.kind=client system=grpc
INFO[0043] storage: deployment log created               command=FragSessionSetupReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
DEBU[0083] eventhandler: event received from application-server  data=7a dev_eui=7703486190e47fde event=up f_cnt=76 f_port=1
DEBU[0083] fuota: ignoring uplink event                  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde f_port=1
DEBU[0085] eventhandler: event received from application-server  data=0202 dev_eui=7703486190e47fde event=up f_cnt=77 f_port=201
INFO[0085] fuota: fragmentation-session setup command received  cid=FragSessionSetupReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0085] fuota: FragSessionSetupAns received           deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde encoding_unsupported=false frag_index=0 frag_session_index_not_supported=false not_enough_memory=true wrong_descriptor=false
INFO[0085] storage: deployment log created               command=FragSessionSetupAns deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
WARN[0103] fuota: fragmentation-session setup reached max. number of attempts, some devices did not complete  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0103] storage: deployment updated                   id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0103] fuota: starting multicast class-c session setup for devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0103] fuota: initiate multicast class-c session setup for device  attempt=1 deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0103] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.DeviceQueueService grpc.time_ms=24.256 span.kind=client system=grpc
INFO[0103] storage: deployment log created               command=McClassCSessionReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
DEBU[0126] eventhandler: event received from application-server  data=6d dev_eui=7703486190e47fde event=up f_cnt=78 f_port=1
DEBU[0126] fuota: ignoring uplink event                  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde f_port=1
DEBU[0127] eventhandler: event received from application-server  data=0400240000 dev_eui=7703486190e47fde event=up f_cnt=79 f_port=200
INFO[0127] fuota: multicast-setup command received       cid=McClassCSessionReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0127] fuota: McClassCSessionAns received            deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde dr_error=false freq_error=false mc_group_id=0 mc_group_undefined=false
INFO[0127] storage: deployment log created               command=McClassCSessionAns deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0127] storage: deployment device updated            deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0127] fuota: multicast class-c session setup completed successful for all devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0127] storage: deployment updated                   id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0127] fuota: starting multicast enqueue             deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0127] fuota: waiting with enqueue until multicast-session starts  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 sleep_time=35.625505588s
INFO[0163] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.MulticastGroupService grpc.time_ms=53.159 span.kind=client system=grpc
INFO[0163] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.MulticastGroupService grpc.time_ms=24.648 span.kind=client system=grpc
INFO[0163] storage: deployment updated                   id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0163] fuota: starting fragmentation-session status request for devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0163] fuota: request fragmentation-session status for device  attempt=1 deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0163] finished client unary call                    grpc.code=OK grpc.method=Enqueue grpc.service=api.DeviceQueueService grpc.time_ms=24.179 span.kind=client system=grpc
INFO[0163] storage: deployment log created               command=FragSessionStatusReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0163] fuota: waiting for multicast-session to end for devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 sleep_time=1m3.879604512s
DEBU[0252] eventhandler: event received from application-server  data=3c dev_eui=7703486190e47fde event=up f_cnt=80 f_port=1
DEBU[0252] fuota: ignoring uplink event                  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde f_port=1
DEBU[0255] eventhandler: event received from application-server  data=0100000000 dev_eui=7703486190e47fde event=up f_cnt=81 f_port=201
INFO[0255] fuota: fragmentation-session setup command received  cid=FragSessionStatusReq deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0255] fuota: FragSessionStatusAns received          deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde frag_index=0 missing_frag=0 nb_frag_received=0 not_enough_matrix_memory=false
INFO[0255] storage: deployment log created               command=FragSessionStatusAns deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0255] storage: deployment device updated            deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 dev_eui=7703486190e47fde
INFO[0255] fuota: fragmentation-session status request completed successful for all devices  deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0255] storage: deployment updated                   id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0255] fuota: deleting multicast-group               deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7
INFO[0255] finished client unary call                    grpc.code=OK grpc.method=Delete grpc.service=api.MulticastGroupService grpc.time_ms=24.502 span.kind=client system=grpc
INFO[0255] fuota: multicast-group deleted                deployment_id=efa37de1-7ea6-42cc-84e2-82167a82c6b7 multicast_group_id=3e7b68c0-a6f2-400c-9b2a-3043dc7df6ce

Now we are trying to send the firmware file.
The reasons for earlier problems were:
I don’t know both the
a. API_TOKEN in the fuota config file
b. JWT_TOKEN in the application server are the same as their names are different.
So, I was changing the key in only one file which leads to an authentication issue.

Thanks
Shabaz

Curt_Black · June 7, 2021, 5:58am

Good to hear, yeah the fuota to app-server seemed like the broken link

Let me know if you have more issues, I am currently going through lots of testing as well right now so might be able to help

Dragan · June 14, 2021, 8:42am

Can anyone help me configure fuota-server. I’m trying to send a command to multiple devices.
Then I get this:

Where I went wrong? The command did not reach the device.

Thank’s in advance for your help.

Dragan · June 14, 2021, 8:49am

I created deployment as in GO example: