Monitororing Gateways

Patrik · September 2, 2021, 9:11am

Hi!

Iam struggling a bit to understand howto monitor if a gateway is “alive” or “online”.

As i understand there are prometheus stats availible on lots of things but i cant seem to find if a specifik gateway is online or not? Or am i missing something here?

It would be nice if there was a intervall that chirpstack uses to se if a gateway has been seen och not and if that intervall is exceeded a mqtt message could be sent on a “status” topic and of course also be published on prometheus as this seems to be the choice of chirpstack.

I prefer zabbix for monitoring and could take the mqtt message and send it to zabbix via node-red but prometheus is not used in my enviroment so i would like it if mqtt could carry this information also since it already is sp extensively used in chirpstack.

If anyone has solved this in another way i would like to hear it.

chopmann · September 2, 2021, 10:43am

we have our own custom “prometheus-exporter” for that. I bet you could replicate the same functionality with Zabbix. The services polls the chirpstack-api triggered by a prometheus-scrape. Integrating it directly to chirpstack might be cumbersome, at least for the prometheus metrics, as in our case we have 3 Chirpstack-AS (NS) containers running, and each would generate in that case the same metrics.

You can also have multiple/different AS connected to the same NS, and each would have their “own” gateways. Even on a single AS, you have different orgs and visibility depending on that.

You could subscribe to the stats event of a gateway and depending on the interval configured, you would then have a “liveness” indicator outside and independent of chirpstack, in case chirpstack is unavailable you could still “see” if a gateway is sending something.

Patrik · September 2, 2021, 11:07am

I have looked at the stats event but it does not make sense as the gatewayid is encoded or garbled, at least not base64 encoded. that would be a way to get some “alive” from the gateways.

iam not so interrested in the prometheus metrics, much more if the gateways are alive or not since they are spread over a large region

chopmann · September 2, 2021, 11:10am

the message is protobuf (default settings) - if you look at broocar github profile you’ll find a repo with ready made libraries.

Patrik · September 2, 2021, 11:41am

No, i have changed from protobuf but the gatewayid is still not readable as you can se below…

{
  "gatewayID": "/MI9//4u19k=",
  "ip": "",
  "time": "2012-01-11T02:11:48Z",
  "location": null,
  "configVersion": "",
  "rxPacketsReceived": 0,
  "rxPacketsReceivedOK": 0,
  "txPacketsReceived": 0,
  "txPacketsEmitted": 0,
  "metaData": {},
  "statsID": "r8waETaaTY+gUn0ZMccLkA=="
}

chopmann · September 2, 2021, 11:44am

it’s base64 → fcc23dfffe2ed7d9

Patrik · September 2, 2021, 12:53pm

Hmmm… then iam doing something wrong…

echo ‘/MI9//4u19k=’ | base64 -d
▒▒=▒▒.▒▒

chopmann · September 2, 2021, 6:53pm

you are right, base64 would not eat it. Had to use python:

import binascii, base64

print(binascii.b2a_hex(base64.b64decode('/MI9//4u19k=')))

I dont know zabbix, but maybe you could run a python script which “does” what you want.

Patrik · September 2, 2021, 9:09pm

Am i wrong in thinking you are first decoding base64 and then doing a hex to ascii?

Iam not good on python, how would it be done in javascript? then i can use it in node-red… cant find a bin2ascii way…

fmgst · September 2, 2021, 9:26pm

You have a bunch of choices but if you plan to use this function in ChirpStack, this script will be run under otto JS VM which supports only ES5; any ES6 will cause script to fail.

With that in mind, see THIS.

chopmann · September 6, 2021, 9:35am

Node-Red runs server-side with nodejs (I tested this with node lts 14 from the REPL)

let buff = Buffer.from('/MI9//4u19k=')
console.log(buff.toString('hex'))

this should work.

Patrik · September 10, 2021, 12:52pm

No not in chirpstack, but node-red would be nice since i need to decode the gateway id and send it to zabbix.

Patrik · September 10, 2021, 12:55pm

That gives 1631278033997 in node-red but its not correct or done. Cant seem to get that to lead med anywhere…

fmgst · September 12, 2021, 12:57am

Try THIS.

Patrik · September 24, 2021, 2:07pm

Well, since i was unable to get this decodeing to work i resorted to the node-red chirpstack integration and created this flow that every five minutes checks all gateways for lastseen value and compares that to current time and sends a sms if its been down for more over 30 minutes.