RAW Frame logger - Python & MQTT

Applications
1

Hi just been getting to know CS v4 very impressed. I’ve used older versions on Gateways.
One thing I wanted was to be able to stash raw logs to a DB. I’m a scripter not a Developer so with the aid of Ghatgpt I got it to convert an example form Go to Python.

After many blind alleys I have a POC that might be of use to others. Brian @bconway I tried to take the lead from a post a while back and use MQTT but it looks like the protoBuf is different on inbound MQTT to that used for REDIS logging.

One thing for joins the “mType”: key doesn’t get set but I can see it’s join as the “devEui” is set.

2023-03-15 19:05:56 JoinRequest DevEUI: 24e124535b314729
{
  "phyPayload": "AAEAKgDAJOEkKUcxW1Mk4SSsqDqw9Ig=",
  "txInfo": {...},
  "rxInfo": [..],
  "devEui": "24e124535b314729",
  "time": "2023-03-15T19:05:56.367195Z"
}

2023-03-15 19:11:44 UnconfirmedDataUP DevAddr: e0043224

{ "phyPayload": "QCQyBOCAVRIFb92AeHShyiTB"
  "txInfo": {...},
  "rxInfo": [..],
  "mType": "UNCONFIRMED_DATA_UP",
  "devAddr": "e0043224",
  "time": "2023-03-15T19:06:44.080123Z"
}

solution

[REDIS]—[xRead up/down stream]—[Publish to MQTT]–[Node-RED]—[InfluxDB]

#########################################################################################
#  ChirpStack REDIS logging READ                                                        #
#  Decode - frame_log.prot UplinkFrameLog() & DownlinkFrameLog()                        #
#  Publish to MQTT eu868/gateway/GUI/logging_UP & logging_Down                          #
#  issue v X.0                                                                          #
#########################################################################################
#import json
import redis
import paho.mqtt.client as mqtt #import the client1
import google.protobuf.json_format as json_format
import google.protobuf.message as message
from   google.protobuf.json_format import MessageToJson
from   chirpstack_api import api

# REDIS
server = "localhost"
port   = 6379
pool   = redis.ConnectionPool(host='localhost', port=6379, db=0)
rdb    = redis.Redis(connection_pool=pool)
key    = "gw:stream:frame"
lid    = "0"
# MQTT
broker_address = "localhost"
client = mqtt.Client("protoBuf")  # create new instance
print("connecting to MQTT")
client.connect(broker_address)    # connect to broker

while True:
    resp = rdb.xread({key: lid}, count=1, block=0)

    if len(resp) != 1:
        raise ValueError("Exactly one stream response is expected")

    for msg in resp[0][1]:
        lid = msg[0].decode()

        if b"up" in msg[1]:
            #print('msg[1][b"up"]',msg[1][b"up"])
            b  = msg[1][b"up"]
            pl = api.frame_log_pb2.UplinkFrameLog()
            pl.ParseFromString(b)
            #print("=== UP ===")
            print(MessageToJson(pl))
            client.publish("eu868/gateways/logging_UP",MessageToJson(pl))

        if b"down" in msg[1]:
            b  = msg[1][b"down"]
            pl = api.frame_log_pb2.DownlinkFrameLog()
            pl.ParseFromString(b)
            #print("=== DOWN ===")
            print(MessageToJson(pl))
            client.publish("eu868/gateways/logging_Down",MessageToJson(pl))
1 Like
2

Not sure I understand why you’re reading out of Redis. There are already a bunch of good integrations that can forward data from ChirpStack.

3

Hi Brian I took my approach from reading the Docs. There is no “Gateway Logging Service” in API the documentation. And [Features/Logging] “Frame logging: ChirpStack provides an option to log all uplink and downlink frames to a Redis stream, which can be consumed by external application(s) for monitoring or logging purposes.”

And the point I started from was: https://github.com/chirpstack/chirpstack/tree/master/examples/frame_log/go/main.go which is connecting to REDIS.

So I’m confused what is the correct way to get access to raw gateway logs?
Thanks, Lawrence

4

For that use case, I pull from the MQTT gateway topic, in the same way that ChirpStack does.

1 Like
5

I tried that as that’s what you recommended in a post on the Forum. But get this error google.protobuf.message.DecodeError: Error parsing message. The protoBuf length is quite different between MQTT payload and REDIS read. Any thoughts?

File "/home/chirplns/.local/mqtt_gw_frame_logger.py", line 25, in on_message
pl.ParseFromString(b) google.protobuf.message.DecodeError: Error parsing message

b'\n$@W}\x0b&\x80\x86\x9c\x05\x89\x98\xde8\x18Wm\xd0\xca\r\xfb\xb6\xa7\xd5\xbc\xc1U\xe7\xd1$\x03\x9f\x11\x8c\xdb\xebB"\x12\x08\xa0\x84\x91\x9e\x03\x12\n\x1a\x08\x08\xc8\xd0\x07\x10\x07(\x01*M\n\x1024e124fffef32f55\x10\xe6\xe4\x01\x1a\x0b\x08\x81\xc6\xcc\xa0\x06\x10\x88\xa8\xe8v"\x0b\x08\x93\xcb\xf7\x89\x05\x10\xc0\xe0\xddv0\xb6\xff\xff\xff\xff\xff\xff\xff\xff\x01=33\x13A@\x02H\x01j\x04\xde\xeb\xc8D\x80\x01\x02'
msgLng: 137

b'\n$@W}\x0b&\x80\x86\x9c\x05\x89\x98\xde8\x18Wm\xd0\xca\r\xfb\xb6\xa7\xd5\xbc\xc1U\xe7\xd1$\x03\x9f\x11\x8c\xdb\xebB\x12\x12\x08\xa0\x84\x91\x9e\x03\x12\n\x1a\x08\x08\xc8\xd0\x07\x10\x07(\x01\x1a\x99\x01\n\x1024e124fffef32f55\x10\xe6\xe4\x01\x1a\x0b\x08\x81\xc6\xcc\xa0\x06\x10\x88\xa8\xe8v"\x0b\x08\x93\xcb\xf7\x89\x05\x10\xc0\xe0\xddv0\xb6\xff\xff\xff\xff\xff\xff\xff\xff\x01=33\x13A@\x02H\x01b\x12\t\xcb\xe9j\xc30\xe1I@\x11\x00\x00\x00\x80\x9c(\xd5\xbfj\x04\xde\xeb\xc8Dz\x19\n\x10region_config_id\x12\x05eu868z\x1b\n\x12region_common_name\x12\x05EU868\x80\x01\x02 \x02*\x08260b7d57:\x0b\x08\x81\xc6\xcc\xa0\x06\x10\x88\xa8\xe8v'
msgLng: 239
6

I can’t speak to the Python code, but make sure you’re using the correct v4 API library. I’ve had no issues decoding gateway messages in Go.

7

Thanks for you time looks like a Python issue.

8

great work, been meaning to try and get the frames for greater metrics for logging and testing purposes for a while.

did you manage to decode the mqtt gateway topic? Not being familiar with grpc/protobuf and lacking understanding how to use it i usually use the http api integration - but i cannot get the gateway frames this way. there are basically no examples anywhere either :laughing:

sorry to tag you @bconway but is this what you do? If so are you able to elaborate on how i would decode the mqtt gateway protobuf. unfortunately i usually use python as well for almost all my scripting.

9

Sure. Assuming you’re looking for tips on how to decode the protobuf and not an approach to pulling from MQTT, here’s an example of decoding gateway Stats. Note that it’s all in Go, I don’t do much Python these days:

That directory has decoding of other gateway payloads (Uplink, etc). They use an intermediary Point KV type that is used by the larger platform.

10

hey mate,

thanks for replying - i will try decode the mqtt messages with that protobuf you linked (not that i’m 100% sure on exactly what I’m doing with grpc/protobuf as I’ve never used it before).

the difference I found so far was the redis stream and code iiLaw posted provides the gateway frames which is what i was interested in so i can log the other traffic around me and what type of msg they are. I currently do this with a python script by reading the docker logs and picking up the unknown devices.

i did run a modified version of his script which worked fine for a few hours but it seems to of killed the redis db and dropped all the assigned activated device addresses although that could of been something else stupid unrelated i did lol.

i think the reason the other code doesn’t work is because when i changed it to json it is not the same payload - it looks more aligned with the gist you sent (ill check this tomorrow).

ps.
this seems to be the address to console that we’re needing to decode

https://example.com/api.InternalService/StreamGatewayFrames
11

Yeah, the internal service endpoints are there to support the web UI and are subject to sudden changes. I already mentioned this above, but I would not advise tapping the Redis streams for pulling data for the same reason.