Need help using ChirpStack V4 API

[Joshua_Smith]

Hi I currently have a sensor joined to my chirpstack 4 lorawan network however I’m not sure how I would go about retrieving the data that is sent by the sensor. Ultimately I want to retrieve the data that is sent it looks like this data:“sN8g6HkKaVoUAKE=” I want to be able to retrieve that value using an api call so I can take the location data that can be retrieve from that string and plot it on a map. The part I am stuck with is how to using chirpstack 4 retrieve that value for each sensor i have on network.

Any help would be appreciated as i’ve had a look online but there doesn’t seem to be a whole lot of code examples out there.

[Liam_Philipp]

You could use a deviceMetricsRequest via the grpc API to retrieve recent data. But as Chirpstack does not store device events for long, the recommended way to pull data out of Chirpstack is using an integration.

Also incase you don’t know, you can add a codec to the device profile of the device in order for Chirpstack to automatically decode the data when it gets it. So instead of data:“sN8g6HkKaVoUAKE=” Chirpstack would show “lat:30, long:60” and such. Most device manufacturers also provide codecs online.

[Joshua_Smith]

Hi Liam,

Thanks for your response I’ll look into doing it that way I have a codec currently but it doesn’t appear to be working I think the oyster 3 Codec online needs to be modified slightly to work with chirp stack I saw a post on here of someone who had it working but they didn’t share the code so I’ll need to figure out what they changed to get it to work.

[Liam_Philipp]

If you share your code someone here could fix it. Typically it’s just the function definition and return statement that need changing.

[Joshua_Smith]

Hi Liam,

The decoder is found here

https://www.oemserver.com/tools/Oyster3LoRaWAN/oyster3-lr-html-decoder.html

I’d need this work inside the Chirpstack Codec. I’d also then need to be able to extract that data using the API which I will move onto now thanks.

[IoTThinks]

Interesting.
The sensor vendor gives customers a quiz when giving a codec

You click View Source on that url to see the “hidden” codec.
It should work with ChirpStack v4.

      function Decoder(bytes, port) {
        try {
          return decodeUplink({
            fPort: port,
            bytes: bytes
          }).data;
        }
        catch {
          return null; 
        }
      }

      function MakeBitParser(bytes, offset, length) {
        return {
          bits: bytes.slice(offset, offset + length),
          offset: 0,
          bitLength: length * 8,
          U32LE: function U32LE(bits) {
            if (bits > 32)
              throw ("Invalid argument!");
            if (this.offset + bits > this.bitLength)
              throw ("Read past end of data!");
      
            var out = 0;
            var total = 0;
            
            while (bits > 0) {
              var byteNum = Math.floor(this.offset / 8);
              var discardLSbs = this.offset & 7;
              var avail = Math.min(8 - discardLSbs, bits);
              var extracted = (this.bits[byteNum] >>> discardLSbs);
              var masked = (extracted << (32 - avail)) >>> (32 - avail);
      
              out |= ((masked << total) >>> 0);
              total += avail;
              bits -= avail;
              this.offset += avail;
            }
      
            return out;
          },
          S32LE: function S32LE(bits) {
            return (this.U32LE(bits) << (32 - bits)) >> (32 - bits);
          }
        };
      }
      
      function ResolveTime(timestamp15, approxReceptionTime) {
        if (timestamp15 === 127)
          return null;
      
        var approxUnixTime = Math.round(approxReceptionTime.getTime() / 1000);
      
        // Device supplies: round(unix time / 15) modulo 127.
        // We're assuming that the uplink was sent some time BEFORE refTime,
        // and got delayed by network lag. We'll resolve the timestamp
        // in the window [approxReceptionTime - 21m, approxReceptionTime + 10m],
        // to allow for 10m of error in approxReceptionTime, and 10m of network lag.
        // So refTime = approxReceptionTime + 10m.
      
        var refTime = approxUnixTime + 600;
        timestamp = timestamp15 * 15;
      
        //                          refTime
        //                             v
        // [              |              |              |              ]
        //           ^              ^              ^              ^
        //       timestamp      timestamp      timestamp      timestamp
      
        //                             refTime
        //                                v
        // [              |              |              |              ]
        //           ^              ^              ^              ^
        //       timestamp      timestamp      timestamp      timestamp
      
        // We want the timestamp option immediately to the left of refTime.
        var refTimeMultiple = Math.floor(refTime / (127 * 15));
        var refTimeModulo = refTime % (127 * 15);
        var closestUnixTime = 0;
      
        if (refTimeModulo > timestamp)
          closestUnixTime = refTimeMultiple * (127 * 15) + timestamp;
        else
          closestUnixTime = (refTimeMultiple - 1) * (127 * 15) + timestamp;
      
        return new Date(closestUnixTime * 1000).toISOString();
      }
      
      function decodeUplink(input) {
        var p = input.fPort;
        var b = MakeBitParser(input.bytes, 0, input.bytes.length);
        var d = {};
        var w = [];
      
        if (p === 1) {
          d.type = "position";
          var l = {};
          l.latitudeDeg = Number((b.S32LE(32) / 1e7).toFixed(7)); // decimal scaling
          l.longitudeDeg = Number((b.S32LE(32) / 1e7).toFixed(7));
          d.inTrip = (b.U32LE(1) !== 0);
          d.fixFailed = (b.U32LE(1) !== 0);
          l.headingDeg = Number((b.U32LE(6) * 5.625).toFixed(2));
          l.speedKmph = b.U32LE(8);
          d.batV = Number((b.U32LE(8) * 0.025).toFixed(3));
          d.inactivityAlarm = null;
          d.batCritical = null;
      
          if (d.fixFailed) {
            d.cached = l;
            //w.push("fix failed");
          } else {
            d = Object.assign(d, l);
          }
        } else if (p === 2) {
          d.type = "downlink ack";
          d.sequence = b.U32LE(7);
          d.accepted = (b.U32LE(1) !== 0);
          d.fwMaj = b.U32LE(8);
          d.fwMin = b.U32LE(8);
          if (input.bytes.length < 6) {
            d.prodId = null;
            d.hwRev = null;
            d.port = null;
          } else {
            d.prodId = b.U32LE(8);
            d.hwRev = b.U32LE(8);
            d.port = b.U32LE(8);
          }
        } else if (p === 3) {
          d.type = "stats";
          d.initialBatV = Number((4.0 + 0.1 * b.U32LE(4)).toFixed(2));
          d.txCount = 32 * b.U32LE(11);
          d.tripCount = 32 * b.U32LE(13);
          d.gnssSuccesses = 32 * b.U32LE(10);
          d.gnssFails = 32 * b.U32LE(8);
          d.aveGnssFixS = b.U32LE(9);
          d.aveGnssFailS = b.U32LE(9);
          d.aveGnssFreshenS = b.U32LE(8);
          d.wakeupsPerTrip = b.U32LE(7);
          d.uptimeWeeks = b.U32LE(9);
        } else if (p === 4) {
          d.type = "position";
          var l = {};
          l.latitudeDeg = Number((256 * b.S32LE(24) / 1e7).toFixed(7)); // decimal scaling, truncated integer
          l.longitudeDeg = Number((256 * b.S32LE(24) / 1e7).toFixed(7));
          l.headingDeg = 45 * b.U32LE(3);
          l.speedKmph = 5 * b.U32LE(5);
          d.batV = b.U32LE(8);
          d.inTrip = (b.U32LE(1) !== 0);
          d.fixFailed = (b.U32LE(1) !== 0);
          d.inactivityAlarm = (b.U32LE(1) !== 0);
          if (b.U32LE(1) === 0)
            d.batV = Number((0.025 * d.batV).toFixed(3));
          else
            d.batV = Number((3.5 + 0.032 * d.batV).toFixed(3));
          crit = b.U32LE(2);
          if (crit === 0)
            d.batCritical = null;
          else if (crit === 1)
            d.batCritical = false;
          else
            d.batCritical = true;
      
          if (d.fixFailed) {
            d.cached = l;
            //w.push("fix failed");
          } else {
            d = Object.assign(d, l);
          }
        } else if (p === 30) {
          d.type = "hello";
          d.fwMaj = b.U32LE(8);
          d.fwMin = b.U32LE(8);
          d.prodId = b.U32LE(8);
          d.hwRev = b.U32LE(8);
          d.resetPowerOn = (b.U32LE(1) !== 0);
          d.resetWatchdog = (b.U32LE(1) !== 0);
          d.resetExternal = (b.U32LE(1) !== 0);
          d.resetSoftware = (b.U32LE(1) !== 0);
          b.U32LE(4);
          d.watchdogReason = b.U32LE(16);
          d.initialBatV = Number((3.5 + 0.032 * b.U32LE(8)).toFixed(2));
        } else if (p === 31) {
          d.type = "stats v3";
          d.ttff = b.U32LE(8);
          d.wakeupsPerTrip = b.U32LE(8);
          d.initialBatV = Number((3.5 + 0.032 * b.U32LE(8)).toFixed(3));
          d.currentBatV = Number((3.5 + 0.032 * b.U32LE(8)).toFixed(3));
          d.batCritical = (b.U32LE(1) !== 0);
          d.batLow = (b.U32LE(1) !== 0);
          d.tripCount = 32 * b.U32LE(14);
          d.uptimeWeeks = b.U32LE(10);
          d.mWhUsed = 10 * b.U32LE(10);
          d.percentLora = 100 / 32 * b.U32LE(5);
          d.percentGnssSucc = 100 / 32 * b.U32LE(5);
          d.percentGnssFail = 100 / 32 * b.U32LE(5);
          d.percentSleepDis = 100 / 32 * b.U32LE(5);
          d.percentOther = 100 - d.percentLora - d.percentGnssSucc - d.percentGnssFail - d.percentSleepDis;
        } else if (p === 33) {
          d.type = "position";
          var l = {};
          d.fixFailed = (b.U32LE(1) !== 0);
          l.latitudeDeg = Number((180 * b.S32LE(23) / (1 << 23)).toFixed(7)); // binary scaling
          l.longitudeDeg = Number((360 * b.S32LE(24) / (1 << 24)).toFixed(7));
          d.inTrip = (b.U32LE(1) !== 0);
          d.timestamp = b.U32LE(7);
          d.time = ResolveTime(d.timestamp, new Date(Date.parse(document.getElementById("rxtime").value.trim())));
          d.batCritical = (b.U32LE(1) !== 0);
          d.inactivityAlarm = (b.U32LE(1) !== 0);
          mins = 2 * b.U32LE(14); // lower bound
          d.inactiveDuration = Math.floor(mins / 1440) + 'd' + Math.floor((mins % 1440) / 60) + 'h' + (mins % 60) + 'm';
          d.batV = Number((3.5 + 0.032 * b.U32LE(8)).toFixed(3));
          l.headingDeg = 45 * b.U32LE(3);
          l.speedKmph = 5 * b.U32LE(5);
      
          if (d.fixFailed) {
            d.cached = l;
            //w.push("fix failed");
          } else {
            d = Object.assign(d, l);
          }
        } else {
          return {
            warnings: ['unknown FPort'],
          };
        }
      
        return {
          data: d,
          warnings: w,
        };
      }

[Joshua_Smith]

Wow that seems to be working thanks do you know if I can use the api to extract that location data now without having to decode the string again once the data is exported.

[datnus]

You need a decoder to decode the data.
Then you can extract the data via REST API or MQTT.

[Joshua_Smith]

@datnus

REST API or MQTT.

Do you know if there is any guides or anything out there for doing API data extraction I’ve done a bit of coding before but I’ve never had to extract data before.

[datnus]

You can read here to listen to MQTT uplink events.
https://www.chirpstack.io/docs/chirpstack/integrations/mqtt.html

After you have the correct codec, the decoded data will be in “object” after every uplink.

[Joshua_Smith]

Hi so I’ve downloaded MQTT explorer however I can’t seem to find where under any of these tabs the object data is kept.

[Liam_Philipp]

In default Chirpstack, messages to/from gateways are posted under your region-
prefix (au915_0 in this case) these are still encrypted and considered the backend. The decrypted messages you want are posted under “application/…”, these are messages Chirpstack posts back to the broker after it has decrypted the region-prefix events and are considered the integration events.

Seeing as you don’t have any application/ events in MQTT explorer, you should check your chirpstack.toml to ensure the MQTT integration is set up, but it should be by default. The section you are looking for is [integration.mqtt] and the “server=” line should be pointed to Chirpstack’s broker. If that is set up correctly and you still do not see “application/…” events there is likely some other issue.

As an aside, an easy way to view the broker rather than using MQTT explorer is to just run mosquitto_sub -t "#" -v on the Chirpstack host (if you are running docker you will have to enter you mosquitto container first, or specify port/host). With that you should be able to see all the events and data.

[Joshua_Smith]

Hi Liam,

I was able to get the application/ message to show in MQTT explorer by leaving it open for a while in the background. I am guessing it only updates when the sensors sends new data.

[Joshua_Smith]

Hi,

So I have setup pipe dream with HTTP integration and can see the post requests being sent to that however when I host a local server chirpstack doesn’t seem to be sending data to it however when I send the requests to the same url as configured in chirpstack via postman it can see the requests. I’m hosting the python script (the webserver) on the same machine as chirpstack is installed on. The only thing that I can maybe think of is that because my url is only http and not https it isn’t posting the data?