Problem decoding up messages in EM300DI

Devices
1

I started this topic thinking that it would be a problem with the Milesight manufacturer’s decode, but apparently they confirm that this problem can come from Chirpstack. Let’s see what you think.

HI, we are integrating this EM300-DI device into Chirpstack 4.8.1, after loading the decode that you have published.

I have a configuration where the totalizer of a water meter is sent every minute.

And I get these error messages randomly due to decoding failure.
image

Here is an image of a UP that cannot decode
image

And here is an image of a UP that decodes correctly
image

EM-300-DI.log.json

I am attaching a more complete log where you can see UP messages correctly decoded and others that are not.

What could be happening?

thanks for your help

2

Not able to see the image.

If the decoder sometimes works and sometimes doesnot work, the codec does not handle the upload payload well.

Sometimes the payload is shorter or lomger.
And sometimes need to ignore some payload.

3

Sorry, I’ll give another example again, let’s see if you can see the images now. According to Milesight support, it seems that it is a Chirpstack problem.

image
image1012×175 7.38 KB

This would be the one that returns the error
data:“AXVkA2cYAQRoZQXhCgAKAAA0skY=”

image

And this one that sends the next minute would be correct
data:“AXVkA2cZAQRoZAXhCgAKAAA0skY=”
image

4

There are minor differences between two payloads.
For temperature and humidity.

image
image977×167 12.8 KB

Save this as test.html.
And both payloads are ok.

<script>
/**
 * Payload Decoder
 *
 * Copyright 2024 Milesight IoT
 *
 * @product EM300-DI
 */
// Chirpstack v4
function decodeUplink(input) {
    var decoded = milesightDeviceDecode(input.bytes);
    return { data: decoded };
}

// Chirpstack v3
function Decode(fPort, bytes) {
    return milesightDeviceDecode(bytes);
}

// The Things Network
function Decoder(bytes, port) {
    return milesightDeviceDecode(bytes);
}

function milesightDeviceDecode(bytes) {
    var decoded = {};

    for (var i = 0; i < bytes.length; ) {
        var channel_id = bytes[i++];
        var channel_type = bytes[i++];

        // BATTERY
        if (channel_id === 0x01 && channel_type === 0x75) {
            decoded.battery = bytes[i];
            i += 1;
        }
        // TEMPERATURE
        else if (channel_id === 0x03 && channel_type === 0x67) {
            // ℃
            decoded.temperature = readInt16LE(bytes.slice(i, i + 2)) / 10;
            i += 2;

            // ℉
            // decoded.temperature = readInt16LE(bytes.slice(i, i + 2)) / 10 * 1.8 + 32;
            // i +=2;
        }
        // HUMIDITY
        else if (channel_id === 0x04 && channel_type === 0x68) {
            decoded.humidity = bytes[i] / 2;
            i += 1;
        }
        // GPIO
        else if (channel_id === 0x05 && channel_type === 0x00) {
            decoded.gpio = readGPIOStatus(bytes[i]);
            i += 1;
        }
        // PULSE COUNTER
        else if (channel_id === 0x05 && channel_type === 0xc8) {
            decoded.pulse = readUInt32LE(bytes.slice(i, i + 4));
            i += 4;
        }
        // PULSE COUNTER (v1.3+)
        else if (channel_id === 0x05 && channel_type === 0xe1) {
            decoded.water_conv = readUInt16LE(bytes.slice(i, i + 2)) / 10;
            decoded.pulse_conv = readUInt16LE(bytes.slice(i + 2, i + 4)) / 10;
            decoded.water = readFloatLE(bytes.slice(i + 4, i + 8));
            i += 8;
        }
        // GPIO ALARM
        else if (channel_id === 0x85 && channel_type === 0x00) {
            decoded.gpio = readGPIOStatus(bytes[i]);
            decoded.gpio_alarm = readGPIOAlarm(bytes[i + 1]);
            i += 2;
        }
        // WATER ALARM
        else if (channel_id === 0x85 && channel_type === 0xe1) {
            decoded.water_conv = readUInt16LE(bytes.slice(i, i + 2)) / 10;
            decoded.pulse_conv = readUInt16LE(bytes.slice(i + 2, i + 4)) / 10;
            decoded.water = readFloatLE(bytes.slice(i + 4, i + 8));
            decoded.water_alarm = readWaterAlarm(bytes[i + 8]);
            i += 9;
        }
        // HISTORICAL DATA
        else if (channel_id === 0x20 && channel_type === 0xce) {
            // maybe not historical raw data
            if (bytes.slice(i).length < 12) break;

            var point = {};
            point.timestamp = readUInt32LE(bytes.slice(i, i + 4));
            point.temperature = readInt16LE(bytes.slice(i + 4, i + 6)) / 10;
            point.humidity = bytes[i + 6] / 2;
            var mode = bytes[i + 7];
            if (mode === 1) {
                point.gpio_type = "gpio";
                point.gpio = bytes[i + 8];
            } else if (mode === 2) {
                point.gpio_type = "pulse";
                point.pulse = readUInt32LE(bytes.slice(i + 9, i + 13));
            }
            decoded.history = decoded.history || [];
            decoded.history.push(point);
            i += 13;
        }
        // HISTORICAL DATA (v2)
        else if (channel_id === 0x21 && channel_type === 0xce) {
            var point = {};
            point.timestamp = readUInt32LE(bytes.slice(i, i + 4));
            point.temperature = readInt16LE(bytes.slice(i + 4, i + 6)) / 10;
            point.humidity = bytes[i + 6] / 2;
            point.alarm = readAlarm(bytes[i + 7]);
            var mode = bytes[i + 8];
            if (mode === 1) {
                point.gpio_type = "gpio";
                point.gpio = readGPIOStatus(bytes[i + 9]);
            } else if (mode === 2) {
                point.gpio_type = "pulse";
                point.water_conv = readUInt16LE(bytes.slice(i + 10, i + 12)) / 10;
                point.pulse_conv = readUInt16LE(bytes.slice(i + 12, i + 14)) / 10;
                point.water = readFloatLE(bytes.slice(i + 14, i + 18));
            }

            decoded.history = decoded.history || [];
            decoded.history.push(point);
            i += 18;
        } else {
            break;
        }
    }

    return decoded;
}

/* ******************************************
 * bytes to number
 ********************************************/
function readUInt16LE(bytes) {
    var value = (bytes[1] << 8) + bytes[0];
    return value & 0xffff;
}

function readInt16LE(bytes) {
    var ref = readUInt16LE(bytes);
    return ref > 0x7fff ? ref - 0x10000 : ref;
}

function readUInt32LE(bytes) {
    var value = (bytes[3] << 24) + (bytes[2] << 16) + (bytes[1] << 8) + bytes[0];
    return (value & 0xffffffff) >>> 0;
}

function readFloatLE(bytes) {
    // JavaScript bitwise operators yield a 32 bits integer, not a float.
    // Assume LSB (least significant byte first).
    var bits = (bytes[3] << 24) | (bytes[2] << 16) | (bytes[1] << 8) | bytes[0];
    var sign = bits >>> 31 === 0 ? 1.0 : -1.0;
    var e = (bits >>> 23) & 0xff;
    var m = e === 0 ? (bits & 0x7fffff) << 1 : (bits & 0x7fffff) | 0x800000;
    var f = sign * m * Math.pow(2, e - 150);

    var v = Number(f.toFixed(2));
    return v;
}

function readGPIOStatus(bytes) {
    // 0: low, 1: high
    switch (bytes) {
        case 0:
            return "low";
        case 1:
            return "high";
        default:
            return "unknown";
    }
}

function readGPIOAlarm(bytes) {
    // 1: gpio alarm, 0: gpio alarm release
    switch (bytes) {
        case 0:
            return "gpio alarm release";
        case 1:
            return "gpio alarm";
        default:
            return "unknown";
    }
}

function readWaterAlarm(bytes) {
    // 1: water outage timeout alarm, 2: water outage timeout alarm release, 3: water flow timeout alarm, 4: water flow timeout alarm release
    switch (bytes) {
        case 1:
            return "water outage timeout alarm";
        case 2:
            return "water outage timeout alarm release";
        case 3:
            return "water flow timeout alarm";
        case 4:
            return "water flow timeout alarm release";
        default:
            return "unknown";
    }
}

function readAlarm(bytes) {
    // 0: none, 1: water outage timeout alarm, 2: water outage timeout alarm release, 3: water flow timeout alarm, 4: water flow timeout alarm release, 5: gpio alarm, 6: gpio alarm release
    switch (bytes) {
        case 0:
            return "none";
        case 1:
            return "water outage timeout alarm";
        case 2:
            return "water outage timeout alarm release";
        case 3:
            return "water flow timeout alarm";
        case 4:
            return "water flow timeout alarm release";
        case 5:
            return "gpio alarm";
        case 6:
            return "gpio alarm release";
        default:
            return "unknown";
    }
}


var okPayload = {"bytes": [
    0x01, 0x75, 0x64, 0x03, 0x67, 0x19, 0x01, 0x04, 0x68, 0x64, 0x05, 0xe1, 0x0a, 0x00, 0x0a, 0x00, 0x00, 0x34, 0xb2, 0x46
]};
console.log(JSON.stringify(decodeUplink(okPayload)));

// AXVkA2cYAQRoZQXhCgAKAAA0skY=
// 0175 6403 6718 0104 6865 05e1 0a00 0a00 00 34 b2 46
var errorPayload = {"bytes": [
    0x01, 0x75, 0x64, 0x03, 0x67, 0x18, 0x01, 0x04, 0x68, 0x65, 0x05, 0xe1, 0x0a, 0x00, 0x0a, 0x00, 0x00, 0x34, 0xb2, 0x46
]};
console.log(JSON.stringify(decodeUplink(errorPayload)));
</script>

Can see the output for both payloads without any error.

image
image1904×396 20.8 KB

I guess QuickJS doesn’t like “readInt16LE” in the temperature.
You can test by replacing “decoded.temperature = readInt16LE(…” to decoded.temperature = 33.33
And see if there is still error.

You click on the + to show the error for the “log” entry.

5

Thank you for your prompt response. I have made the proposed change and this is the result.

image
image792×387 16.1 KB

image
image1055×142 4.43 KB

image
image
image640×558 20.8 KB

6

0175640367180104686505e10a000a000034b246 is ok for me.
Able to decode.

image
image1385×619 71.9 KB

7

0175640367190104686405e10a000a000034b246 is also OK for me.
Able to decode.

image
image1531×619 78.5 KB

Codec from you. No change.

image
image1525×710 23.3 KB

I’m using Chirpstack 4.8.1 on Ubuntu.

8

Are you sure you are using the latest ChirpStack v4.8.1?

I put in some extra brackets to simulate errors in codec.
Chirpstack v4.8.1 has detail error if the decode script has some errors.

image
image1548×423 57.7 KB

m

9

Indeed, I am on 4.8.1 running on this Dragino LPS8v2 hardware, these errors continue to appear randomly.
Thanks for taking your time!

image
image1498×719 51.9 KB

image

10

Oh, you are using the embedded ChirpStack on Dragino LPS8v2?
I havenot tried it yet.
No idea if it has performance issue or something.

I’m using ChirpStack on an Ubuntu VM.
The codec seems ok on my ChirpStack VM.

11

As a curiosity, Milesight support left us this comment that may make sense.

In addition, you also need to check your CPU usage. We tested that when the CPU usage is at 100%, QuickJS will timeout after 1 second, and the reported data will also prompt the same error.

I have not seen CPU overload problems, but I have not monitored it accurately either.
image

And here how the containers are.

image
image1627×214 11.2 KB

2 Likes
12

Thanks a lot for sharing.

I am also curious about the size of the project suitable for these embedded ChirpStack in Dragino LPS8v2.

How many gateways and devices are you using to connect to the embedded ChirpStack?

The sending interval seems to be every minute.

Thanks again.

13

At the moment I ONLY have one gateway, LPS8v2 itself, and three devices right now, I also run NODERED.

The interval time is 5min, each node sends the information.

1 Like
14

With so little devices and low sending frequency and there are a few “timeout” when decoding payload.

So it is better to use external Chirpstack server/Cloud.
Or we can find a way to increase the timeout for QuickJS or optimize Milesight’s codec.

15

I truly believe that the hardware of this destination is powerful enough for this application, I see the CPU load at very low values. Would it be very complicated to try to increase the timeout for QuickJS? Thank you!!

1 Like
16

Let me try to find how to increase QuickJS timeout at weekend.

No promise btw.

1 Like