De-duplication logic

shayan · December 19, 2017, 12:07pm

Hallo,

I’m using the LoRaServer product and everything is working great .

I had a question about one of my requirements that’s the deduplication of packets sent by a node and received by multiple gateways. I’m looking on the web and on the website but I can’t find anything about it, from what I see on my network there is also no duplicated data coming in.

The reason I need to know is that the teachers can ask questions when I’m defending my project in a couple of weeks so I need to know how it works.

Hope you guys can help me!
Greetings,
Shayan

brocaar · December 19, 2017, 12:41pm

This is the logic related to de-duplication:

github.com

brocaar/chirpstack-network-server/blob/master/internal/uplink/collect.go#L32


const (
	CollectKeyTempl     = "lora:ns:rx:collect:%s"
	CollectLockKeyTempl = "lora:ns:rx:collect:%s:lock"
)


// collectAndCallOnce collects the package, sleeps the configured duraction and
// calls the callback only once with a slice of packets, sorted by signal
// strength (strongest at index 0). This method exists since multiple gateways
// are able to receive the same packet, but the packet needs to processed
// only once.
// It is safe to collect the same packet received by the same gateway twice.
// Since the underlying storage type is a set, the result will always be a
// unique set per gateway MAC and packet MIC.
func collectAndCallOnce(p *redis.Pool, rxPacket gw.UplinkFrame, callback func(packet models.RXPacket) error) error {
	b, err := proto.Marshal(&rxPacket)
	if err != nil {
		return errors.Wrap(err, "marshal uplink frame error")
	}


	c := p.Get()
	defer c.Close()

shayan · December 19, 2017, 1:12pm

Thanks for the quick reaction!

hasiflo · October 23, 2018, 10:51am

I want to hook up on this topic.

As I remember the deduplication is done in the gateway-bridge. Is this right?
If I have a few gateways where the gateway-bridge is directly on the gateway not global at the server, and 2 gateways are in range of one node. Does the message then gets published twice at the mqtt?
Or is there also de-duplication at the network-server?

Best regards.

iegomez · October 23, 2018, 12:42pm

No, it’s done at the loraserver level.

sil · October 17, 2019, 1:39pm

Hi, i also have a question about deduplication.

If LoraServer deduplicates a packet, can i see this in the logs?
so far i could not find it on the LoraServer Logs - but maybe i just don’t know what message to see.

So: Does anyone know if e deduplicated and deleted packet is logged? If so, where? and which message should show up?

regards,
Sil

bconway · October 17, 2019, 2:16pm

Deduplication is such a common occurrence with a LoRa network, I don’t think it would make sense to log it, or at least not by default. The most obvious indication that a payload was deduped is that it shows multiple gateways in the device payload (assuming you have GW metadata turned on).