Redis data format error after upgrade

vdavy · November 7, 2019, 9:31am

Hi there,

We currently have the stack in these old version :

Loraappserver : 0.21.0
Gatewaybridge : 2.4.0
Loraserver: 0.26.3

And we want to upgrade to the latest version. We have already set up a new server with new stack and it is working with our sensors so our new config files are OK. We have also the outputed data form sensors on MQTT.

During the migration, I steped the upgrade with these versions for network server :

1.0.0
2.0.0
2.8.2
3.0.0
latest

We got no error for the postgres db, so I did the same process for app server part :

1.0
2.0.0
2.6.1
3.0.0
3.1.0
3.3.0
latest

Right here, no problem with postgres part. Finally for the gateway bridge, I directly use the latest version available.

But I think the redis data format was not properly upgraded to the new protobuf format introduced in network server v2.0.0 (according to changelog https://www.chirpstack.io/network-server/overview/changelog/)

Here is some stuff making me thinking this :

this error “wrong type (lorawan.AES128Key) for received device session nwkskey (code 2)” - see attached screenshot
these logs in the app server when displaying the above page :
time=“2019-11-07T08:42:13Z” level=error msg=“finished client unary call” ctx_id=25ed3198-d7f1-44ad-b495-12174b7fb09d error=“rpc error: code = Unknown desc = gob: wrong type (lorawan.AES128Key) for received field DeviceSession.NwkSKey” grpc.code=Unknown grpc.ctx_id=b1e2b66b-f150-4ef3-9397-81b2d1297a93 grpc.duration=1.998309ms grpc.method=GetDeviceActivation grpc.service=ns.NetworkServerService span.kind=client system=grpc
time=“2019-11-07T08:42:13Z” level=error msg=“finished unary call with code Unknown” ctx_id=25ed3198-d7f1-44ad-b495-12174b7fb09d error=“rpc error: code = Unknown desc = gob: wrong type (lorawan.AES128Key) for received field DeviceSession.NwkSKey” grpc.code=Unknown grpc.method=GetActivation grpc.service=api.DeviceService grpc.start_time=“2019-11-07T08:42:12Z” grpc.time_ms=194.45 peer.address=“127.0.0.1:55438” span.kind=server system=grpc
I went into the redis and displayed 2 keys : one for a previous device (using ABP) added before the upgrade (devEUI 000000002100b46a) and another one (devEUI 0018b200000211db) which had the old data format (not working), so I made a new JR/JA and the device was working after, but the keys was regenerated during OTAA and keys format in redis seems to have been upgraded/regenerated

Here is the data :

127.0.0.1:6379> get lora:ns:device:000000002100b46a
"\xfe\x02\xa5\xff\x9d\x03\x01\x01\rDeviceSession\x01\xff\x9e\x00\x01$\x01\x0fDeviceProfileID\x01\x0c\x00\x01\x10ServiceProfileID\x01\x0c\x00\x01\x10RoutingProfileID\x01\x0c\x00\x01\aDevAddr\x01\xff\xa0\x00\x01\x06DevEUI\x01\xff\x86\x00\x01\aJoinEUI\x01\xff\x86\x00\x01\aNwkSKey\x01\xff\xa2\x00\x01\x06FCntUp\x01\x06\x00\x01\bFCntDown\x01\x06\x00\x01\x12SkipFCntValidation\x01\x02\x00\x01\bRXWindow\x01\x04\x00\x01\aRXDelay\x01\x06\x00\x01\x0bRX1DROffset\x01\x06\x00\x01\x05RX2DR\x01\x06\x00\x01\x0cRX2Frequency\x01\x04\x00\x01\x0cTXPowerIndex\x01\x04\x00\x01\x02DR\x01\x04\x00\x01\x03ADR\x01\x02\x00\x01\x18MinSupportedTXPowerIndex\x01\x04\x00\x01\x18MaxSupportedTXPowerIndex\x01\x04\x00\x01\x0eMaxSupportedDR\x01\x04\x00\x01\aNbTrans\x01\x06\x00\x01\x0fEnabledChannels\x01\xff\xa4\x00\x01\x15EnabledUplinkChannels\x01\xff\xa4\x00\x01\x13ExtraUplinkChannels\x01\xff\xa8\x00\x01\x12ChannelFrequencies\x01\xff\xa4\x00\x01\rUplinkHistory\x01\xff\xac\x00\x01\rLastRXInfoSet\x01\xff\xae\x00\x01\x16LastDevStatusRequested\x01\xff\x88\x00\x01\x14LastDevStatusBattery\x01\x06\x00\x01\x13LastDevStatusMargin\x01\x04\x00\x01\x0eLastDownlinkTX\x01\xff\x88\x00\x01\x0cBeaconLocked\x01\x02\x00\x01\nPingSlotNb\x01\x04\x00\x01\nPingSlotDR\x01\x04\x00\x01\x11PingSlotFrequency\x01\x04\x00\x00\x00\x13\xff\x9f\x06\x01\x01\aDevAddr\x01\xff\xa0\x00\x00\x00\x11\xff\x85\x06\x01\x01\x05EUI64\x01\xff\x86\x00\x00\x00\x19\xff\xa1\x01\x01\x01\tAES128Key\x01\xff\xa2\x00\x01\x06\x01 \x00\x00\x13\xff\xa3\x02\x01\x01\x05[]int\x01\xff\xa4\x00\x01\x04\x00\x00%\xff\xa7\x04\x01\x01\x14map[int]band.Channel\x01\xff\xa8\x00\x01\x04\x01\xff\xa6\x00\x00.\xff\xa5\x03\x01\x02\xff\xa6\x00\x01\x03\x01\tFrequency\x01\x04\x00\x01\x05MinDR\x01\x04\x00\x01\x05MaxDR\x01\x04\x00\x00\x00&\xff\xab\x02\x01\x01\x17[]storage.UplinkHistory\x01\xff\xac\x00\x01\xff\xaa\x00\x00Q\xff\xa9\x03\x01\x01\rUplinkHistory\x01\xff\xaa\x00\x01\x04\x01\x04FCnt\x01\x06\x00\x01\x06MaxSNR\x01\b\x00\x01\x0cTXPowerIndex\x01\x04\x00\x01\x0cGatewayCount\x01\x04\x00\x00\x00\x18\xff\xad\x02\x01\x01\tRXInfoSet\x01\xff\xae\x00\x01\xff\x9a\x00\x00v\xff\x99\x03\x01\x01\x06RXInfo\x01\xff\x9a\x00\x01\b\x01\x03MAC\x01\xff\x86\x00\x01\x04Time\x01\xff\x88\x00\x01\x11TimeSinceGPSEpoch\x01\x04\x00\x01\tTimestamp\x01\x06\x00\x01\x04RSSI\x01\x04\x00\x01\aLoRaSNR\x01\b\x00\x01\x05Board\x01\x04\x00\x01\aAntenna\x01\x04\x00\x00\x00\n\xff\x87\x05\x01\x02\xff\x8e\x00\x00\x00\xfe\x02\x11\xff\x9e\x01$836e8ada-679e-4fc4-a816-31a385afa48f\x01$a1f47f30-dee1-466a-a919-3174014f615d\x01$6d5db27e-4ce2-4b2b-b5d7-91f069397978\x01\x04j\xb4\x00!\x01\bj\xb4\x00!\x00\x00\x00\x00\x02\x10\x19\xff\x84+\xff\xd9GC$k6|.\xff\x90\xff\x94*\x1fr\x01\xfe\x16\xf3\x01\xfe\x01v\x01\x01\x02\x01\x03\xfcg\xa7\xcc\x10\x02\n\x05\x01\x02\x03\x00\x02\x04\x01\x00\x02\x14\x01\xfe\x16\xdf\x01\xfe\x16@\x02\x02\x00\x01\xfe\x16\xe0\x01\xf8\x9a\x99\x99\x99\x99\x99%@\x02\x02\x00\x01\xfe\x16\xe1\x01\xfe&@\x02\x02\x00\x01\xfe\x16\xe2\x01\xfe#@\x02\x02\x00\x01\xfe\x16\xe3\x01\xf8ffffff\"@\x02\x02\x00\x01\xfe\x16\xe4\x01\xf8ffffff$@\x02\x02\x00\x01\xfe\x16\xe5\x01\xfe%@\x02\x02\x00\x01\xfe\x16\xe6\x01\xf8\x9a\x99\x99\x99\x99\x99!@\x02\x02\x00\x01\xfe\x16\xe7\x01\xf8ffffff$@\x02\x02\x00\x01\xfe\x16\xe8\x01\xf8ffffff\"@\x02\x02\x00\x01\xfe\x16\xe9\x01\xfe$@\x02\x02\x00\x01\xfe\x16\xea\x01\xfe\x1e@\x02\x02\x00\x01\xfe\x16\xeb\x01\xfe%@\x02\x02\x00\x01\xfe\x16\xec\x01\xf8\x9a\x99\x99\x99\x99\x99%@\x02\x02\x00\x01\xfe\x16\xed\x01\xf8ffffff&@\x02\x02\x00\x01\xfe\x16\xee\x01\xfe\"@\x02\x02\x00\x01\xfe\x16\xef\x01\xfe%@\x02\x02\x00\x01\xfe\x16\xf0\x01\xfe\"@\x02\x02\x00\x01\xfe\x16\xf1\x01\xfe\"@\x02\x02\x00\x01\xfe\x16\xf2\x01\xf8\x9a\x99\x99\x99\x99\x99#@\x02\x02\x00\x01\x01\x01\b\xf0\x00\x00 \x00\x00\x00\x00\x01\x0f\x01\x00\x00\x00\x0e\xd5T\x82~5~\x13\x98\xff\xff\x02\xfc\xb7\x84\x05\xf3\x01\xff\xcb\x01\xf8\x9a\x99\x99\x99\x99\x99#@\x00\x03\xff\xfe\x01\x0f\x01\x00\x00\x00\x0e\xd5T\x82\x7f\x17\xca\xea\xad\x00\x00\x00"
127.0.0.1:6379> get lora:ns:device:0018B200000211DB
(nil)
127.0.0.1:6379> get lora:ns:device:0018b200000211db
"\n$c6befdf9-c5af-4ac9-93f7-d4721c3c9f03\x12$3cf1abef-8952-4d1e-95f9-81cca85d9c08\x1a$6d5db27e-4ce2-4b2b-b5d7-91f069397978\"\x04\x06\\=\x89*\b\x00\x18\xb2\x00\x00\x02\x11\xdb2\b\xcb)\x87\xe4:\x06\xbe\xf5:\x10\xde\xd3\x1f\xc7\x12\x04\xc6\xa6\xae\xdb\x1e\\C\xe7\xcdCB\x10\xde\xd3\x1f\xc7\x12\x04\xc6\xa6\xae\xdb\x1e\\C\xe7\xcdCJ\x10\xde\xd3\x1f\xc7\x12\x04\xc6\xa6\xae\xdb\x1e\\C\xe7\xcdCP\x06X\x05p\x01\x88\x01\x88\xcc\xcf\x9e\x03\x90\x01\x01\xa0\x01\x01\xa8\x01\x05\xb8\x01\x01\xc2\x01\x03\x00\x01\x02\xda\x01\t\b\x01\x15\x00\x00\xe0@ \x01\xda\x01\x0b\b\x02\x15\x00\x00\x18A\x18\x01 \x01\xda\x01\x0b\b\x03\x15\x00\x00\x00A\x18\x01 \x01\xda\x01\x0b\b\x04\x15\x00\x00(A\x18\x01 \x01\xda\x01\x0b\b\x05\x15\x00\x00 A\x18\x01 \x01\xe8\x01\x80\x80\xe8\xe8\xb3\xfd\x80\xd9\xa1\x01\x80\x02\x8d\xe4\xf4\x85\xc2\xe9\xb4\xea\x15\xaa\x02\x051.0.1\xb0\x02\x01"

Am I right thinking this error is coming from here ?
How we can have the redis data upgraded to new protobuf format ? Many of our nearly 100000 devices recorded in network server are using abp, so we can’t use OTAA rejoin to regenerate keys.

Many thanks

fr3driiik · November 21, 2019, 7:10am

I have almost the exact same problem.
Started from repository for version 0.
Upgraded everything by changing repository version to 1. Everything went fine, data still flowing.
Upgraded everything by changing repository version to 2. Now I have the same redis device session format error. I can see that there is some migration code for this, but it gives an error.

I can get the same error by running loraserver print-ds *deviceEUI*, which gives

FATA[0000] get device-session error error="gob decode error: gob: wrong type (lorawan.AES128Key) for received field DeviceSession.NwkSKey"

I will need to do the same upgrade to an installation with more than 3000 devices, so I would like to find out how this can be solved. I can not be sure that the devices will try to rejoin, so I need to keep the session.

vdavy · November 21, 2019, 8:20am

Seems to be the same error.

Where have you seen migration code for redis data ? Might interested to have a look to it.

fr3driiik · November 21, 2019, 8:43am

Here is the part of the code that looks like it should do the migration and creates the errors doing so: GetDeviceSession with migration for version 2

vdavy · November 21, 2019, 1:56pm

Yes, I reached it - thanks.

Something is questioning me : the old data is never converted to new format when reaching former data (gob serialization).
Maybe it would be interesting to convert the old data to new protobuf format and overwrite the gob data with the new protobuf format ?

brocaar · November 21, 2019, 2:32pm

What happens (and still is present) is that when Protobuf unmarshal fails, it tries using Gob. Then on write data is written into the binary Protobuf format.

The actual cause has to do with:

Gob can encode a value of any type implementing the GobEncoder or encoding.BinaryMarshaler interfaces by calling the corresponding method, in that order of preference.

Gob can decode a value of any type implementing the GobDecoder or encoding.BinaryUnmarshaler interfaces by calling the corresponding method, again in that order of preference.

In the 0.26.x versions, lorawan.AES128Key did not implement the BinaryMarshaler and BinaryUnmarshaler interfaces. Support for these interfaces was added ~ 10 months ago. This changed the way how Gob tries to decode the lorawan.AES128Key field and causes the error.

The error goes away when removing the interface implementation (MarshalBinary and UnmarshalBinary methods).

github.com

brocaar/lorawan/blob/master/phypayload.go#L106


      
          	if len(b) != len(k) {
          		return fmt.Errorf("lorawan []byte must have length %d", len(k))
          	}
          	copy(k[:], b)
          	return nil
          }
          
          // Value implements driver.Valuer.
          func (k AES128Key) Value() (driver.Value, error) {
          	return k[:], nil
          }
          
          // MarshalBinary encodes the key to a slice of bytes.
          func (k AES128Key) MarshalBinary() ([]byte, error) {
          	b := make([]byte, len(k))
          	for i, v := range k {
          		// little endian
          		b[len(k)-i-1] = v
          	}
          	return b, nil
          }

fr3driiik · November 21, 2019, 2:45pm

Alright, thanks for the clarification.
Can you think of any workaround that does not include modifying code and rebuilding?