Uplink rejected after end device wakes up and sends data

Hi All,

I am running some experiments with low power end device. The device is a Heltec Wireless Stick Lite (Arduino) and it’s using LMIC library found here: https://github.com/mcci-catena/arduino-lmic

Basically following the pattern below:

  1. Join Network
  2. Send sensor data payload
  3. Save session data (network id, devaddr, session keys and up/dn counters)
  4. Go to sleep for some time (1 minute for testing)
  5. Wake up device and LoRa radio
  6. Restore previously saved session data using LMIC_setSession()
  7. Repeat from step #2

The uplink message is received by the server as unconfirmed, but it is rejected with the following error

Sep 04 11:18:01 rak-gateway loraserver[548]: time="2019-09-04T11:18:01-07:00" level=error msg="processing uplink frame error" data_base64=QKsjAQCAAAABFXvlQq7sWi5Rd7DT error="get device-session error: device-session does not exist or invalid fcnt or mic"

I am not sure what device-session is?

Well, I found the issue. The fCnt was wrong and always set to 0. That’s because on the end device, the counters must be set after the call to restore the LMIC session, otherwise the counters are reset.

So the correct LMIC sequence call is

 LMIC_setSession(netid, devaddr, nwkKey, artKey);
 LMIC.seqnoUp = last_sequence_up;
 LMIC.seqnoDn = last_sequence_dn;

Now the Lora Server is happy :wink: and fCnt is incremented as expected

Hey @amir, I am also trying to use wireless stick lite with MCCI-LMIC, and the Join Request never gets accepted for some reason. Could you post your example code and which version of LMIC you are using? Thanks a bunch!

@stk The issue I posted here is very specific and it only happened when using the low power mode on the wireless stick. The ESP32-Pico low power mode does not preserve memory. So the board soft restarts when waking up (i.e it will execute the setup call again).
And for that reason the Join session has to be stored in flash memory and restore as if doing an ABP (instead of the initial OTAA). That didn’t work for me, because I was not preserving the message counters and restoring them.

The LMIC library I used is this one: https://github.com/mcci-catena/arduino-lmic and my code is pretty much the same as the ttn_otaa example here: https://github.com/mcci-catena/arduino-lmic/blob/master/examples/ttn-otaa/ttn-otaa.ino
with the pin mapper adapted for the ESP32-Pico.

I can post the example later (not at my computer right now).
If the Join request is failing for you initially (i.e regardless of low power), there could be multiple causes:

  1. Keys on the end device don’t match the ones defined on the loraserver side: so double/triple check these APPEUI, DEVEUI and APPKEY.
  2. Make sure the lmic_pins are defined correctly for your board
  3. Make sure the lmic/config.h from the library has the correct definition for your region. I am in the US, so I had to define
// the HopeRF RFM95 boards.
#define CFG_sx1276_radio 1 

And the frequency configuration also

//# define CFG_eu868 1
# define CFG_us915 1
  1. Makes sure the channel list used on the end device matches what is defined on your gateway. You have to make sure you add this. Othewise, you may not receive any messages from the gateway.
    // in the US, with TTN, it saves join time if we start on subband 1 (channels 8-15). This will
    // get overridden after the join by parameters from the network. If working with other
    // networks or in other regions, this will need to be changed.
    LMIC_selectSubBand(1);

I think these are the most common things I had to check to have a successful Join request.

As a side note, I have given up on the Heltec ESP32 boards as I could never achieve the low power they claim the board was capable of. As much as I really like working with the Espressif chip, the Heltec boards didn’t meet my requirements.
I am currently using prototypes from Rocket Scream Technologies based on the Samd21 https://www.rocketscream.com/blog/product/mini-ultra-pro-v3-with-radio/. I am able to achieve about 20uA in standby mode, and around 14K uplinks on a 750mAh rechargeable CR123a battery. My devices sleep for 30 minutes and then sends between 6-12 bytes when they are awake.

Cheers,
Amir.

Hey @amir, thanks a lot for checking back and the detailed answer!

I know about the OTAA keys being lost when not saving them to non-volatile memory and re-establishing the session later on with matching uplink and downlink counters. I was trying to get OTAA (or any downlink, for that matter) to work with the wireless stick lite, and consistently failed (even though the sketch worked with other hardware platforms). I also used the same arduino-lmic (different versions, since the last one appears to have broken code written against earlier versions). Keys were correct (they worked on other platforms), pins appeared to have been correct (sending messages via APB worked, but the downlink message after an OTAA join request or any downlink message was not processed), and all the relevant settings for frequencies were up to speed.

I suspected this was a timing issue, but even with the relaxed settings I could not get the downlink window to work properly. Any code examples that worked for you would be a great help! But good to know that the low power settings didn’t work anyways for you. Maybe Heltec is just a dead end :disappointed:

#include <heltec.h>
#include <lmic.h>
#include <hal/hal.h>
#include <SPI.h>
#include <Arduino.h>
#include <math.h>
#include "esp_deep_sleep.h"
#include <driver/adc.h> 
#include <driver/dac.h> 


#define uS_TO_S_FACTOR 1000000  /* Conversion factor for micro seconds to seconds */
#define TIME_TO_SLEEP  30        /* Time ESP32 will go to sleep (in seconds) */

// Schedule TX every this many seconds (might become longer due to duty cycle limitations).
//const unsigned TX_INTERVAL = 60 * 30;
const unsigned TX_INTERVAL = 60;


// This EUI must be in little-endian format, so least-significant-byte
// first. When copying an EUI from ttnctl output, this means to reverse
// the bytes. For TTN issued EUIs the last bytes should be 0xD5, 0xB3,
// 0x70.
static const u1_t PROGMEM APPEUI[8] = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
void os_getArtEui (u1_t* buf) {
  memcpy_P(buf, APPEUI, 8);
}

// This should also be in little endian format, see above.
static const u1_t PROGMEM DEVEUI[8] = { //add dev eui here };
void os_getDevEui (u1_t* buf) {
  memcpy_P(buf, DEVEUI, 8);
}

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from ttnctl can be copied as-is.
static const u1_t PROGMEM APPKEY[16] = { //add app key here };
void os_getDevKey (u1_t* buf) {
  memcpy_P(buf, APPKEY, 16);
}

const uint8_t PACKET_SIZE = 8;
uint8_t fullPayload[PACKET_SIZE];

static osjob_t sendjob;

// Pin mapping for Heltec ESP32 and Wireless Stick Lite
const lmic_pinmap lmic_pins = {
  .nss = 18,  //CS pin
  .rxtx = LMIC_UNUSED_PIN,
  .rst = 14, //RST PIN
  //.dio = {26, 35, 34},  //DIO 0, 1, 2
  .dio = {26, 35, LMIC_UNUSED_PIN},  //DIO 0, 1, 2
};

// save the session info in RTC memory during deep sleep
RTC_DATA_ATTR u4_t netid = 0;
RTC_DATA_ATTR devaddr_t devaddr = 0;
RTC_DATA_ATTR u1_t nwkKey[16];
RTC_DATA_ATTR u1_t artKey[16];
RTC_DATA_ATTR bool isShutdown = false;
RTC_DATA_ATTR u4_t sequence_up = 0;
RTC_DATA_ATTR u4_t sequence_dn = 0;

void printSessionInfo() {
  Serial.print("netid: ");
  Serial.println(netid, DEC);
  Serial.print("devaddr: ");
  Serial.println(devaddr, HEX);
  Serial.print("appSessionKey: ");
  for (int i = 0; i < sizeof(artKey); ++i) {
    Serial.print(artKey[i], HEX);
  }
  Serial.println("");
  Serial.print("netSessiomKey: ");
  for (int i = 0; i < sizeof(nwkKey); ++i) {
    Serial.print(nwkKey[i], HEX);
  }
  Serial.println("");
  Serial.print("sequence up: ");
  Serial.println(sequence_up, DEC);
  Serial.print("sequence dn: ");
  Serial.println(sequence_dn, DEC);
  Serial.println("");

}

void shutDownRadio() {
   Serial.println(F("Shutting LMIC Down"));
   sequence_up = LMIC.seqnoUp;
   sequence_dn = LMIC.seqnoDn;
   LMIC_shutdown();
   isShutdown = true;   
}

void onEvent (ev_t ev) {
  switch (ev) {
    case EV_SCAN_TIMEOUT:
      Serial.println(F("EV_SCAN_TIMEOUT"));
      break;
    case EV_BEACON_FOUND:
      Serial.println(F("EV_BEACON_FOUND"));
      break;
    case EV_BEACON_MISSED:
      Serial.println(F("EV_BEACON_MISSED"));
      break;
    case EV_BEACON_TRACKED:
      Serial.println(F("EV_BEACON_TRACKED"));
      break;
    case EV_JOINING:
      Serial.println(F("EV_JOINING"));
      break;
    case EV_JOINED:
      Serial.println(F("EV_JOINED"));
      LMIC_getSessionKeys(&netid, &devaddr, nwkKey, artKey);
      printSessionInfo();
      LMIC_setLinkCheckMode(0);

      break;
    case EV_RFU1:
      Serial.println(F("EV_RFU1"));
      break;
    case EV_JOIN_FAILED:
      Serial.println(F("EV_JOIN_FAILED"));
      break;
    case EV_REJOIN_FAILED:
      Serial.println(F("EV_REJOIN_FAILED"));
      break;
    case EV_TXCOMPLETE:
      Serial.println(F("EV_TXCOMPLETE Done Sending data (includes waiting for RX windows)"));
      if (LMIC.txrxFlags & TXRX_ACK)
        Serial.println(F("Received ack"));
      if (LMIC.dataLen) {
        Serial.print(F("Received "));
        Serial.print(LMIC.dataLen);
        Serial.println(F(" bytes of payload"));
      }
      // Schedule next transmission
      os_setTimedCallback(&sendjob, os_getTime() + sec2osticks(TX_INTERVAL), do_send);

      Serial.println(F("Shutting LMIC Down"));
      sequence_up = LMIC.seqnoUp;
      sequence_dn = LMIC.seqnoDn;
      
      Serial.print("LMIC sequence up number:");
      Serial.println (sequence_up);
      Serial.print("LMIC sequence dn number:");
      Serial.println(sequence_dn);
      
      LMIC_shutdown();
      isShutdown = true;
      Serial.println(F("Going to sleep now"));
      Serial.flush();
      pinMode(25, OUTPUT); digitalWrite(25, LOW); //The on board LED will be OFF in wake up period
      delay(100);

      pinMode(23, INPUT);

      pinMode(36, INPUT);
      pinMode(37, INPUT);
      pinMode(38, INPUT);
      pinMode(39, INPUT);
      pinMode(4, INPUT);
      pinMode(2, INPUT);
      pinMode(15, INPUT);
      pinMode(32, INPUT);
      pinMode(33, INPUT);
      pinMode(25, INPUT);
      pinMode(9, INPUT);
      pinMode(10, INPUT);
      adc_power_off();
      esp_deep_sleep_start();
      break;

    case EV_LOST_TSYNC:
      Serial.println(F("EV_LOST_TSYNC"));
      break;

    case EV_RESET:
      Serial.println(F("EV_RESET"));
      break;

    case EV_RXCOMPLETE:
      // data received in ping slot
      Serial.println(F("EV_RXCOMPLETE"));
      break;

    case EV_LINK_DEAD:
      Serial.println(F("EV_LINK_DEAD"));
      break;
    case EV_LINK_ALIVE:
      Serial.println(F("EV_LINK_ALIVE"));
      break;
    case EV_SCAN_FOUND:
      Serial.println(F("EV_SCAN_FOUND"));
      break;
    case EV_TXSTART:
      Serial.println(F("EV_TXSTART"));
      break;
    default:
      break;
  }
}

void initLoraRadio() {

  dac_i2s_disable();

  // LMIC init
  Serial.println("LMIC: Initializing...");
  os_init();
  
  // Reset the MAC state. Session and pending data transfers will be discarded.
  Serial.println("LMIC: Reseting...");
  LMIC_reset();
  // this is hack to correct timing for RX1 and RX2 receive window
  LMIC_setClockError(5 * MAX_CLOCK_ERROR / 100);
  
  //For CFG_us915 - 
  LMIC_selectSubBand(1);
  if (isShutdown) {
    isShutdown = false;
    Serial.println("Was shutdown..set session");
    printSessionInfo();
    LMIC_setSession(netid, devaddr, nwkKey, artKey);
    LMIC_setSeqnoUp(sequence_up);
    LMIC.seqnoDn = sequence_dn;
  }
}


void print_wakeup_reason() {
  esp_sleep_wakeup_cause_t wakeup_reason;

  wakeup_reason = esp_sleep_get_wakeup_cause();

  switch (wakeup_reason)
  {
    case ESP_SLEEP_WAKEUP_EXT0 : Serial.println("Wakeup caused by external signal using RTC_IO"); break;
    case ESP_SLEEP_WAKEUP_EXT1 : Serial.println("Wakeup caused by external signal using RTC_CNTL"); break;
    case ESP_SLEEP_WAKEUP_TIMER : Serial.println("Wakeup caused by timer"); break;
    case ESP_SLEEP_WAKEUP_TOUCHPAD : Serial.println("Wakeup caused by touchpad"); break;
    case ESP_SLEEP_WAKEUP_ULP : Serial.println("Wakeup caused by ULP program"); break;
    default : Serial.printf("Wakeup was not caused by deep sleep: %d\n", wakeup_reason); break;
  }
}

void setup() {

  Serial.begin(9600);
  delay(4000);
  
  adc_power_on();

  print_wakeup_reason();
  esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
  esp_deep_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_OFF);
  Serial.println("Setup ESP32 to sleep for every " + String(TIME_TO_SLEEP) + " Seconds");

  Serial.println("STARTING...");

  initLoraRadio();

  Serial.println("LMIC: Starting...");
  do_send(&sendjob);
}

void do_send(osjob_t* j) {
  // Check if there is not a current TX/RX job running
  if (LMIC.opmode & OP_TXRXPEND) {
    Serial.println(F("OP_TXRXPEND, not sending"));
  } else {
    Serial.println("[do_send] Get sensor data...");

    get_sensor_data();
    LMIC_setTxData2(1, fullPayload, sizeof(fullPayload), 0);
  }
  // Next TX is scheduled after TX_COMPLETE event.
}

void get_sensor_data() {
  uint16_t l, p;
  int8_t t, h, m, g;

  // some fake data
    t = 21;
    p = 1012;
    h = 45;
    m = 75;
    g = 17;
    l = 1234;

  fullPayload[0] = t;
  fullPayload[1] = h;
  fullPayload[2] = lowByte(p);
  fullPayload[3] = highByte(p);
  fullPayload[4] = m;
  fullPayload[5] = g;
  fullPayload[6] = lowByte(l);
  fullPayload[7] = highByte(l);
}

void loop() {
  os_runloop_once();
}

This is the sample code, including some attempts at powering down components of the board to improve the low power current. The lowest I could get was 1.5mA which is really far from the 30uA they claimed. I even tried to use their own sample app (not using any LoraWAN) and couldn’t get that low. But the join before and after sleep works well.

One other thing I forgot to mention is the use of
LMIC_setClockError(5 * MAX_CLOCK_ERROR / 100);

I had to increase the error tolerance rate to compensate for the arduino internal oscillator lack of precision.
And in addition to this, make sure your device is at least 3m away from the gateway you are testing with to prevent the end device signal to bounce back. Also I have my loraserver DeviceProfile_Otaa profile configure with LoraWAN MAC version 1.0.2. I am not sure if the latest version of LMIC supports 1.0.3.

There are lots of small details and lots of trial and error :slight_smile:

There are possibly two distinct problems here.

For one, re-using already used join nonces is a problem. LMiC has a habit of randomly picking one, but then linearly incrementing from there. So if your random number generator is bad, you’ll take longer and longer each time to walk to an unused number. Of course you should not be rejoining often to begin with.

Another is that timing on the ESP32 platform seems somewhere between broken and unclear. If you don’t care about burning battery you could configure the radio to start receiving early and for longer (more symbols of preamble search) that reasonable. But efficient usage would depend on figuring out how to do accurate time delays on the platform, which is complex on the ESP32 in the realm of ESP-IDF, and only gets worse when adding an Arduino emulation layer on top.

Hello @amir,
good jobs to switch off and restart LoRa stack, but there are some details that i don’t understand.

By using LMIC_reset,

  • you are always transmitting on the same channel when you call LMIC_setTxData2 ?
  • also, you are breaking the duty cycle if you call the do_send function too early ?

Erwan

Hi @Erwan

I am not totally sure, but I don’t think the Reset causes LMIC to always send on the same channel. At least that doesn’t appear to be the case, based on the logs I see from the loraserver. I haven’t looked too deeply in the library to see how the channel is chosen. But, I know you can configure and even force LMIC to send on specific channels. I think that’s how you can actually use single channel gateways (e.g by disabling channels using
LMIC_disableChannel).

I haven’t looked into the duty cycles at all, since the device I am working on will sleep between 15 to 60 minutes, before sending its sensor data. So I think the interval is long enough. It is all dependent on the region/country you are in. Also I know some radios will check the duty cycle and notify when there is no free channel available, so it’s possible LMIC library does take advantage of that to ensure the duty cycle isn’t broken.

The send is scheduled after TX_COMPLETE is received, which includes all RX intervals and do_send will call LMIC_setTxData2 only if there is no pending TX or RX. So I think the code above is fine. The only think I noticed is that it is scheduling the next send, and then going to sleep, which doesn’t seem totally right. In this case, since the device will restart, I don’t think the scheduling is needed.