Display data in ThingsBoard

gradoj · February 28, 2018, 10:21pm

Hello. I’m looking for some guidance on how best to publish data to a separate app server. I have an instance of LoraServer running and ThingsBoard running on a different computer. At first I thought I could get Thingsboard to subscribe to an mtqq topic but it seems its an mtqq server as well. I tried the http integration but couldn’t get that to work I think because ThingsBoard is expecting a flat name pair json format and Loraserver is sending the whole lorawan datapacket:

https://thingsboard.io/docs/reference/protocols/

{“applicationID”:“1”,“applicationName”:“testapp”,“deviceName”:“seeeduino”,“devEUI”:“xxxxf6064bfdxxxx”,“rxInfo”:[{“mac”:“xxxx40ffff29xxxx”,“rssi”:-49,“loRaSNR”:7.2,“name”:“LairdRG191”,“latitude”:xx.xxxx,“longitude”:-xxx.xxxx,“altitude”:0}],“txInfo”:{“frequency”:904300000,“dataRate”:{“modulation”:“LORA”,“bandwidth”:125,“spreadFactor”:7},“adr”:true,“codeRate”:“4/5”},“fCnt”:358,“fPort”:8,“data”:“A4gHybTumY0BrIQ=”,“object”:{“gpsLocation”:{“3”:{“latitude”:51.0388,“longitude”:-114.0339,“altitude”:1097}}}}

I finally got Thingsboard to accept data by forwarding mtqq messages with a python script. Notice I had to pull out just the geolocation data before publishing to Thingsboard:

def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe(“application/1/node/xxxxf6064bfdxxxx/#”)

def on_message(client, userdata, msg):
print(msg.payload.decode())
a=json.loads(msg.payload.decode())
print(json.dumps(a[‘object’][‘gpsLocation’][‘3’]))
pclient.publish(“v1/devices/me/telemetry”,json.dumps(a[‘object’][‘gpsLocation’][‘3’]))

I feel like I’m publishing to Thingsboard the hard way. What is the proper way of doing this? Can I use the http integration by formatting the data first? What have other people done? Thanks.

nikospps · March 1, 2018, 7:25am

Hello @gradoj!

Do you have any authentication privilleges in your MQTT Server?

gradoj · March 1, 2018, 5:00pm

I have to login to the LoraServer mtqq server but the Thingsboard mtqq server is open. But it is working and updating Thingsboard just fine. I really just want to know what the design intention is for LoraServer to get the payload to an AppServer. Is there a better way to do this? More complete code:

import paho.mqtt.client as mqtt
import json

def on_connect(client, userdata, flags, rc):
  print("Connected with result code "+str(rc))
  client.subscribe("application/1/node/xxxxf6064bfdxxxx/#")

def on_message(client, userdata, msg):
  print(msg.payload.decode())
  a=json.loads(msg.payload.decode())
  print(json.dumps(a['object']['gpsLocation']['3']))
  pclient.publish("v1/devices/me/telemetry",json.dumps(a['object']['gpsLocation']['3']))

client = mqtt.Client()
pclient = mqtt.Client()
client.username_pw_set('xxxxxx', password='xxxxxx')
pclient.username_pw_set('xxxxIU0fw0CmcLnbxxxx', password=None)
client.connect("x.x.x.x",1883,60)
pclient.connect("x.x.x.x",1883,60)
 
client.on_connect = on_connect
client.on_message = on_message

client.loop_forever()

nikospps · March 1, 2018, 5:05pm

Using the above code, you received data or you issued any-connection-problem.

gradoj · March 1, 2018, 5:12pm

Again code works fine here is the output of a new message:

Connected with result code 0

{“applicationID”:“1”,“applicationName”:“testapp”,“deviceName”:“seeeduino”,“devEUI”:“xxxxf6064bfdxxxx”,“rxInfo”:[{“mac”:“xxxx40ffff29xxxx”,“rssi”:-55,“loRaSNR”:9.8,“name”:“LairdRG191”,“latitude”:xx.xxxx,“longitude”:-xxx.xxxx,“altitude”:x}],“txInfo”:{“frequency”:904100000,“dataRate”:{“modulation”:“LORA”,“bandwidth”:125,“spreadFactor”:7},“adr”:true,“codeRate”:“4/5”},“fCnt”:5,“fPort”:8,“data”:“A4gAAAAAAAAAAAA=”,“object”:{“gpsLocation”:{“3”:{“latitude”:x,“longitude”:x,“altitude”:x}}}}
{“latitude”: x, “longitude”: x, “altitude”: x}

nikospps · March 1, 2018, 5:18pm

So, you receive rc=0 (Connection established) and you parse json data, as in mqtt broker or http integration, right?

gradoj · March 1, 2018, 5:28pm

Yes, I connect to LoraServer mtqq server(client) rc=0 and register a new message callback. When a new message is received I pull out the GPS coordinates and publish them to the ThingsBoard mtqq server(pclient). The ThingsBoard displays the location data properly as it is supposed to.

As i’ve stated this works fine just looking for the ‘proper’ way to do this. I assume LoraServer is designed to work with other AppServers such as ThingsBoard or The Things Network. I just don’t understand how I would use the HTTP integration feature as it is. Has anyone used TTN or ThingsBoard as an AppServer with LoraServer?

iegomez · March 1, 2018, 5:48pm

I think you are a bit confused, as you are actually running loraserver and lora-app-server right now. The latter offers ways of publishing your data to other parties, through an mqtt topic or an http integration, but it will send the whole json format. Also, a custom codec will operate over the byte array data and add it to the payload, but will keep the rest of the schema.

So, what you are really looking for is to change lora-app-server’s behaviour, or at least extend it, and then publish your data. This may be achieved in 2 ways:

The first and most straightforward is the one that you are attempting, i.e. having a middleman subscribe to the data topics (or to an http integration), reformat the data to your needs and then send it over to your final endpoint.

The other way is to modify lora-app-server from source to either change the existing functionality or extend it. For example, in my lora-app-server I wrote a custom function that publishes a custom mqtt message to my frontend. It goes like this:

// processUpData creates an rxMessage and notifies the websocket channel.
func (h *MQTTHandler) processUpData(b []byte, payload handler.DataUpPayload) {
	var iface interface{}

	json.Unmarshal(b, &iface)
	ifaceMap := iface.(map[string]interface{})
	dataStr, _ := ifaceMap["data"].(string)

	rxMessage := storage.RxMessage{
		DevEUI:     payload.DevEUI,
		Message:    dataStr,
		Rssi:       int32(payload.RXInfo[0].RSSI),
		Snr:        payload.RXInfo[0].LoRaSNR,
		GatewayMAC: payload.RXInfo[0].MAC,
	}

	//Decode base64 message
	_, decodeErr := base64.StdEncoding.DecodeString(rxMessage.Message)
	if decodeErr != nil {
		log.Errorf("handler/mqtt: rx_message create error: decode error %s", decodeErr)
		return
	}

	var clientData storage.ClientData

	clientData, sErr := storage.CreateRxMessage(config.C.PostgreSQL.DB, config.C.Sparkpost.Client, &rxMessage)
	if sErr != nil {
		log.Errorf("handler/mqtt: rx_message create error: %s", sErr)
		return
	}

	jsonClientData, _ := json.Marshal(clientData)
	jsonMessage := string(jsonClientData)

	topic := fmt.Sprintf("application/%d/device/%s/data", payload.ApplicationID, payload.DevEUI)
	fmt.Printf("Sending to frontend topic: %s\n", topic)
	if token := h.conn.Publish(topic, 0, false, []byte(jsonMessage)); token.Wait() && token.Error() != nil {
		log.Errorf("handler/mqtt: publish data to frontend error: %s", token.Error())
	}

	//Send alarms

	for _, v := range clientData.Alarms {
		if v.Show && v.Type != "none" {
			jsonAlarm, _ := json.Marshal(v)
			alarmMessage := string(jsonAlarm)
			alarmTopic := fmt.Sprintf("application/%d/device/%s/alarm", payload.ApplicationID, payload.DevEUI)

			fmt.Printf("Sending alarm topic: %s\n", alarmTopic)
			if token := h.conn.Publish(alarmTopic, 0, false, []byte(alarmMessage)); token.Wait() && token.Error() != nil {
				log.Errorf("handler/mqtt: publish alarm to frontend error: %s", token.Error())
			}
		}
	}

}

So then I just extend the SendDataUp function from the mqtt handler, firing a goroutine that executes my function:

// SendDataUp sends a DataUpPayload.
func (h *MQTTHandler) SendDataUp(payload handler.DataUpPayload) error {
	b, err := json.Marshal(payload)
	if err != nil {
		return fmt.Errorf("handler/mqtt: data-up payload marshal error: %s", err)
	}

	topic := fmt.Sprintf("application/%d/node/%s/rx", payload.ApplicationID, payload.DevEUI)
	log.WithField("topic", topic).Info("handler/mqtt: publishing data-up payload")

	if token := h.conn.Publish(topic, 0, false, b); token.Wait() && token.Error() != nil {
		return fmt.Errorf("handler/mqtt: publish data-up payload error: %s", err)
	}

	//Everything's fine, process the data
	go h.processUpData(b, payload)

	return nil
}

gradoj · March 2, 2018, 7:00pm

Thanks I appreciate it. I understand but may have been using the wrong terminology. Yes, I am running lora-app-server but I was thinking of ThingsBoard as the app-server but I really should just call it the front-end I guess. Time for me to get the toolchain setup.

kdannen · May 4, 2018, 7:21pm

Or you could just use something like node-red to filter, reformat and publish your data from one MQTT server to the other. Depending on the scope of your project this might be much quicker.

brocaar · July 10, 2019, 8:41am

Please note that LoRa App Server now integrates with ThingsBoard directly. This was released as LoRa App Server 3.1.0: https://forum.loraserver.io/t/release-lora-app-server-v3-1/5197

We have also made a guide to set this up: https://www.loraserver.io/guides/thingsboard/.

jayesh.patel · July 11, 2019, 3:46am

@brocaar

Does it measn we can integrate Thingsboard with community edition ? or we need to have Professional Edition to integrate with LoRa-App-Server ?

Regards
Jayesh

brocaar · July 11, 2019, 7:14am

I guess it works with both I have tested with the open-source version / community edition.

Ronald · August 3, 2019, 1:09pm

Hi. I am playing arround with thingsboard integrator. Currently I am using the live demo (you need a registration for it) and raspberry pi 3 with the ready to go image (https://artifacts.loraserver.io/downloads/lora-gateway-os/raspberrypi/raspberrypi3/3.0.0test1/).
My device (arduion mkr1300) sends json formatted string currently.

I configured as described here https://www.loraserver.io/lora-app-server/integrate/sending-receiving/thingsboard/.
If am using Base64 codec (default behavior in device profile configuration) so I see only the common properties in the thingsboard.io UI (details of device, e.g. application_id, dev_eui, …). Fine
BUT
It is not possible for me to get my specific payload to this platform. This should have something to do with codec function.
a) return a static content -> I see telemetry data in thingsboard UI
function Decode(fPort, bytes) {
return {“temperature”: 22.5, “pressure”: 0.8};
}

b) return convertet plain content -> I see no telemetry data in thingsboard UI
I checked this function in jsfiddle and its works with my sample data.
function Decode(fPort, bytes) {
var base64String = “”;
var decodedString = “”;

// convert array to string
for (var i = 0; i < bytes.length; ++i) {
base64String += String.fromCharCode(bytes[i]);
}

// decode base64 string to a string
decodedString = atob(base64String);
return decodedString;
}

c) I checked content with http integrator and hookbin (parallel to thingsboard)
If I am using static contents (above) -> working, If I am using my the function (above) only base64 encoded content is included.

Root cause of error
I reduced payload in my LoRa node and now I am able to see logs in device data tab. Here the problem is “js vm error: ReferenceError: ‘atob’ is not defined”.
Where I can see what frameworks are loaded in context or where I can add libraries (e.g. JSON, …)?

Thanks a lot.

Ronald · August 3, 2019, 3:51pm

Maybe another person will have the same problem. That´s why give the solution here.
The custom javascript codec function will not need to decode/encode base64 stuff. Means the underlying layer will do this for us. The input for decoding function is decoded base64 array.
So in my case I have to take the content and convert it into a string:

for (var i = 0; i < bytes.length; ++i) {
myDataString += String.fromCharCode(bytes[i]);
}

After that your specific protocoll have to implemented. In my case a separated values with “;”, e.g. 22.5;0.8

var values = decodedString.split(“;”)

At last put it into a new “json object”, e.g.

return {“temperature”: values[0], “pressure”: values[1]};

Hope this helps, if other people have the same problem in understanding this topic.

sanjai98 · August 20, 2019, 10:29am

I have integrated Loraserver and Thingsboard according to the guide.
I am receiving data on Loraserver but the not on Thingsboard.
can you please tell me what have i done wrong?

brocaar · August 20, 2019, 11:42am

Things that come to my mind for debuggin:

Did you configure the Thingsboard integration under your application
Did you configure the ThingsBoardAccessToken variable for your device
In the device data tab, do you see your decoded payloads
Did you check the logs of the lora-app-server process

sanjai98 · August 23, 2019, 6:30am

This is the logs of my lora-app-server.
Now i’m able to get the device attributes on thingsboard but im still not receiving any telemtry.

CJ_K · July 20, 2021, 10:34pm

Hey @brocaar, I am currently using the Thingsboard integration to TB CE, which works great. Im wondering if there are any plans to include downlinks to this integration?