Restrict MQTT Broker

RogerioCassares · February 9, 2018, 3:05pm

I am sorry.

Here it is:

mosquitto version 1.4.8 (build date Mon, 26 Jun 2017 09:31:02 +0100)

iegomez · February 9, 2018, 3:19pm

Oh, that’s a pretty outdated version, from a year ago or so. I remember having some issues with mosquitto alone, and also when running jpmen’s plugin (never tried older versions with mine) with the mosquitto version from the distro’s packages, which I think was 1.4.9 or something like that.
Since that version, they’ve fixed a lot of issues. Also, I recall having big troubles related to websockets with the old version, which were fixed when I switched to current one, 1.4.14.

Perhaps updating your mosquitto version could “magically” fix everything. Maybe there’s some hope! The sad part is that, unless this package works off the shell for you, or there’s another deb package available somewhere, you’ll have to build it yourself.

But not to worry! Though really outdated, this docs helped me build mosquitto with websockets support some time ago: https://github.com/bapowell/bapowell.github.io/wiki/Mosquitto-Build-Notes-(Linux). Just look for the newest version of every package instead of using the links given there. And the plus is that, if you ever want to connect your js frontend to mosquitto, you’ll already have the websockets part ready.

RogerioCassares · February 9, 2018, 6:23pm

Thanks to advise that mosquitto is not updated!
Certainly I will check!

But, @iegomez maybe i have some good news!

I have just follow the steps above that I have mentioned to reproduce the error.

EDIT:

2.1. sudo timedatectl set-ntp 0
2.2. sudo date +%T -s "06:20:00"
2.3. sudo hwclock --systohc

STARTING MANNUALLY MOSQUITTO: sudo mosquitto -c /etc/mosquitto/mosquitto.conf

INFO[0262] Files acl check with user loraappserver, topic: application/1/node/0004a30b001e0990/rx, clientid: 806f0372-07d2-42a1-9bbf-ef2e084091d4 and acc: 2
 
INFO[0262] Files acl check passed.                      
DEBU[0262] user loraappserver acl authenticated with backend Files
 
DEBU[0262] setting acl cache (granted = true) for loraappserver
 
DEBU[0262] Acl is %!s(bool=true) for user loraappserver
 
DEBU[0262] checking acl cache for det
                  
DEBU[0262] Superuser check with backend Files
          
DEBU[0262] Superuser check with backend Postgres
       
DEBU[0262] Acl check with backend Files
                
INFO[0262] Files acl check with user det, topic: application/1/node/0004a30b001e0990/rx, clientid: mqtt_a862904f.92d99 and acc: 1
 
WARN[0262] Files acl check failed.                      
DEBU[0262] Acl check with backend Postgres
             
DEBU[0262] user det acl authenticated with backend Postgres
 
DEBU[0262] setting acl cache (granted = true) for det
  
DEBU[0262] Acl is %!s(bool=true) for user det
          
DEBU[0275] checking acl cache for loragw
               
DEBU[0275] found in cache: loragw
                      
DEBU[0275] checking acl cache for loraserver
           
DEBU[0275] found in cache: loraserver
                  
DEBU[0275] checking acl cache for loraappserver
        
DEBU[0275] Superuser check with backend Files
          
DEBU[0275] Superuser check with backend Postgres
       
DEBU[0275] Acl check with backend Files
                
INFO[0275] Files acl check with user loraappserver, topic: application/1/node/0004a30b001e4c78/rx, clientid: 806f0372-07d2-42a1-9bbf-ef2e084091d4 and acc: 2
 
INFO[0275] Files acl check passed.                      
DEBU[0275] user loraappserver acl authenticated with backend Files
 
DEBU[0275] setting acl cache (granted = true) for loraappserver
 
DEBU[0275] Acl is %!s(bool=true) for user loraappserver
 
DEBU[0275] checking acl cache for det
                  
DEBU[0275] Superuser check with backend Files
          
DEBU[0275] Superuser check with backend Postgres
       
DEBU[0275] Acl check with backend Files
                
INFO[0275] Files acl check with user det, topic: application/1/node/0004a30b001e4c78/rx, clientid: mqtt_a862904f.92d99 and acc: 1
 
WARN[0275] Files acl check failed.                      
DEBU[0275] Acl check with backend Postgres
             
DEBU[0275] user det acl authenticated with backend Postgres
 
DEBU[0275] setting acl cache (granted = true) for det
  
DEBU[0275] Acl is %!s(bool=true) for user det
          
DEBU[0278] checking acl cache for loragw
               
DEBU[0278] Superuser check with backend Files
          
DEBU[0278] Superuser check with backend Postgres
       
DEBU[0278] PG get superuser error: pq: syntax error at or near "ation_id"
 
DEBU[0278] Acl check with backend Files
                
INFO[0278] Files acl check with user loragw, topic: gateway/7276ff00080801db/stats, clientid: 908e786c-389a-416c-8cda-0dd5563c1762 and acc: 2
 
INFO[0278] Files acl check passed.                      
DEBU[0278] user loragw acl authenticated with backend Files
 
DEBU[0278] setting acl cache (granted = true) for loragw
 
DEBU[0278] Acl is %!s(bool=true) for user loragw
       
DEBU[0278] checking acl cache for loraserver
           
DEBU[0278] Superuser check with backend Files
          
DEBU[0278] Superuser check with backend Postgres
       
DEBU[0278] PG get superuser error: pq: syntax error at or near "ation_id"
 
DEBU[0278] Acl check with backend Files
                
INFO[0278] Files acl check with user loraserver, topic: gateway/7276ff00080801db/stats, clientid: 1237ace6-10aa-45e2-bc45-e4bf1dc8d179 and acc: 1
 
INFO[0278] Files acl check passed.                      
DEBU[0278] user loraserver acl authenticated with backend Files
 
DEBU[0278] setting acl cache (granted = true) for loraserver
 
DEBU[0278] Acl is %!s(bool=true) for user loraserver
   
DEBU[0285] checking acl cache for loragw
               
DEBU[0285] found in cache: loragw
                      
DEBU[0285] checking acl cache for loraserver
           
DEBU[0285] found in cache: loraserver
                  
^CINFO[0291] Cleaning up plugin

At other opportunity in logs it has been also registered

DEBU[1310] PG check acl error: pq: invalid message format

When I stop the manual mosquitto, and restart it again, it can run, but the connection with postgresql plugin is totally disable.

To work it again i have to reboot the machine.

Can you find the error?

I did not find the files to see where the must be, but i guess this is a big light to us continue to debuggin the project .

EDIT1: Please see the line error at 6.25 (it not registered but i have checked the system clock when it was at this time)

DEBU[0278] PG get superuser error: pq: syntax error at or near "ation_id"

EDIT2: We have an user “det” connecting over the broker using the postgresql plugin option

iegomez · February 9, 2018, 6:57pm

Nice! Thanks for taking such time and effort on debugging it.

I’m guessing that’s a typo at your pg_superquery, where you probably wrote “applic ation_id” instead of “application_id”, which would render a syntax error. That error would prevent superusers to get authorized, but wouldn’t crash the plugin, mosquitto or postgres.

That’s weird, it means that postgres is getting some strange input in the query string. I’ve seen reports about Postgres complaining on something like ‘\u0020’ in a string that doesn’t start with E, as it is a unicode symbol, or others regarding table encoding, etc. Could you share any context, i.e., more log lines? I think a “strange” username or clientid containing some offending symbol could cause this. Also, that’s probably a bug, as those should get correctly escaped, so I’ll take a look at it anyway.

You mean you see this kinf of message being printed when restarting mosquitto? If so, could you share the “error_here” part?

ping database error, will retry in 2s: error_here

Yeah, I can see it, and it seems to work fine as these lines of the log show:

DEBU[0275] user det acl authenticated with backend Postgres
DEBU[0275] setting acl cache (granted = true) for det
DEBU[0275] Acl is %!s(bool=true) for user det

iegomez · February 9, 2018, 7:03pm

By the way, @brocaar, I’m not sure if you are fine with us having this discussion here, as it’s gotten quite long and active and may be distracting people from other topics. If you’d like, we could move it to an issue at the plugin’s repo, unless you prefer it to be here if anyone attempts to use the plugin with loraserver.

brocaar · February 9, 2018, 10:30pm

No worries, the whole discussion is stil self-contained in a single topic so please continue the discussion!

iegomez · February 14, 2018, 1:50pm

@RogerioCassares, I’ve just pushed a little change to fix a bug with the cache redis db, which wasn’t getting set as the given option and defaulted to 0. Now it gets set correctly and defaults to db 3 if none is given.

RogerioCassares · February 14, 2018, 2:41pm

HI @iegomez!

At first thanks to the reply you have mentioned the mosquitto was out of dated! So I have updated mosquitto mannualy with .deb packages. I had to installed others packages also because websockets and another lib were needed.

I was waiting for the application running somedays without had broken as it was and I guess we got it. Let me push from git hub and substitute the plugin.so file and see what happens! I hope nothing happens

Thinkin about this error

It is all correct at .conf file, but i really suspect mosquitto was broken to read that line that something occuries and it read it cutting some words. So organization_id was identified by just ation_id

Well. Let me try and test it for some days. So I retrieve the result! It is cleaner to compile at one machine and just copý the plugin.so to a place on the server running mosquitto. Remembering they might be same Operating System

iegomez · February 14, 2018, 2:48pm

Nice to know it’s been working with the updated mosquitto!

I hope too!

Well, go on testing it and let me know if anything weird happens or something goes wrong. Also, if you find some use case that isn’t supported yet, I’d be happy to add it too. Thanks a lot for your effort and patience.

Cheers!

RogerioCassares · February 15, 2018, 6:11pm

MQTT authentication & authorization

The LoRa Server project does not handle MQTT authentication and authorization To make sure that not all data is exposed to all uses, it is advised to setup MQTT authentication & authorization.

For example, you could give every gateway its own login restricted to its own set of MQTT topics and you could give each user its own login, restricted to a set of applications.

Mosquitto

For Mosquitto there are multiple ways to setup authentication and authorization. This can be pre-configured in so called password and ACL (access control list) files and / or can be retrieved dynamically from the LoRa App Server user tables. In the latter case, LoRa App Server users are able to login with their own credentials and are limited to the applications to which they have access.

Static password and ACL file

These steps describe how to setup Mosquitto with a static password and ACL file. In case you would like to setup Mosquitto so that users and permissions are retrieved from LoRa App Server, go to the next section (Mosquitto Auth Plugin).

Passwords

Using the mosquitto_passwd command, it is possible to create a password file for authentication.

Example to create a password file and add add an username (use the -c only the first time as it will create a new file):

sudo mosquitto_passwd -c /etc/mosquitto/passwd <user_name>

ACLs

The access control list file will map usernames to a set of topics. Write this file to /etc/mosquitto/acls. An example is:

user loraserver_gw
topic write gateway/+/stats
topic write gateway/+/rx
topic read gateway/+/tx

user loraserver_ns
topic read gateway/+/stats
topic write gateway/+/tx
topic read gateway/+/rx

user loraserver_as
topic write application/+/node/+/rx
topic write application/+/node/+/join
topic write application/+/node/+/ack
topic write application/+/node/+/error
topic read application/+/node/+/tx

user bob
topic read application/123/node/+/+
topic write application/123/node/+/tx

The access parameter for each topic can be read, write or readwrite. Note that + is a wildcard character (e.g. all gateways, applications or nodes in the above example).

Mosquitto configuration

Then edit the /etc/mosquitto/mosquitto.conf config file so that it contains the following entries:

password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/acls
allow_anonymous false

Mosquitto Go Auth Plugin (users and permissions from LoRa App Server)

To setup Mosquitto so that it retrieves the users and permissions from the LoRa App Server database, you need to setup the mosquitto-go-auth plugin. This project provides authentication and authorization to Mosquitto using various backends. In our case we’re interested in the PostgreSQL and files backend.

Installing-the-requirements

Newest-mosquitto-version

Please make sure your mosquitto is currently updated at version 1.4.14. As @iegomez said at this topic ,it is very important the mosquitto new version is installed. Please reffers to this link to mosquitto packages. The instructions to compile from source in linux OS could be descripted here.

To verify the version of mosquitto:

mosquitto -h

If version is older and it was installed with sudo apt-get install commands you can remove all mosquitto to install new one:

sudo apt-get --purge remove mosquitto
sudo rm -r /etc/mosquitto

Some-apt-get-packages

Before the mosquitto-go-auth can be compiled, you need to install the following requirements (can be installed using sudo apt-get install ...):

git
mosquitto-dev
libmosquitto-dev
postgresql-server-dev-9.6
build-essential
libssl-dev

Golang-tool

In order to compile the project you must have golang eviorment installed.

From golang sources you can download the .deb file from 1.9.4 version executing command above in ~/Downloads respos, for example:

wget https://dl.google.com/go/go1.9.4.linux-amd64.tar.gz

Download the archive and set it up to /usr/local repos:

sudo tar -C /usr/local -xzf go1.9.4.linux-amd64.tar.gz

Edit the ~/.bashrc file to export the enviorment variables. Add the following lines to show where go env variables should be:

nano ~/.bashrc

export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go

Reload the variables on bash:

source ~/.bashrc

Test the instalation verifying go version on bash:

go -v

Also, you can see go env variables:

go env

Next, go to $GOPATH/src/github.com/iegomez repos and then clone the mosquitto-go-auth repository:

cd $GOPATH/src/github.com/iegomez 
sudo git clone https://github.com/iegomez/mosquitto-go-auth.git

Compiling-mosquitto-go-auth-plugin

Please go to the repos you have just cloned at step before, install requirements with make requirements and after that run make command to build the plugin:

cd $GOPATH/src/github.com/iegomez/mosquitto-go-auth
make requirements
make

Moving-the-plugin-only

From this moment, it is generated a file named go-auth.so. In fact, just this file you have to copy to somewhere in the project and indicate the path where it is in next configuration's steps.

In order to copy using ssh you can execute

scp /home/{USER}/go/src/github.com/iegomez/mosquitto-go-auth/go-auth.so {SSH_USER}@{HOST}:/home/{SSH_USER}/

Then inside the instance move to anywhere. I just decided to move to /usr/bin repos.

sudo mv /home/{USER}/go-auth.so /usr/bin

Configure mosquitto-go-auth

Create a directory and empty files for additional static passwords and ACLs:

sudo mkdir /etc/mosquitto/mosquitto-auth-plug
sudo touch /etc/mosquitto/mosquitto-auth-plug/passwords
sudo touch /etc/mosquitto/mosquitto-auth-plug/acls

Write the following content to /etc/mosquitto/conf.d/mosquitto-auth-plug.conf:

allow_anonymous false
auth_plugin /usr/bin/go-auth.so
auth_opt_backends files, postgres
auth_opt_log_level debug

#Cache
auth_opt_cache true
auth_opt_cache_reset false
auth_opt_cache_db 3

#Files
auth_opt_password_path /etc/mosquitto/mosquitto-auth-plug/passwords
auth_opt_acl_path /etc/mosquitto/mosquitto-auth-plug/acls

#POSTGRESQL
auth_opt_pg_host localhost
auth_opt_pg_port 5432
auth_opt_pg_dbname loraserver_as
auth_opt_pg_user loraserver_as
auth_opt_pg_password loraserver_as
auth_opt_pg_userquery select password_hash from "user" where username = $1 and is_active = true limit 1
auth_opt_pg_superquery select count(*) from "user" where username = $1 and is_admin = true
auth_opt_pg_aclquery select distinct 'application/' || a.id || '/#' from "user" u inner join organization_user ou on ou.user_id = u.id inner join organization o on o.id = ou.organization_id inner join application a on a.organization_id = o.id where u.username = $1 and $2 = $2

In case you want to log all the things that happens at mosquitto you can set log_type all at /etc/mosquitto/mosquitto.conf file. So this archive might be as it is shownd bellow:

# Place your local configuration in /etc/mosquitto/conf.d/#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

log_type all

listener 1883
listener 1884
protocol websockets

include_dir /etc/mosquitto/conf.d

You might want to change the following configuration, to match your LoRa App Server configuration:

auth_opt_pg_host: database hostname
auth_opt_pg_port: database port
auth_opt_pg_dbname: database name
auth_opt_pg_user: database username
auth_opt_pg_pass: database password

Static passwords

As LoRa Gateway Bridge, LoRa Server and LoRa App Server also make use of MQTT, you might want to configure static passwords for these services.

To generate a password readable by mosquitto-go-auth-plug, use the following command in compiling enviorment. This will prompt the password, returning a hashed version of it, example:

/home/{USER}/go/src/github.com/iegomez/mosquitto-go-auth/pw -p loraserver
    PBKDF2$sha512$100000$BEGKGOS34ug55ghRIZ1Tjw==$VW6bgmJSXgtYo4i4eucJGYAZwwVfRX3TZhQuTZEPSQLmFFkRvuHpW/C/fl9Fh7bW2n8z1u7Lb/D19jy9VH5fjQ==
/home/{USER}/go/src/github.com/iegomez/mosquitto-go-auth/pw -p loraappserver
    PBKDF2$sha512$100000$BEGKGOS34ug55ghRIZ1Tjw==$VW6bgmJSXgtYo4i4eucJGYAZwwVfRX3TZhQuTZEPSQLmFFkRvuHpW/C/fl9Fh7bW2n8z1u7Lb/D19jy9VH5fjQ==
/home/{USER}/go/src/github.com/iegomez/mosquitto-go-auth/pw -p loragw
    PBKDF2$sha512$100000$5tCi1hl2GjcPfBwXZnbH1w==$iFCpM9ojXc3LHWKw/N0pMftU1gDXRSQ9GTQb6OF/SSFv3y7yQO2bR3ky3KY4e0STTjLmOaE9DkyomMu6VjjO3g==

You can set others flags to make more interation or change to sha256. The default values for no flags are -a flag with sha512 algorithm and -i flag to 100000 interations. More details, please see pw-gen.go at ~/go/src/github.com/iegomez/mosquitto-go-auth/pw-gen.

You now need to write this output to the /etc/mosquitto/mosquitto-auth-plug/passwords file, where each line is in the format USERNAME:PASSWORDHASH. In the end your passwords file should look like this:

loraserver:PBKDF2$sha512$100000$cp9dNSsDSp1gI8rYysvTeg==$H2MLbpPHymTEcST5HqJ4BCM/P3iB57M7V9dOLcd0ZVBrEiDVac1OCzxlV+NDZAxDEmSbYm/q9ksN1GoWwTzrnw==
loraappserver:PBKDF2$sha512$100000$w2uA3T+hx2rwJ0TR/PatHA==$YuUBZjTrMCkzS7zKLHNXp0pzTVaFh/sz6VkOj8dns/IGJtVYdUeZD+VEpCjLwd3PZQ4BvANgge9b7+hXl7zvZQ==
loragw:PBKDF2$sha512$100000$7gYqBX59mLopTIH2hh4RWA==$7Ln/EP00RFfzJAqwWxllDs8w7MEpgVqei/l2fhJz0bXBCOiI3V2KY0p7rkgI2Gl1tIMGU6OODwhOiAVWaEdt3A==

Static ACLs

For the static passwords created in the previous step, you probably want to limit these logins to a certain set of topics. For this you can add ACL rules to limit the set of topics per username in the file /etc/mosquitto/mosquitto-auth-plug/acls. An example:

user loraserver_gw
topic write gateway/+/stats
topic write gateway/+/rx
topic read gateway/+/tx

user loraserver_ns
topic read gateway/+/stats
topic write gateway/+/tx
topic read gateway/+/rx

user loraserver_as
topic write application/+/node/+/rx
topic write application/+/node/+/join
topic write application/+/node/+/ack
topic write application/+/node/+/error
topic read application/+/node/+/tx

Debugging-the-plugin

It is a very interesting tools always debugging the tools to validate the first steps of a project. With plugin this is not different.

Please note that mosquitto have to be mannually stopped and all process refering to the port is running mosquitto must be killed. Instead of that besides the mosquitto not start correctly, it wont prompt the correct info to debug running tool.

Of course the debug option must be in some file. This is written at /etc/mosquitto/conf.d/mosquitto-auth-plug.conf with line auth_opt_log_level debug. Please refer to Read.md file at documentation of mosquitto-go-auth for more informations about the possibilities using this plugin.

So, once mosquitto has been initiated on boot, stop mosquitto. Then kill all the process at port that mosquitto is running. As default, the port is 1883. Verify if there is something at the port yet. Nothing must be showned. Then run mosquitto mannualy. The commands cited here are in sequence bellow:

sudo systemctl stop mosquitto.service
sudo fuser -k 1883/tcp
sudo lsof -i :1883
sudo mosquitto -c /etc/mosquitto/mosquitto.conf

Every users, pub and sub events and topics received over MQTT broker will be registered at this logs. To stop this logs, please stop this service by pressing Ctrl^C. So close all conections on port that is running mosquitto and start it by rebooting the machine, or just start as it would be initiating on boot. This steps are showned bellow:

Ctrl^C (at the terminal logs are running)
sudo fuser -k 1883/tcp
sudo lsof -i :1883
sudo systemctl start mosquitto.service

A long therm topic at LoRa forum was discussed in order to validate this plugin. Please note that is really simple to use even compling it! The steps mentioned here were the ones necessary to reach the user/password from LoRa-App-Server to connect over the server's MQTT broker.

RogerioCassares · February 15, 2018, 6:20pm

Hi @brocaar!

Thanks a lot to share the space with us in order to reach at the final of this topic!

I dont know if you already tested the @iegomez plugin, but for me it is really working!

Excuse me that I got your source code in html and modify to this new use of plugin.

I had edited a documentation resumming all the topic, based in your doc with MQTT auth & auth. So what i have change is the part of plugin. Maybe you could check this one. I have detailed as one I think is easier to go!

I hope help some people with that documentation avoiding not hit the head to the wall as i did and take some time to absorve and understand it at all.

@iegomez, I guess if could verify if something is missing and even make some wiki or let me write something in order to your plugins meet with the LoRa project just with the necessary in that case.

As you guys help us, i would be glad to share your projects’ experience with you.

Thanks again you guys.

Any problem I post here some doubt.

God blesses you at all!

iegomez · February 15, 2018, 9:22pm

Hi, @RogerioCassares. Thanks for the summary! I’ll take a better look at it tomorrow so it may be included at the repo as an example of using the plugin for loraserver.

iegomez · February 16, 2018, 4:09pm

I made some slight modifications, but it’s almost the same as your original post, so I think it should fully reflect your experience. I already added it to the repo here, and linked to it at the README. Thanks again for the report!

Cheers!

RogerioCassares · February 20, 2018, 3:35pm

Hello @iegomez After some time the plugin running, it happened again

The only reason is that i guess is because iI have reproduce the error turning back system time the same number of days that it works. IT is how there was a flag or something like that. I really dont know.

On Saturday morning, the error happened for first time after compiling the new plugin you just have pulled. The error happend for about the 6.25 am again.

So Yesteday I had a version before and turned it on . Today at 6.25 am again the error happend.

Since the mosquitto was not running manually, I took the logs from /var/log/mosquitto/mosquitto.log

/var/log/mosquitto$ sudo cat mosquitto.log

1519118727: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519119159: New connection from 54.191.90.143 on port 1883.
1519119160: Socket error on client <unknown>, disconnecting.
1519119161: New connection from 54.191.90.143 on port 1883.
1519119163: New connection from 54.191.90.143 on port 1883.
1519119163: Socket error on client <unknown>, disconnecting.
1519119167: Socket error on client <unknown>, disconnecting.
1519119167: New connection from 54.191.90.143 on port 1883.
1519119170: Socket error on client <unknown>, disconnecting.
1519120528: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519122329: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519124130: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519125931: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519127732: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519127946: New connection from 54.191.90.143 on port 1883.
1519127947: Socket error on client <unknown>, disconnecting.
1519127947: New connection from 54.191.90.143 on port 1883.
1519127949: Socket error on client <unknown>, disconnecting.
1519127949: New connection from 54.191.90.143 on port 1883.
1519127954: Socket error on client <unknown>, disconnecting.
1519127957: New connection from 54.191.90.143 on port 1883.
1519127960: Socket error on client <unknown>, disconnecting.
1519128611: New connection from 54.191.90.143 on port 1883.
1519128612: Socket error on client <unknown>, disconnecting.
1519128613: New connection from 54.191.90.143 on port 1883.
1519128615: Socket error on client <unknown>, disconnecting.
1519128615: New connection from 54.191.90.143 on port 1883.
1519128619: Socket error on client <unknown>, disconnecting.
1519128619: New connection from 54.191.90.143 on port 1883.
1519128622: Socket error on client <unknown>, disconnecting.
1519129257: New connection from 54.191.90.143 on port 1883.
1519129258: Socket error on client <unknown>, disconnecting.
1519129258: New connection from 54.191.90.143 on port 1883.
1519129260: Socket error on client <unknown>, disconnecting.
1519129260: New connection from 54.191.90.143 on port 1883.
1519129264: Socket error on client <unknown>, disconnecting.
1519129264: New connection from 54.191.90.143 on port 1883.
1519129267: Socket error on client <unknown>, disconnecting.
1519129533: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519129954: New connection from 54.191.90.143 on port 1883.
1519129955: Socket error on client <unknown>, disconnecting.
1519129956: New connection from 54.191.90.143 on port 1883.
1519129958: Socket error on client <unknown>, disconnecting.
1519129958: New connection from 54.191.90.143 on port 1883.
1519129962: Socket error on client <unknown>, disconnecting.
1519129962: New connection from 54.191.90.143 on port 1883.
1519129965: Socket error on client <unknown>, disconnecting.
1519130637: New connection from 54.191.90.143 on port 1883.
1519130638: Socket error on client <unknown>, disconnecting.
1519130638: New connection from 54.191.90.143 on port 1883.
1519130640: Socket error on client <unknown>, disconnecting.
1519130640: New connection from 54.191.90.143 on port 1883.
1519130644: Socket error on client <unknown>, disconnecting.
1519130644: New connection from 54.191.90.143 on port 1883.
1519130647: Socket error on client <unknown>, disconnecting.
1519131317: New connection from 54.191.90.143 on port 1883.
1519131318: Socket error on client <unknown>, disconnecting.
1519131320: New connection from 54.191.90.143 on port 1883.
1519131324: Socket error on client <unknown>, disconnecting.
1519131324: New connection from 54.191.90.143 on port 1883.
1519131327: Socket error on client <unknown>, disconnecting.
1519131334: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519131958: New connection from 54.191.90.143 on port 1883.
1519131959: Socket error on client <unknown>, disconnecting.
1519131960: New connection from 54.191.90.143 on port 1883.
1519131962: Socket error on client <unknown>, disconnecting.
1519131962: New connection from 54.191.90.143 on port 1883.
1519131966: Socket error on client <unknown>, disconnecting.
1519131966: New connection from 54.191.90.143 on port 1883.
1519131969: Socket error on client <unknown>, disconnecting.
1519132616: New connection from 54.191.90.143 on port 1883.
1519132617: Socket error on client <unknown>, disconnecting.
1519132620: New connection from 54.191.90.143 on port 1883.
1519132624: Socket error on client <unknown>, disconnecting.
1519132624: New connection from 54.191.90.143 on port 1883.
1519132627: Socket error on client <unknown>, disconnecting.
1519133135: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519133257: New connection from 54.191.90.143 on port 1883.
1519133258: Socket error on client <unknown>, disconnecting.
1519133258: New connection from 54.191.90.143 on port 1883.
1519133260: Socket error on client <unknown>, disconnecting.
1519133260: New connection from 54.191.90.143 on port 1883.
1519133264: Socket error on client <unknown>, disconnecting.
1519133264: New connection from 54.191.90.143 on port 1883.
1519133267: Socket error on client <unknown>, disconnecting.
1519133949: New connection from 54.191.90.143 on port 1883.
1519133950: Socket error on client <unknown>, disconnecting.
1519133950: New connection from 54.191.90.143 on port 1883.
1519133952: Socket error on client <unknown>, disconnecting.
1519133953: New connection from 54.191.90.143 on port 1883.
1519133957: Socket error on client <unknown>, disconnecting.
1519133957: New connection from 54.191.90.143 on port 1883.
1519133960: Socket error on client <unknown>, disconnecting.
1519134591: New connection from 54.191.90.143 on port 1883.
1519134592: Socket error on client <unknown>, disconnecting.
1519134592: New connection from 54.191.90.143 on port 1883.
1519134594: Socket error on client <unknown>, disconnecting.
1519134595: New connection from 54.191.90.143 on port 1883.
1519134599: Socket error on client <unknown>, disconnecting.
1519134599: New connection from 54.191.90.143 on port 1883.
1519134602: Socket error on client <unknown>, disconnecting.
1519134936: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519135282: New connection from 54.191.90.143 on port 1883.
1519135283: Socket error on client <unknown>, disconnecting.
1519135283: New connection from 54.191.90.143 on port 1883.
1519135285: Socket error on client <unknown>, disconnecting.
1519135286: New connection from 54.191.90.143 on port 1883.
1519135290: Socket error on client <unknown>, disconnecting.
1519135290: New connection from 54.191.90.143 on port 1883.
1519135293: Socket error on client <unknown>, disconnecting.
1519135924: New connection from 54.191.90.143 on port 1883.
1519135925: Socket error on client <unknown>, disconnecting.
1519135925: New connection from 54.191.90.143 on port 1883.
1519135927: Socket error on client <unknown>, disconnecting.
1519135931: New connection from 54.191.90.143 on port 1883.
1519135934: Socket error on client <unknown>, disconnecting.
1519136574: New connection from 54.191.90.143 on port 1883.
1519136575: Socket error on client <unknown>, disconnecting.
1519136575: New connection from 54.191.90.143 on port 1883.
1519136577: Socket error on client <unknown>, disconnecting.
1519136578: New connection from 54.191.90.143 on port 1883.
1519136582: Socket error on client <unknown>, disconnecting.
1519136582: New connection from 54.191.90.143 on port 1883.
1519136585: Socket error on client <unknown>, disconnecting.
1519136737: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519137213: New connection from 54.191.90.143 on port 1883.
1519137213: Socket error on client <unknown>, disconnecting.
1519138022: New connection from 54.191.90.143 on port 1883.
1519138023: Socket error on client <unknown>, disconnecting.
1519138024: New connection from 54.191.90.143 on port 1883.
1519138026: Socket error on client <unknown>, disconnecting.
1519138026: New connection from 54.191.90.143 on port 1883.
1519138030: Socket error on client <unknown>, disconnecting.
1519138030: New connection from 54.191.90.143 on port 1883.
1519138033: Socket error on client <unknown>, disconnecting.
1519138538: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1519138702: New connection from 54.191.90.143 on port 1883.
1519138703: Socket error on client <unknown>, disconnecting.
1519138703: New connection from 54.191.90.143 on port 1883.
1519138705: Socket error on client <unknown>, disconnecting.
1519138705: New connection from 54.191.90.143 on port 1883.
1519138709: Socket error on client <unknown>, disconnecting.
1519138710: New connection from 54.191.90.143 on port 1883.
1519138713: Socket error on client <unknown>, disconnecting.
1519139382: New connection from 54.191.90.143 on port 1883.
1519139383: Socket error on client <unknown>, disconnecting.
1519139383: New connection from 54.191.90.143 on port 1883.
1519139385: Socket error on client <unknown>, disconnecting.
1519139389: New connection from 54.191.90.143 on port 1883.
1519139392: Socket error on client <unknown>, disconnecting.

I have looked for the lines this logs started going wrong:

So searching for some more different infos I have found

1518699066: New connection from 179.212.90.112 on port 1883.
1518699066: New client connected from 179.212.90.112 as det (c1, k60, u'det').
1518699274: Socket error on client det, disconnecting.
1518700398: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518702199: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518704000: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518705801: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518707602: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518709403: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518711204: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518713005: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518714806: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518716607: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518718408: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518720209: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518722010: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518723811: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518725612: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518727413: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518729214: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518731015: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518732816: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518734617: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518736418: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518738219: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518740020: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518741821: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518743622: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518745423: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518747224: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518749025: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518750826: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518752627: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518753469: New connection from 54.191.90.143 on port 1883.
1518753558: Client <unknown> has exceeded timeout, disconnecting.
1518753558: Socket error on client <unknown>, disconnecting.
1518754428: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518756229: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518758030: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518759831: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518761632: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518763433: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518765234: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518767035: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518767370: New connection from 54.191.90.143 on port 1883.
1518767371: Socket error on client <unknown>, disconnecting.
1518767377: New connection from 54.191.90.143 on port 1883.
1518767381: Socket error on client <unknown>, disconnecting.
1518768836: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
1518770051: New connection from 54.191.90.143 on port 1883.
1518770053: New connection from 54.191.90.143 on port 1883.

The FILES it is all ok because the nodes are sending and receiveing data confirmed from he server and the databese is being updated by lora server also.

Please, what can it be ?

EDIT: i AM NOT ENTERING WHITH CLIENT-ID PARAMETER. IS THAT A PROBLEM?

iegomez · February 20, 2018, 4:22pm

Hi, Rogerio. I’m currently on a week off from work and will be back next tuesday, so I’ve only my phone with me and can’t do any debugging.

That said, could you aldo share the journal for mosquitto? Any error from the plugin should appear there.

Also, check permissions for /var/lib/mosquitto/ dir and the db file, maybe mosquitto isn’t able to write to it.

Regarding clientid, if I remember correctly, mosquitto allows one connection per clientid, so if none is set another connection without clientid would disconnect an existing one. But all connections appear to be from the same location, so it doesn’t seem to be the problem. You could try setting it anyway, maybe it could fail in conjunction with the permissions on the mosquitto db.

That’s what I can think of right now, hope it helps.

RogerioCassares · February 20, 2018, 5:09pm

Hi @iegomez!

Sorry for bothering you!

I will try set the client-id so and see what we got!

I have checked the status of all services we are running and I have found something very interesting!

I realize the postgresql had failed!

With sudo systemctl status postgresql.service i had the following lines:

● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2018-02-20 13:51:05 -03; 10min ago
  Process: 11409 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 11409 (code=exited, status=0/SUCCESS)

Feb 20 13:51:05 networkserver systemd[1]: Starting PostgreSQL RDBMS...
Feb 20 13:51:05 networkserver systemd[1]: Started PostgreSQL RDBMS.

Please have a look that it is active(failed) but weirdly it allows that loraserver and lora-app-server work with it with no problems!

At /var/log/postgresql/postgresql-9.5-main.log i had the following lines:

2018-02-20 13:51:12 -03 [11415-1] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:51:12 -03 [11415-2] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:51:12 -03 [11415-3] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:51:12 -03 [11415-4] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:12 -03 [11415-5] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:12 -03 [11415-6] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:12 -03 [11415-7] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:12 -03 [11415-8] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:14 -03 [11415-9] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:14 -03 [11415-10] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:14 -03 [11415-11] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:14 -03 [11415-12] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:15 -03 [11415-13] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:15 -03 [11415-14] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:15 -03 [11415-15] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:15 -03 [11415-16] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:15 -03 [11415-17] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:52:15 -03 [11415-18] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:15 -03 [11415-19] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:15 -03 [11415-20] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:15 -03 [11415-21] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:16 -03 [11415-22] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:16 -03 [11415-23] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:16 -03 [11415-24] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:16 -03 [11415-25] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:16 -03 [11415-26] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:52:24 -03 [11415-27] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:24 -03 [11415-28] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:24 -03 [11415-29] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:24 -03 [11415-30] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:42 -03 [11415-31] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:42 -03 [11415-32] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:30 -03 [11415-33] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:30 -03 [11415-34] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:30 -03 [11415-35] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:30 -03 [11415-36] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-37] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-38] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-39] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-40] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-41] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-42] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-43] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-44] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-45] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:53:39 -03 [11415-46] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-47] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:39 -03 [11415-48] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:39 -03 [11415-49] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:44 -03 [11415-50] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:44 -03 [11415-51] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:44 -03 [11415-52] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:44 -03 [11415-53] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:44 -03 [11415-54] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:53:47 -03 [11415-55] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:47 -03 [11415-56] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:47 -03 [11415-57] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:47 -03 [11415-58] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:47 -03 [11415-59] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:53:51 -03 [11415-60] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:51 -03 [11415-61] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:51 -03 [11415-62] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:53:51 -03 [11415-63] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:53:51 -03 [11415-64] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:54:01 -03 [11415-65] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:01 -03 [11415-66] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:54:01 -03 [11415-67] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:01 -03 [11415-68] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:54:01 -03 [11415-69] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-20 13:54:12 -03 [11415-70] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:12 -03 [11415-71] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:54:12 -03 [11415-72] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:12 -03 [11415-73] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:54:30 -03 [11415-74] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:30 -03 [11415-75] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:54:30 -03 [11415-76] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:54:30 -03 [11415-77] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id

EDIT: This occurs only this issue happens and restarting postgresql separetally with sudo systemctl restart postgresql.service it stays with active(failed). However sudo systemctl status sudo systemctl status postgresql@9.5-main.service is active and running! Please node that the log above is about the one is running and active

Note this error is that the one happened when beated 6.25 am .

At sudo journalctl -u mosquitto everything seems to be ok and at sudo systemctl status mosquitto , it is active (running) !

EDIT: Please note that the ACLquery is all correct and it have the erros just after some time! Exiting postgresql

iegomez · February 20, 2018, 7:33pm

That’s good to know. I’ll add some debugging messages and try to reproduce the error as soon as I get back.

RogerioCassares · February 23, 2018, 6:13pm

Hello @iegomez I will just replying here the things for when you could see and we are thinking about the issues

I had now running the mosqitto manually so it just stoped the logs:

DEBU[0150] Superuser check with backend Files
          
DEBU[0150] Superuser check with backend Postgres
       
DEBU[0150] Acl check with backend Files
                
INFO[0150] Files acl check with user loraappserver, topic: application/1/node/0004a30b001abe98/rx, clientid: 318fa11b-ddf7-4378-b02d-432e7a6f4ee8 and acc: 2
 
INFO[0150] Files acl check passed.                      
DEBU[0150] user loraappserver acl authenticated with backend Files
 
DEBU[0150] setting acl cache (granted = true) for loraappserver
 
DEBU[0150] Acl is %!s(bool=true) for user loraappserver
 
Segmentation fault (core dumped)

It has occurred only with this specific node. We are sending 20 bytes and it was workink but exactly when the message from node is send, it brokes plugin. But it sounds weird beacause it was working!

iegomez · February 24, 2018, 12:16am

Could you share the message that crashes the plugin in order to reproduce the problem?

RogerioCassares · February 26, 2018, 2:19pm

Hi @iegomez! Pretty sure! One message is that one above in latest reply. Strange is thar only with a node it chashes when node sent. But things still were worked at loraserver. Just the broker have crashed.

2018-02-20 13:51:12 -03 [11415-1] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:51:12 -03 [11415-2] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:51:12 -03 [11415-3] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:51:12 -03 [11415-4] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:12 -03 [11415-5] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:12 -03 [11415-6] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:12 -03 [11415-7] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1
2018-02-20 13:52:12 -03 [11415-8] loraserver_as@loraserver_as STATEMENT:  tion_user ou on ou.user_id = u.id inner join organization o on o.id
2018-02-20 13:52:14 -03 [11415-9] loraserver_as@loraserver_as ERROR:  syntax error at or near "tion_user" at character 1

I t happened after sometime only and it cant resolv by itself. Studyng this case, it seems postgresql plugin only have crashed, so the connection using theses crashed also! Files is ok! May can you add more logs to postgres plugin please? It is very instable yet. Some days runs, but others it chashes consecutively!

Can we make testes some tests focused on postgresql plugin?

EDIT: I dont know if it helps, but a thing in Node-RED is that it connects automatically. I do not need to deploy the project. But please look at this: Even if when plugin postgres auth crashes, it appears connected at Node-RED. However, when I deploy the project again it suddenly just change to disconnected. It is like we have an hysteresis at the structure. Please note that Node-Red stays connected automatically and when plugin started again, I do not need to deploy again at Node-RED.

EDIT2: In that interface the error is occurring maybe sometime the acl query is failing? Can we log it with more details?

EDIT3: I got some postgres logs at /var/log/postgresql/postgresql-9.5-main.log

So the error seems to the one before. Maybe it is parsing confusing at some point? It is formated to JSON ain some point or what can it be? Note that it is some issue you just have oppend herer: Postgres invalid message format · Issue #2 · iegomez/mosquitto-go-auth · GitHub

2018-02-26 09:10:52 -03 [1386-32853] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:10:52 -03 [1386-32854] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:10:52 -03 [1386-32855] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:22 -03 [1386-32856] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:22 -03 [1386-32857] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:22 -03 [1386-32858] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:22 -03 [1386-32859] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:45 -03 [1386-32860] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:45 -03 [1386-32861] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:45 -03 [1386-32862] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:45 -03 [1386-32863] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:52 -03 [1386-32864] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:52 -03 [1386-32865] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:11:52 -03 [1386-32866] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:11:52 -03 [1386-32867] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:12:37 -03 [1386-32868] loraserver_as@loraserver_as ERROR:  syntax error at or near "aRate" at character 1
2018-02-26 09:12:37 -03 [1386-32869] loraserver_as@loraserver_as STATEMENT:  aRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr"
2018-02-26 09:12:37 -03 [1386-32870] loraserver_as@loraserver_as ERROR:  syntax error at or near ":" at character 1
2018-02-26 09:12:37 -03 [1386-32871] loraserver_as@loraserver_as STATEMENT:  :"7276ff00080801db","time":"0001-01-01T00:00:00Z","timestamp":18802
2018-02-26 09:12:37 -03 [1386-32872] loraserver_as@loraserver_as ERROR:  syntax error at or near ""modulation"" at character 1
2018-02-26 09:12:37 -03 [1386-32873] loraserver_as@loraserver_as STATEMENT:  "modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":true,"
2018-02-26 09:12:37 -03 [1386-32874] loraserver_as@loraserver_as ERROR:  syntax error at or near ""modulation"" at character 1
2018-02-26 09:12:37 -03 [1386-32875] loraserver_as@loraserver_as STATEMENT:  "modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":true,"
2018-02-26 09:12:37 -03 [1386-32876] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:12:52 -03 [1386-32877] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:12:52 -03 [1386-32878] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:13:15 -03 [1386-32879] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:13:15 -03 [1386-32880] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:13:17 -03 [1386-32881] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xeb 0xc4 0x41
2018-02-26 09:13:17 -03 [1386-32882] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x98
2018-02-26 09:13:17 -03 [1386-32883] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:13:17 -03 [1386-32884] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:13:17 -03 [1386-32885] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:13:17 -03 [1386-32886] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:13:17 -03 [1386-32887] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:13:52 -03 [1386-32888] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:13:52 -03 [1386-32889] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:13:56 -03 [1386-32890] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xeb 0xc4 0x41
2018-02-26 09:13:56 -03 [1386-32891] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x98
2018-02-26 09:13:56 -03 [1386-32892] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:13:56 -03 [1386-32893] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr":true
2018-02-26 09:13:56 -03 [1386-32894] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:13:56 -03 [1386-32895] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"adr":true
2018-02-26 09:13:56 -03 [1386-32896] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:14:00 -03 [1386-32897] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:14:00 -03 [1386-32898] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:14:00 -03 [1386-32899] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:14:00 -03 [1386-32900] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:14:00 -03 [1386-32901] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:14:44 -03 [1386-32902] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xeb 0xc4 0x41
2018-02-26 09:14:44 -03 [1386-32903] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x98
2018-02-26 09:14:44 -03 [1386-32904] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:14:44 -03 [1386-32905] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:14:44 -03 [1386-32906] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:14:44 -03 [1386-32907] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:14:44 -03 [1386-32908] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:14:45 -03 [1386-32909] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:14:45 -03 [1386-32910] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:14:45 -03 [1386-32911] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:14:53 -03 [1386-32912] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:14:53 -03 [1386-32913] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:15:13 -03 [1386-32914] loraserver_as@loraserver_as ERROR:  syntax error at or near "{" at character 1
2018-02-26 09:15:13 -03 [1386-32915] loraserver_as@loraserver_as STATEMENT:  {"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":true,
2018-02-26 09:15:13 -03 [1386-32916] loraserver_as@loraserver_as ERROR:  syntax error at or near "{" at character 1
2018-02-26 09:15:13 -03 [1386-32917] loraserver_as@loraserver_as STATEMENT:  {"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":true,
2018-02-26 09:15:13 -03 [1386-32918] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:15:23 -03 [1386-32919] loraserver_as@loraserver_as ERROR:  syntax error at or near "{" at character 1
2018-02-26 09:15:23 -03 [1386-32920] loraserver_as@loraserver_as STATEMENT:  {"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":true,
2018-02-26 09:15:23 -03 [1386-32921] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:15:53 -03 [1386-32922] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:15:53 -03 [1386-32923] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:16:06 -03 [1386-32924] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xeb 0xc4 0x41
2018-02-26 09:16:06 -03 [1386-32925] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x98
2018-02-26 09:16:06 -03 [1386-32926] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:16:06 -03 [1386-32927] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:16:06 -03 [1386-32928] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:16:06 -03 [1386-32929] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:16:06 -03 [1386-32930] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:16:07 -03 [1386-32931] loraserver_as@loraserver_as ERROR:  syntax error at or near "":{"" at character 1
2018-02-26 09:16:07 -03 [1386-32932] loraserver_as@loraserver_as STATEMENT:  ":{"modulation":"LORA","bandwidth":125,"spreadFactor":10},"adr":tru
2018-02-26 09:16:07 -03 [1386-32933] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:16:45 -03 [1386-32934] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:16:45 -03 [1386-32935] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:16:53 -03 [1386-32936] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:16:57 -03 [1386-32937] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xeb 0xc4 0x41
2018-02-26 09:16:57 -03 [1386-32938] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x98
2018-02-26 09:16:58 -03 [1386-32939] loraserver_as@loraserver_as ERROR:  syntax error at or near "ataRate" at character 1
2018-02-26 09:16:58 -03 [1386-32940] loraserver_as@loraserver_as STATEMENT:  ataRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"ad
2018-02-26 09:16:58 -03 [1386-32941] loraserver_as@loraserver_as ERROR:  syntax error at or near "ataRate" at character 1
2018-02-26 09:16:58 -03 [1386-32942] loraserver_as@loraserver_as STATEMENT:  ataRate":{"modulation":"LORA","bandwidth":125,"spreadFactor":7},"ad
2018-02-26 09:16:58 -03 [1386-32943] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:17:23 -03 [1386-32944] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:17:23 -03 [1386-32945] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:17:45 -03 [1386-32946] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:17:45 -03 [1386-32947] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:17:53 -03 [1386-32948] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:18:23 -03 [1386-32949] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:18:23 -03 [1386-32950] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:19:07 -03 [1386-32951] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:19:07 -03 [1386-32952] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xb0
2018-02-26 09:19:23 -03 [1386-32953] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:19:23 -03 [1386-32954] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0xb0
2018-02-26 09:19:45 -03 [1386-32955] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:19:45 -03 [1386-32956] loraserver_as@loraserver_as ERROR:  invalid message format
[...]
2018-02-26 09:41:23 -03 [1386-33161] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:41:23 -03 [1386-33162] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:41:45 -03 [1386-33163] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:41:45 -03 [1386-33164] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:42:45 -03 [1386-33165] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:42:45 -03 [1386-33166] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:42:53 -03 [1386-33167] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:42:53 -03 [1386-33168] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:42:59 -03 [1386-33169] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:42:59 -03 [1386-33170] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:42:59 -03 [1386-33171] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:42:59 -03 [1386-33172] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:42:59 -03 [1386-33173] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:43:01 -03 [1386-33174] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:43:01 -03 [1386-33175] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:43:01 -03 [1386-33176] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:43:15 -03 [1386-33177] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:43:15 -03 [1386-33178] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:43:23 -03 [1386-33179] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:43:23 -03 [1386-33180] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:43:53 -03 [1386-33181] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:43:53 -03 [1386-33182] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:44:17 -03 [1386-33183] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:44:17 -03 [1386-33184] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:44:17 -03 [1386-33185] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:44:17 -03 [1386-33186] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:44:17 -03 [1386-33187] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:44:28 -03 [1386-33188] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:44:28 -03 [1386-33189] loraserver_as@loraserver_as ERROR:  invalid message format
2018-02-26 09:44:28 -03 [1386-33190] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 09:44:45 -03 [1386-33191] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:44:45 -03 [1386-33192] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
2018-02-26 09:44:53 -03 [1386-33193] loraserver_as@loraserver_as ERROR:  invalid byte sequence for encoding "UTF8": 0x90
[...]
2018-02-26 11:25:10 -03 [1386-34063] loraserver_as@loraserver_as ERROR:  syntax error at or near "xPacketsReceivedOK" at character 1
2018-02-26 11:25:10 -03 [1386-34064] loraserver_as@loraserver_as STATEMENT:  xPacketsReceivedOK":0,"txPacketsReceived":0,"txPacketsEmitted":0,"customData":nul
2018-02-26 11:25:15 -03 [1386-34065] loraserver_as@loraserver_as ERROR:  insufficient data left in message
2018-02-26 11:25:15 -03 [1386-34066] loraserver_as@loraserver_as ERROR:  insufficient data left in message

EDIT 4: REFFERING TO THIS TOPIC.

RogerioCassares:

Hello @iegomez I will just replying here the things for when you could see and we are thinking about the issues

I had now running the mosqitto manually so it just stoped the logs:

DEBU[0150] Superuser check with backend Files

DEBU[0150] Superuser check with backend Postgres

DEBU[0150] Acl check with backend Files

INFO[0150] Files acl check with user loraappserver, topic: application/1/node/0004a30b001abe98/rx, clientid: 318fa11b-ddf7-4378-b02d-432e7a6f4ee8 and acc: 2

INFO[0150] Files acl check passed.
DEBU[0150] user loraappserver acl authenticated with backend Files

DEBU[0150] setting acl cache (granted = true) for loraappserver

DEBU[0150] Acl is %!s(bool=true) for user loraappserver

Segmentation fault (core dumped)

It has occurred only with this specific node. We are sending 20 bytes and it was workink but exactly when the message from node is send, it brokes plugin. But it sounds weird beacause it was working!

At this node it was working perfectly. But when it had a confirmation message back to the node the plugin brokes It was accepting the conf message, but now not anymore.

EDIT 5: At other node we have update to receive conf from the server and it had worked as well. Only that node crashes the plugin suddenly with messages before registered: Segmentation fault (core dumped)