We had a problem which started very early yestreday morning when our App-Server lost contact with all of our gateways. Upon investigation I discovered that all but one of the accounts in the /etc/mosquitto/passwd had dissappeared. I added the accounts back in, gateways still were not being seen.
The lora-gateway-bridge logs were showing a continual stream of the normal udp packets going back and forward between gateways and the bridge - however no lorawan packets were coming through from my devices and the gateways were showing a status of not having been seen for 1 day.
I then scovered that if I stopped the mosquitto service and started mosquitto from the prompt $ that evrything worked fine, however if I started mosquitto as a service nothing would work. The problem with this was the process was running within the session.
To solve the probelm I had to completly remove mosquitto and all associated files and reinstall.
Hi. Did you check the logs for the failing service?
Just a thought, but considering that the passwords file changed, maybe someone tampered not only with it, but also with permissions, so running mosquitto from terminal with sudo would work, but running the service (usually as the user mosquitto) could fail.
In other note, do you have an scenario where you need to do mosquitto authentication/authorization not covered by regular mosquitto auth (i.e., plain passwords and acls files)?
Yes someone tampering is always a possibility but it was strange in that they left the loraserver_as mqtt account intact.
The mosquitto service itself would start with no obvious issues or errors. The link bwteeen the lora-gateway-bridge and Mosquitto just didnt appear to work. Authenication etc all appeared to be working fine.
No - I dont at the moment have any requirement which isnt covered by regular mosquitto auth. Thats why I was content with static account and acls.
I thought it might be useful for someone encountering a similar issue to stick this post up with some detail on how to resolve it and also see if anyone else appears with a similar issue.
Thanks, @brocaar. I just pushed a fix to deal with the same issue but regarding Redis (it was indeed looping for Postgres, Mysql and Sqlite3, but Redis just failed when not available).
@stephenb Yep, it seems kind of odd, that’s why I thought someone could’ve tampered with files and permissions, not in a malicious way but rather just in a strange/wrong way.
As for the auth question and @brocaar’s mention of the plugin, I was asking because though it passes tests and works just fine for my daily use, I’m lacking real not-lab use testing to see how well it works. So, if you’re willing to give it a try and give me some feedback, it’d be most appreciated.
Yes I will give it a try. We have a trial under way just at the moment and I dont want to cause any more disruption but once it is completed in the next week or so I will try it out.
