I am running into sporadic UFW BLOCK errors in my syslog. This is resulting in data not getting to chirpstack. I have confirmed that the sensor is not the issue by confirming the FCNT is incrementing. This was also confirmed with the sensor vendor.
I have the UFW enabled with access to ports 8080, 1700, 8086 and 8087.
I have taken a look at the gateway router and checked the logs. I do not see anything there that would indicate it was resetting.
I am running chirpstackv4 on ubuntu 22.04 LTS.
Any ideas or support would be much welcomed. This is resulting in inconsistent data reporting.
Hi @jidggins ,
@Jerome73 for context the issue I am trying to resolve is random loss of data between the gateway and gateway server. This tends to happen at off hours. Although, this morning the gateway can not connect for several hours. I rebooted the gateway, router, and server. It seemed to fix it for a short period of time and now has failed to connect again.
I have confirmed it is not the end devices as the FCNT increases or resets with confirmed FCNT. So I do believe it is between the gateway server and gateway.
You can see the loss of date after 8/1 00:43:18 through 8/1 8/1 07:23:13. The FCNT increase during this time which confirms the end device is working. I did lose around the same time frame from a second device.
Below are the logs from slightly before through during the loss of data.
Aug 1 04:53:24 vultr kernel: [31132.150691] [UFW BLOCK] IN=enp1s0 OUT= MAC=56:00:04:83:eb:ed:fe:00:04:83:eb:ed:08:00 SRC=103.57.130.82 DST=45.77.159.171 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=4161 DF PROTO=TCP SPT=41831 DPT=8201 WINDOW=512 RES=0x00 SYN URGP=0#033 [2m2023-08-01T04:53:25.409292Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:53:25.411534Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:53:25.412280Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:53:25.412856Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:53:25.413347Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T04:53:55.421145Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:53:55.423764Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:53:55.424530Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:53:55.425071Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:53:55.425586Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T04:54:25.425219Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:54:25.428249Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:54:25.428757Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:54:25.429086Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:54:25.429654Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T04:54:55.455427Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:54:55.459541Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:54:55.460097Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:54:55.460576Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:54:55.461137Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T04:55:25.456654Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:55:25.458697Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:55:25.459379Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:55:25.459926Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:55:25.460372Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T04:55:55.469092Z#033[0m #033 [32m INFO#033[0m #033 [2mchirpstack::gateway::backend::mqtt#033[0m#033[2m:#033 [0m Message received from gateway #033 [3mregion_config_id#033[0m#033[2m=#033[0m"us915_0" #033 [3mtopic#033[0m#033[2m=#033[0m"us915_0/gateway/c0ee40ffff2a3a01/event/stats" #033 [3mqos#033[0m#033[2m=#033[0m0 #033 [3mjson#033[0m#033[2m=#033[0mfalse#033 [2m2023-08-01T04:55:55.471105Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::gateway#033[0m#033[2m:#033 [0m Gateway state updated #033 [3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033 [2m2023-08-01T04:55:55.471730Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mHOUR#033 [2m2023-08-01T04:55:55.472213Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mDAY#033 [2m2023-08-01T04:55:55.472667Z#033[0m #033 [32m INFO#033[0m #033 [1mstats#033[0m#033[1m{#033 [0m#033[3mgateway_id#033[0m#033[2m=#033[0mc0ee40ffff2a3a01#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::storage::metrics#033[0m#033[2m:#033 [0m Metrics saved #033 [3mname#033[0m#033[2m=#033[0mgw:c0ee40ffff2a3a01 #033 [3maggregation#033[0m#033[2m=#033[0mMONTH#033 [2m2023-08-01T05:04:12.264152Z#033[0m #033 [32m INFO#033[0m #033 [1mhttp#033[0m#033[1m{#033 [0m#033[3mmethod#033[0m#033[2m=#033[0m"GET" #033 [3muri#033[0m#033[2m=#033[0m/ #033 [3mversion#033[0m#033[2m=#033[0mHTTP/1.1#033[1m}#033 [0m#033[2m:#033 [0m #033 [2mchirpstack::api#033[0m#033[2m:#033 [0m Finished processing request #033 [3mstatus#033[0m#033[2m=#033[0m"200" #033 [3mlatency#033[0m#033[2m=#033[0m102.673µs
I am making the assumption that the UFW has something to do with the failing gateway. The failures are sporadic and mostly happen during the night time.
I don’t think UFW has something to do with that. Sorry to say something obvious, but are you sure that the gateway is not plugged to an automatically commuted plug? Or that the router is not stopping at night hours?
We have this issue regularly in shopping malls, they shut down the power during nights to spare energy…
@Jerome73 I am not losing power as this is my home modem router. I am fine if it is obvious as I am no expert, so I am glad to remove UFW as a potential issue.
The plug is a standard wall outlet. Both the router and gateway are plugged in.
I did disable wifi and switched to ethernet and the gateway seemed to kick back on. Still not sure this is the root cause.
system
October 30, 2023, 2:45pm
6
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.
