Hi there, Chirpstack Forum
Me and my team are working on switching our infrastructure to using the Basic Stations protocol instead of a simple packet forwarder configuration. We deployed the chirpstack server on an azure container with auto-generated certificates for TLS, on these machines we can see the error displayed at the bottom of this post the console. even after raising the log level from the toml configuration file, no useful information is displayed to better understand this error.
Despite this, when it works, it does mostly fine, and our use-cases are fullfilled. The reason why i’m posting this issue is that our chirpstack server randomly stopped working two times, which of course is bad news: the first time, shortly after putting these changes into testing, and the second one after i think a week (today), while performing some tests on a device connected to the only gateway managed by chirpstack in test. On the cloud machine’s console there’s no error related to the “crash”, the hosted machines simply stop working and need a manual restart, even though normally these machines restart themselves in case of crashes. The ones in production, still using the previous configuration, work flawlessly without TLS as a Packet Forwarder, while having a lot more load. We don’t want to push these changes into production until we fully understand the correlation between this error and the crashes and eventually how to fix this.
Our theory right now is that the fault isn’t on Azure for the crash: we have a prod machine working fine with a packet forwarder protocol and a much higher load but rather we think it’s a software bug on the chirpstack-server/bridge.
We really could use the help figuring how to not have this error spammed in the machine’s log system
The IP address is not known to us.
If any of you need more informations on the configuration files feel free to ask!
Thank you.
Below the last two minutes of log from our bridge in cloud. as you can see, it stopped working without error and the machine’s status was still running, but the chirpstack interface on port 8080 was down and no messages were being sent or received.
time="2024-11-04T14:32:31.090459279Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 topic=eu868/gateway/ac1f09fffe092639/event/stats time="2024-11-04T14:32:31.285839875Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3919395646 time="2024-11-04T14:32:31.285974107Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3919395646 time="2024-11-04T14:32:31.972126334Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=579942068 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:31.972245058Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=579942068 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:32.281841952Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=579942068 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:32.281954122Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:32:32.47479254Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=899838093 time="2024-11-04T14:32:32.474924238Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=899838093 time="2024-11-04T14:32:33.168789942Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=193563172 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:33.168947928Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=193563172 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:39.268411992Z" level=info msg="backend/basicstation: join-request received" gateway_id=ac1f09fffe092639 uplink_id=1091023595 time="2024-11-04T14:32:39.268548839Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=1091023595 2024/11/04 14:32:41 http: TLS handshake error from 10.92.0.10:61497: EOF 2024/11/04 14:32:42 http: TLS handshake error from 10.92.0.9:56685: EOF time="2024-11-04T14:32:43.387659104Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2750086232 time="2024-11-04T14:32:43.387804607Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2750086232 time="2024-11-04T14:32:46.322612387Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=472631033 time="2024-11-04T14:32:46.322735919Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=472631033 time="2024-11-04T14:32:47.001226471Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=921519149 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:47.001394216Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=921519149 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:47.320206935Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=921519149 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:47.320347209Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:32:52.888794991Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3351597107 time="2024-11-04T14:32:52.888982298Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3351597107 time="2024-11-04T14:32:53.558075388Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=2672079704 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:53.559588081Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=2672079704 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:54.874934349Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=2672079704 gateway_id=ac1f09fffe092639 time="2024-11-04T14:32:54.875420663Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack 2024/11/04 14:32:56 http: TLS handshake error from 10.92.0.10:61669: EOF 2024/11/04 14:32:57 http: TLS handshake error from 10.92.0.9:56846: EOF time="2024-11-04T14:32:58.07516651Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2331011442 time="2024-11-04T14:32:58.07531556Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2331011442 time="2024-11-04T14:33:01.089811166Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 topic=eu868/gateway/ac1f09fffe092639/event/stats 2024/11/04 14:33:11 http: TLS handshake error from 10.92.0.10:61842: EOF 2024/11/04 14:33:12 http: TLS handshake error from 10.92.0.9:57019: EOF time="2024-11-04T14:33:16.32937702Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=4219592401 time="2024-11-04T14:33:16.329513386Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=4219592401 time="2024-11-04T14:33:17.036586631Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=237171275 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:17.036710093Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=237171275 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:17.329937417Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=237171275 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:17.330047795Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:33:17.474972942Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=4091832996 time="2024-11-04T14:33:17.475465446Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=4091832996 time="2024-11-04T14:33:18.154555784Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=1247919848 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:18.154762623Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=1247919848 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:19.458420391Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=1247919848 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:19.458618703Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:33:22.897714809Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2754097555 time="2024-11-04T14:33:22.898036229Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2754097555 time="2024-11-04T14:33:23.604890268Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=390961905 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:23.605062373Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=390961905 gateway_id=ac1f09fffe092639 2024/11/04 14:33:26 http: TLS handshake error from 10.92.0.10:62013: EOF time="2024-11-04T14:33:26.740029321Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2827740187 time="2024-11-04T14:33:26.74017783Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2827740187 2024/11/04 14:33:27 http: TLS handshake error from 10.92.0.9:57190: EOF time="2024-11-04T14:33:31.090270064Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 topic=eu868/gateway/ac1f09fffe092639/event/stats time="2024-11-04T14:33:31.319364288Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2682020597 time="2024-11-04T14:33:31.319629827Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2682020597 2024/11/04 14:33:41 http: TLS handshake error from 10.92.0.10:62190: EOF 2024/11/04 14:33:42 http: TLS handshake error from 10.92.0.9:57351: EOF time="2024-11-04T14:33:46.354080045Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=1530943227 time="2024-11-04T14:33:46.354219507Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=1530943227 time="2024-11-04T14:33:49.419571755Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3962999472 time="2024-11-04T14:33:49.419729491Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3962999472 time="2024-11-04T14:33:52.891905473Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=2595794312 time="2024-11-04T14:33:52.892058781Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=2595794312 time="2024-11-04T14:33:53.561638519Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=4182640958 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:53.561790495Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=4182640958 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:53.893661553Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=4182640958 gateway_id=ac1f09fffe092639 time="2024-11-04T14:33:53.89381043Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack 2024/11/04 14:33:56 http: TLS handshake error from 10.92.0.10:62379: EOF 2024/11/04 14:33:57 http: TLS handshake error from 10.92.0.9:57510: EOF time="2024-11-04T14:34:01.090551296Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 topic=eu868/gateway/ac1f09fffe092639/event/stats time="2024-11-04T14:34:04.093410943Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3031046898 time="2024-11-04T14:34:04.093578868Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3031046898 2024/11/04 14:34:11 http: TLS handshake error from 10.92.0.10:62557: EOF 2024/11/04 14:34:12 http: TLS handshake error from 10.92.0.9:57676: EOF time="2024-11-04T14:34:16.361294385Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=492823958 time="2024-11-04T14:34:16.361406606Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=492823958 time="2024-11-04T14:34:17.063819183Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=1017516906 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:17.064005727Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=1017516906 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:17.350530397Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=1017516906 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:17.350710235Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:34:17.488942361Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3878431389 time="2024-11-04T14:34:17.489075451Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3878431389 time="2024-11-04T14:34:18.172781844Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=3432684414 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:18.172906848Z" level=info msg="backend/basicstation: downlink-frame message sent to gateway" downlink_id=3432684414 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:19.478918928Z" level=info msg="backend/basicstation: downlink transmitted message received" downlink_id=3432684414 gateway_id=ac1f09fffe092639 time="2024-11-04T14:34:19.47912777Z" level=info msg="integration/mqtt: publishing event" event=ack qos=0 topic=eu868/gateway/ac1f09fffe092639/event/ack time="2024-11-04T14:34:22.92095637Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3104938248 time="2024-11-04T14:34:22.921090261Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3104938248 2024/11/04 14:34:26 http: TLS handshake error from 10.92.0.10:62737: EOF time="2024-11-04T14:34:26.762441151Z" level=info msg="backend/basicstation: uplink frame received" gateway_id=ac1f09fffe092639 uplink_id=3627873374 time="2024-11-04T14:34:26.762556899Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=eu868/gateway/ac1f09fffe092639/event/up uplink_id=3627873374 2024/11/04 14:34:27 http: TLS handshake error from 10.92.0.9:57855: EOF