Problems with Liveframes and Dashboard with new version

ChirpStack Application Server
#1

Hello, since the upgrade to the newest stable (3.16), the live frames in the gateways do not work any more. We always get 1 Frame, which says “Invalid Date” and then no more frames are shown.

immagine
immagine1562×115 7.44 KB 

redis
time="2021-07-30T10:51:05.639245578Z" level=error msg="handle received ping error: get ping lookup error: get ping lookup error: redis: nil"
time="2021-07-30T10:51:05.639330111Z" level=error msg="finished unary call with code Internal" ctx_id=95e267a9-16ba-4b78-975b-fcb8a5f92f6e error="rpc error: code = Internal desc = handle received ping error: get ping lookup error: get ping lookup error: redis: nil" grpc.code=Internal grpc.method=HandleProprietaryUplink grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T10:51:05Z" grpc.time_ms=0.96 peer.address="192.168.68.242:38064" span.kind=server system=grpc

time="2021-07-30T10:53:05.218553863Z" level=error msg="handle received ping error: get ping lookup error: get ping lookup error: redis: nil"
time="2021-07-30T10:53:05.218606142Z" level=error msg="finished unary call with code Internal" ctx_id=045e6bfc-5247-4a1f-9305-0d7ddef22f97 error="rpc error: code = Internal desc = handle received ping error: get ping lookup error: get ping lookup error: redis: nil" grpc.code=Internal grpc.method=HandleProprietaryUplink grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T10:53:05Z" grpc.time_ms=0.758 peer.address="192.168.68.242:38064" span.kind=server system=grpc
time="2021-07-30T10:53:06.516402996Z" level=error msg="handle received ping error: get ping lookup error: get ping lookup error: redis: nil"
time="2021-07-30T10:53:06.516451642Z" level=error msg="finished unary call with code Internal" ctx_id=fe5ca9c8-c2c4-4ee1-a495-23135ee3b5c5 error="rpc error: code = Internal desc = handle received ping error: get ping lookup error: get ping lookup error: redis: nil" grpc.code=Internal grpc.method=HandleProprietaryUplink grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T10:53:06Z" grpc.time_ms=0.765 peer.address="192.168.68.242:38064" span.kind=server system=grpc
time="2021-07-30T10:53:06.550962741Z" level=error msg="handle received ping error: get ping lookup error: get ping lookup error: redis: nil"
time="2021-07-30T10:53:06.551181039Z" level=error msg="finished unary call with code Internal" ctx_id=7e35e0cf-a8b6-48ad-bcc2-f75cb2ac2b11 error="rpc error: code = Internal desc = handle received ping error: get ping lookup error: get ping lookup error: redis: nil" grpc.code=Internal grpc.method=HandleProprietaryUplink grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T10:53:06Z" grpc.time_ms=0.774 peer.address="192.168.68.242:38064" span.kind=server system=grpc
time="2021-07-30T10:53:07.283968825Z" level=error msg="handle received ping error: get ping lookup error: get ping lookup error: redis: nil"
time="2021-07-30T10:53:07.284017757Z" level=error msg="finished unary call with code Internal" ctx_id=2d441ed2-8888-49e0-a865-8d6b04895e7c error="rpc error: code = Internal desc = handle received ping error: get ping lookup error: get ping lookup error: redis: nil" grpc.code=Internal grpc.method=HandleProprietaryUplink grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T10:53:07Z" grpc.time_ms=0.812 peer.address="192.168.68.242:38064" span.kind=server system=grpc

websocket
time="2021-07-30T11:01:41.166133945Z" level=info msg="metrics saved" aggregation="[MINUTE HOUR DAY MONTH]" ctx_id=4ed57625-266a-455b-b0f4-ec130ec4579a name="gw:fcc23dfffe0b6bbe"
time="2021-07-30T11:01:41.166187221Z" level=info msg="finished unary call with code OK" ctx_id=4ed57625-266a-455b-b0f4-ec130ec4579a grpc.code=OK grpc.method=HandleGatewayStats grpc.service=as.ApplicationServerService grpc.start_time="2021-07-30T11:01:41Z" grpc.time_ms=9.209 peer.address="192.168.68.242:38064" span.kind=server system=grpc
time="2021-07-30T11:01:44.11111111111Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF"

Also the dashboard shows all Gatways/devices as offline, even if i see them all as “a few seconds” ago in the gatway list.

#2

Which Redis version are you using? Have you also upgraded the ChirpStack Network Server to the latest version?

#3

Redis version is 5-Alpine. Application server is 3.16 Network server is 3.13. Should be the latest versions.

We are running everyhing on Kubernetes inside AWS.

Edit: My Bad, did not see the 3.14 Version of the network server. This resolves the part of the live frames.

But the dashboard still shows all devices as inactive, while all af them have “last seen” at max 1 hour ago.

Also could you help us with the “ping lookup error” in the logs above?