Dashboard (Active Devices, Active Gateways, DR) behind Loadbalancer

sil · September 29, 2022, 12:02pm

Hello,

I am using two Chirpstack Application Servers behind a loadbalancer. The Dashboard does only show this:

The Gateway Map is okey - i can see the Gateway. When i connect directly to one of the Application Servers, i can see the Data in the Dashboard like this:

the logs from the application server look like this on connect via LB

time="2022-09-29T11:50:30Z" level=info msg="finished unary call with code OK" ctx_id=1f3a9741-c8e6-4671-b66d-ae279e3af4a5 grpc.code=OK grpc.method=Profile grpc.service=api.InternalService grpc.start_time="2022-09-29T11:50:30Z" grpc.time_ms=4.958 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:50:30Z" level=info msg="finished unary call with code OK" ctx_id=1a7424d8-784b-41ee-9815-cab1808b861a grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:30Z" grpc.time_ms=2.467 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:30Z" level=info msg="finished unary call with code OK" ctx_id=82acb10a-d976-45ba-8edb-74fba1d246c7 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:30Z" grpc.time_ms=2.433 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:30Z" level=info msg="finished unary call with code OK" ctx_id=8b7d8d4f-ecf7-465b-8604-be76e4435eb2 grpc.code=OK grpc.method=Settings grpc.service=api.InternalService grpc.start_time="2022-09-29T11:50:30Z" grpc.time_ms=0.029 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=614cda28-938c-42c6-b472-b99170ec46d0 grpc.code=OK grpc.method=Settings grpc.service=api.InternalService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=0.03 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=3082071e-657b-47b5-9706-e98b9f7cbb75 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=2.624 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=0cea28b2-8d52-4407-93da-c45ac15acc7d grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=4.858 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=19305394-6c21-4162-9af0-9ecb79e7099b grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=2.36 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=34bf4b37-161f-4982-9b14-24fab58431d2 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=2.704 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=f43f495a-cd8a-4051-928e-063adf4d5e54 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=6.024 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=3b42ef93-1c57-4c46-83bb-a7bde6e9bf70 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=5.838 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=0d6b012b-f595-4f4d-9cfc-369e82bf963a grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=2.205 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=c1b1eed6-ad6d-4492-a2d5-f795644f6871 grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=4.095 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=856e59a6-05a5-4a34-aa04-d27aab7c8a63 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=6.72 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=e7b0f558-3669-420e-83cf-4236702d968b grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=7.84 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:50:31Z" level=info msg="finished unary call with code OK" ctx_id=9daf14db-7a2c-439e-9a34-4b56d68bbf6b grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:50:31Z" grpc.time_ms=13.791 peer.address="127.0.0.1:53146" span.kind=server system=grpc

and like this if i connect directly:

time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=d76c4154-43bd-47c0-8668-ad0098bf4b38 grpc.code=OK grpc.method=Profile grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.181 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=f5286145-0c5a-4331-ada4-75eb8788ecd7 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=2.99 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=5b1df01f-5047-439a-bffe-c90af17f7967 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=2.217 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=908b377b-32be-45ed-939d-4bc0cf4a23d9 grpc.code=OK grpc.method=Settings grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=0.029 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=0c46f1e9-27c1-42ce-bb5e-065f55caba68 grpc.code=OK grpc.method=Settings grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=0.042 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=aeb141c3-397f-4a35-b76e-97789c506f04 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=2.841 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=6c4fd6c0-3679-404f-aea1-2d3728cb5d1a grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.303 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=8f9699f4-f72c-4696-ae63-4550e0ef44ca grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=8.239 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=1c513eb7-6759-4062-b993-7268bc1b9483 grpc.code=OK grpc.method=GetDevicesSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=9.203 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=b03e3c42-2bdd-4a87-a557-09d74624cd39 grpc.code=OK grpc.method=GetGatewaysSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=13.582 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=781ccf83-f251-4f10-8d0d-3a1f69f8bc56 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=3.872 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=117e8949-f2ac-46f3-91d0-bf126f5b246b grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=8.312 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=bdd6df42-4632-4fe1-9456-9bc34b2ffca2 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=3.11 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=7f4bd802-bc2a-4ba8-b31c-effbb01136a7 grpc.code=OK grpc.method=GetGatewaysSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=14.952 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=cd55d088-ff77-44a8-b2fa-ce6e574d037a grpc.code=OK grpc.method=GetDevicesSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=19.179 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=1c51f0f0-ccb7-4582-b7e8-26bd0dd39abf grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=4.206 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=cdd9fed6-a264-40b1-8cb1-27ae35644363 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=7.962 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=cbc70726-5dce-404e-80fd-06987e3ceef4 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=6.001 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=e30716f3-dae5-41a8-b4a5-cd33b6303e52 grpc.code=OK grpc.method=GetDevicesSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.354 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=731b0174-30ef-42da-a0ef-a10a48c21782 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=7.809 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=6ab1e3ce-ef5b-4800-845b-bbc6c4ffc31f grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=2.913 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=0063c818-3f11-490a-b6ce-0908cb6d9da1 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.973 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=27e5ec44-5949-4cc6-b238-06a112d67e27 grpc.code=OK grpc.method=GetGatewaysSummary grpc.service=api.InternalService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=17.256 peer.address="127.0.0.1:53154" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=2e2792fe-1721-4651-87dd-0844bfd1804e grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=8.334 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=02704538-96be-4d9d-94a9-32e6dffbf954 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.283 peer.address="127.0.0.1:53124" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=49aa3a6f-4a16-4308-a8d6-7999fa983bef grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=4.731 peer.address="127.0.0.1:53146" span.kind=server system=grpc
time="2022-09-29T11:51:00Z" level=info msg="finished unary call with code OK" ctx_id=f28cad59-5b33-4abd-a480-8e562fbfae58 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time="2022-09-29T11:51:00Z" grpc.time_ms=5.021 peer.address="127.0.0.1:53124" span.kind=server system=grpc

Does anyone has an idea, what this could be?

thanks, sil

Jose_Miguel_Concha · September 29, 2022, 1:35pm

we have a similar problem using Elastic Beanstalk

brocaar · September 29, 2022, 2:04pm

Have you checked the responses in the browser for these requests?

sil · September 29, 2022, 2:26pm

i can see the followin errors in the dev tools of the browser (over the load balancer). can’t see this direct.

the file looks like this:

i have tested some more - and it seams to be a issue with google chrome. it works also over the loadbalancer with edge and firefox… strange.

brocaar · September 29, 2022, 3:45pm

Could it be a caching issue?

sil · October 3, 2022, 5:09am

perfect - delete the broswercache was the solution.