Dashboard Missing & Random Logouts - ChirpStack on Kubernetes

ChirpStack (v4)
1

Hello,

Could someone help me troubleshoot a few issues I’m encountering with ChirpStack (CS) deployed in a Kubernetes environment? Don’t understand what could be so different.

TL;DR

After deploying ChirpStack to Kubernetes and adding a Gateway/Device:

  • The web UI dashboard is blank
  • I’m intermittently auto-logged out
  • Browser console shows errors like:
    • unknown content
    • no authorization provided

Setup Overview

  • ChirpStack Deployment:

    • Kubernetes: using chirpstack/chirpstack (also tested chirpstack/chirpstack:4)
    • Docker (reference deployment): using docker-compose.yml with same image/version, commented out all services besides chirpstack, postgres, redis.

  • MQTT Broker: TBMQ with username/password authentication (sessions confirmed active)

  • Database:
    Steps: Fresh install → Password change → Add Gateway/Device

  • Configuration Files:

    • Kubernetes: configmaps for chirpstack.toml, and region files us915_0.toml, us915_1.toml
    • Docker: same TOML files; Even tried removing unused TOML files to rule that out

What Works

Docker Deployment

:white_check_mark: Fully functional:

  • Add/remove gateways and devices
  • Dashboard data displays correctly
  • UI is stable and responsive
  • Able to see gRPC stream (LoRaWAN frames / Events)

Kubernetes Deployment

:white_check_mark: Gateway appears online
:white_check_mark: Can add/remove gateways/devices
:white_check_mark: LB configured for HTTP2

:x: Dashboard is blank

  • Dashboard section is missing from the DOM (confirmed via browser inspection)

:x: Intermittent forced logouts

  • Errors like no authorization provided in the console
  • Sometimes logout happens instantly after login, other times after a delay

:x: LoRaWAN frames / Events

  • Unable to view gRPC stream (LoRaWAN frames / Events)

What I’ve Tried

  • Ensured TOML config parity between Docker and Kubernetes deployments, “identical”
  • Port-forwarded directly to the ChirpStack pod (bypassing LoadBalancer) to eliminate proxy/header issues
  • Inspected browser console — errors persist even with direct access

If anyone has seen similar issues or can suggest next steps for debugging (e.g., header forwarding, ingress/session handling in K8s, etc.), I’d really appreciate your input.

Thanks in advance!

Below partial logs to include only useful information to keep message short. Everything below is in a collapsed section.

CS Logs (Removed a lot of "Message received from gateway" messages) 
2025-06-23T22:06:16.373845Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=55.011µs
2025-06-23T22:06:21.709634Z  INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=4.094904ms
2025-06-23T22:06:21.711821Z  INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=6.483886ms
2025-06-23T22:06:21.713977Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=10.19197ms
2025-06-23T22:06:21.716025Z  INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=10.89678ms
2025-06-23T22:06:21.816810Z  INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=6.518647ms
2025-06-23T22:06:27.522435Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=46.281µs
2025-06-23T22:06:30.289801Z  INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=7.094704ms
2025-06-23T22:06:30.299914Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=17.381686ms
2025-06-23T22:06:30.301172Z  INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=15.613032ms
2025-06-23T22:06:30.913290Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=41.731µs
2025-06-23T22:06:31.384480Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.651µs
2025-06-23T22:06:40.857794Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::gateway: Gateway partially updated gateway_id=647fdafffe0117df
2025-06-23T22:06:40.858480Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=HOUR
2025-06-23T22:06:40.858632Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=DAY
2025-06-23T22:06:40.858998Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=MONTH
2025-06-23T22:06:42.537497Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.461µs
2025-06-23T22:06:45.926794Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=42.582µs
2025-06-23T22:06:46.400169Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=49.721µs
2025-06-23T22:06:48.088440Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=1.65038ms
...
2025-06-23T22:06:57.552947Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.052µs
Console log from the browser (removed duplicate lines just to keep short(er)) 
:8080/#/tenants/d24d…19a1/applications:1 Uncaught (in promise) Error: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
ListDevices.tsx:268 {limit: 10, offset: 0, search: '', applicationId: '9cf1bb34-aea4-488d-a7df-df5ce41e4755', multicastGroupId: '', …}
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
extractOriginPath @ content-script.js:1
fingerprintPage @ content-script.js:1
observeCheckoutMutations @ content-script.js:1
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/applications/9cf1bb34-aea4-488d-a7df-df5ce41e4755/devices/70b3d57050014946:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/applications/9cf1bb34-aea4-488d-a7df-df5ce41e4755/devices/70b3d57050014946:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
Console log from the Kubernetes version I removed a lot just to keep it somewhat short 
background-redux-new.js:1 Uncaught (in promise) Error: No tab with id: 2026259967.
/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
DeviceStore.ts:191 
            
            
           POST https://motmod/api.DeviceService/GetLinkMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.DeviceServiceClient.getLinkMetrics @ device_grpc_web_pb.js:1036
(anonymous) @ DeviceStore.ts:191
(anonymous) @ DeviceDashboard.tsx:74
(anonymous) @ DeviceDashboard.tsx:96
(anonymous) @ DeviceDashboard.tsx:101
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
DeviceStore.ts:180 
            
            
           POST https://motmod/api.DeviceService/GetMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.DeviceServiceClient.getMetrics @ device_grpc_web_pb.js:975
(anonymous) @ DeviceStore.ts:180
(anonymous) @ DeviceDashboard.tsx:53
(anonymous) @ DeviceDashboard.tsx:97
(anonymous) @ DeviceDashboard.tsx:101
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
extractOriginPath @ content-script.js:1
fingerprintPage @ content-script.js:1
observeCheckoutMutations @ content-script.js:1
childList
TO @ react-dom.production.min.js:248
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:254
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'no authorization provided', stack: 'Error: no authorization provided\n    at new U (htt…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 16, metadata: {…}}
GatewayStore.ts:108 
            
            
           POST https://motmod/api.GatewayService/GetMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.GatewayServiceClient.getMetrics @ gateway_grpc_web_pb.js:485
(anonymous) @ GatewayStore.ts:108
(anonymous) @ GatewayDashboard.tsx:61
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
GatewayStore.ts:122 
            
            
           POST https://motmod/api.GatewayService/GetDutyCycleMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.GatewayServiceClient.getDutyCycleMetrics @ gateway_grpc_web_pb.js:546
(anonymous) @ GatewayStore.ts:122
(anonymous) @ GatewayDashboard.tsx:78
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
/#/login:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmsdfhjmgacbclihgmdje/content-script.js:1:5713
/#/login:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
2

Hello, still trying to figure out what’s going on, wish someone else had also deployed this to Kubernetes :slight_smile:

Anyway, the update is that I can do some GRPC commands, but not all. Below are the commands I tried and the output, or “lack thereof”.
The image is from the browser console showing calls that work and ones that are always (pending).

image
image628×253 12.9 KB

gRPCurl TEst Commands

These command were done with kubectl port-forward command so I could just bypass the LB. Some work, some hang… sanitized.

UserService/List (Works)

grpcurl -plaintext \
  -proto api/proto/api/user.proto \
  -import-path api/proto/ \
  -H "Authorization: Bearer <TOKEN>." \
  -d '{"limit": 1}' \
  localhost:8885 api.UserService/List

{
  "totalCount": 3,
  "result": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "createdAt": "2024-08-23T00:37:05.196049Z",
      "updatedAt": "2024-08-23T00:37:05.196049Z",
      "email": "admin",
      "isAdmin": true,
      "isActive": true
    }
  ]
}

GatewayService/List (Works)

grpcurl -plaintext \
  -proto api/proto/api/gateway.proto \
  -import-path api/proto/ \
  -H "Authorization: Bearer <TOKEN>." \
  -d '{"limit": 1}' \
  localhost:8885 api.GatewayService/List

{
  "totalCount": 14,
  "result": [
    {
      "tenantId": "00000000-0000-0000-0000-000000000000",
      "gatewayId": "GWID",
      "name": "NAME",
      "description": "",
      "location": {
        "latitude": 0,
        "longitude": 0
      },
      "properties": {
        "iccid": "0000000000000000000",
        "mqtt_forwarder_version": "4.3.1",
        "region_common_name": "US915",
        "region_config_id": "us915_0"
      },
      "createdAt": "2025-04-09T02:33:46.939581Z",
      "updatedAt": "2025-04-09T02:33:46.939581Z",
      "lastSeenAt": "2025-04-09T18:02:44.067260Z",
      "state": "OFFLINE"
    }
  ]
}

GatewayService/GetDutyCycleMetrics (Does not return - hangs)

grpcurl -plaintext \
  -proto api/proto/api/gateway.proto \
  -import-path api/proto/ \
  -H "Authorization: Bearer <TOKEN>." \
  -d '{"gateway_id": "<GATEWAY_ID>","start": "<START>","end": "<END>"}' \
  localhost:8885 api.GatewayService/GetDutyCycleMetrics

GatewayService/GetMetrics (Does not return - hangs)

grpcurl -plaintext \
  -proto api/proto/api/gateway.proto \
  -import-path api/proto/ \
  -H "Authorization: Bearer <TOKEN>." \
  -d '{"gatewayId": "<GATEWAY_ID>","start": "<START>","end": "<END>"}' \
  localhost:8885 api.GatewayService/GetMetrics
3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.