Dashboard Missing & Random Logouts - ChirpStack on Kubernetes

ChirpStack (v4)
1

Hello,

Could someone help me troubleshoot a few issues I’m encountering with ChirpStack (CS) deployed in a Kubernetes environment? Don’t understand what could be so different.

TL;DR

After deploying ChirpStack to Kubernetes and adding a Gateway/Device:

  • The web UI dashboard is blank
  • I’m intermittently auto-logged out
  • Browser console shows errors like:
    • unknown content
    • no authorization provided

Setup Overview

  • ChirpStack Deployment:

    • Kubernetes: using chirpstack/chirpstack (also tested chirpstack/chirpstack:4)
    • Docker (reference deployment): using docker-compose.yml with same image/version, commented out all services besides chirpstack, postgres, redis.

  • MQTT Broker: TBMQ with username/password authentication (sessions confirmed active)

  • Database:
    Steps: Fresh install → Password change → Add Gateway/Device

  • Configuration Files:

    • Kubernetes: configmaps for chirpstack.toml, and region files us915_0.toml, us915_1.toml
    • Docker: same TOML files; Even tried removing unused TOML files to rule that out

What Works

Docker Deployment

:white_check_mark: Fully functional:

  • Add/remove gateways and devices
  • Dashboard data displays correctly
  • UI is stable and responsive
  • Able to see gRPC stream (LoRaWAN frames / Events)

Kubernetes Deployment

:white_check_mark: Gateway appears online
:white_check_mark: Can add/remove gateways/devices
:white_check_mark: LB configured for HTTP2

:x: Dashboard is blank

  • Dashboard section is missing from the DOM (confirmed via browser inspection)

:x: Intermittent forced logouts

  • Errors like no authorization provided in the console
  • Sometimes logout happens instantly after login, other times after a delay

:x: LoRaWAN frames / Events

  • Unable to view gRPC stream (LoRaWAN frames / Events)

What I’ve Tried

  • Ensured TOML config parity between Docker and Kubernetes deployments, “identical”
  • Port-forwarded directly to the ChirpStack pod (bypassing LoadBalancer) to eliminate proxy/header issues
  • Inspected browser console — errors persist even with direct access

If anyone has seen similar issues or can suggest next steps for debugging (e.g., header forwarding, ingress/session handling in K8s, etc.), I’d really appreciate your input.

Thanks in advance!

Below partial logs to include only useful information to keep message short. Everything below is in a collapsed section.

CS Logs (Removed a lot of "Message received from gateway" messages) 
2025-06-23T22:06:16.373845Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=55.011µs
2025-06-23T22:06:21.709634Z  INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=4.094904ms
2025-06-23T22:06:21.711821Z  INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=6.483886ms
2025-06-23T22:06:21.713977Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=10.19197ms
2025-06-23T22:06:21.716025Z  INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=10.89678ms
2025-06-23T22:06:21.816810Z  INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=6.518647ms
2025-06-23T22:06:27.522435Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=46.281µs
2025-06-23T22:06:30.289801Z  INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=7.094704ms
2025-06-23T22:06:30.299914Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=17.381686ms
2025-06-23T22:06:30.301172Z  INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=15.613032ms
2025-06-23T22:06:30.913290Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=41.731µs
2025-06-23T22:06:31.384480Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.651µs
2025-06-23T22:06:40.857794Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::gateway: Gateway partially updated gateway_id=647fdafffe0117df
2025-06-23T22:06:40.858480Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=HOUR
2025-06-23T22:06:40.858632Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=DAY
2025-06-23T22:06:40.858998Z  INFO stats{gateway_id=647fdafffe0117df}: chirpstack::storage::metrics: Metrics saved name=gw:647fdafffe0117df aggregation=MONTH
2025-06-23T22:06:42.537497Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.461µs
2025-06-23T22:06:45.926794Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=42.582µs
2025-06-23T22:06:46.400169Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=49.721µs
2025-06-23T22:06:48.088440Z  INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=1.65038ms
...
2025-06-23T22:06:57.552947Z  INFO gRPC{uri=/}: chirpstack::api: Finished processing request status="200" latency=48.052µs
Console log from the browser (removed duplicate lines just to keep short(er)) 
:8080/#/tenants/d24d…19a1/applications:1 Uncaught (in promise) Error: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
ListDevices.tsx:268 {limit: 10, offset: 0, search: '', applicationId: '9cf1bb34-aea4-488d-a7df-df5ce41e4755', multicastGroupId: '', …}
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
extractOriginPath @ content-script.js:1
fingerprintPage @ content-script.js:1
observeCheckoutMutations @ content-script.js:1
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/applications/9cf1bb34-aea4-488d-a7df-df5ce41e4755/devices/70b3d57050014946:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/applications/9cf1bb34-aea4-488d-a7df-df5ce41e4755/devices/70b3d57050014946:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
:8080/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
Console log from the Kubernetes version I removed a lot just to keep it somewhat short 
background-redux-new.js:1 Uncaught (in promise) Error: No tab with id: 2026259967.
/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmabikagpnhjmgacbclihgmdje/content-script.js:1:5713
/#/tenants/d24dd5c4-614c-4935-9c05-54ee7d6d19a1/gateways/647fdafffe0117df:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
DeviceStore.ts:191 
            
            
           POST https://motmod/api.DeviceService/GetLinkMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.DeviceServiceClient.getLinkMetrics @ device_grpc_web_pb.js:1036
(anonymous) @ DeviceStore.ts:191
(anonymous) @ DeviceDashboard.tsx:74
(anonymous) @ DeviceDashboard.tsx:96
(anonymous) @ DeviceDashboard.tsx:101
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
DeviceStore.ts:180 
            
            
           POST https://motmod/api.DeviceService/GetMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.DeviceServiceClient.getMetrics @ device_grpc_web_pb.js:975
(anonymous) @ DeviceStore.ts:180
(anonymous) @ DeviceDashboard.tsx:53
(anonymous) @ DeviceDashboard.tsx:97
(anonymous) @ DeviceDashboard.tsx:101
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
content-script.js:1 cornhusk, shared-service, error: TypeError: Failed to construct 'URL': Invalid URLInvalid url: 
extractOriginPath @ content-script.js:1
fingerprintPage @ content-script.js:1
observeCheckoutMutations @ content-script.js:1
childList
TO @ react-dom.production.min.js:248
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:252
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:257
ol @ react-dom.production.min.js:251
OO @ react-dom.production.min.js:254
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'no authorization provided', stack: 'Error: no authorization provided\n    at new U (htt…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 16, metadata: {…}}
GatewayStore.ts:108 
            
            
           POST https://motmod/api.GatewayService/GetMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.GatewayServiceClient.getMetrics @ gateway_grpc_web_pb.js:485
(anonymous) @ GatewayStore.ts:108
(anonymous) @ GatewayDashboard.tsx:61
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
GatewayStore.ts:122 
            
            
           POST https://motmod/api.GatewayService/GetDutyCycleMetrics 504 (Gateway Timeout)
qr @ index.js:44
po @ index.js:65
(anonymous) @ index.js:59
El.ri.X @ index.js:59
n.api.GatewayServiceClient.getDutyCycleMetrics @ gateway_grpc_web_pb.js:546
(anonymous) @ GatewayStore.ts:122
(anonymous) @ GatewayDashboard.tsx:78
Ey @ react-dom.production.min.js:243
af @ react-dom.production.min.js:285
$O @ react-dom.production.min.js:272
cc @ react-dom.production.min.js:127
xH @ react-dom.production.min.js:282
Op @ react-dom.production.min.js:280
kO @ react-dom.production.min.js:269
I @ scheduler.production.min.js:13
z @ scheduler.production.min.js:14
helpers.ts:7 API error:  U {message: 'Unknown Content-type received.', stack: 'Error: Unknown Content-type received.\n    at new U…motmod/assets/index-CLIOoR6L.js:533:16235)', code: 2, metadata: {…}}
/#/login:1 Error handling response: TypeError: Cannot read properties of undefined (reading 'isCheckout')
    at chrome-extension://clmkdohmsdfhjmgacbclihgmdje/content-script.js:1:5713
/#/login:1 Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received