504 Gateway Time-out on LoRaWAN Frames (v4)

ChirpStack (v4)
1

I’ve set up ChirpStack v4 on AWS Elastic Beanstalk (load balanced environment), and when accessing the LoRaWAN frames tab on a Gateway the data never loads. I looked into the brower’s developer tools and I found that the request is failing with a 504 Gateway Time-out error.

My suspicion is that the request is actually never reaching the EC2 instance. My reasoning:

  • From the request headers I’m deducing that the request is being sent through gRPC
  • The load balancer is an Application Load Balancer
  • The Load Balancer Listener is using port 80 (HTTP)
  • The Load Balancer Target Group is using protocol version HTTP (which means HTTP/1.1)
  • HTTP/1.1 is not compatible with gRPC

Acting on that suspicion, I’ve tried:

  • Creating a Listener on port 443 (HTTPS)
  • Creating a Target Group using protocol version HTTP2 (HTTP/2, partially compatible with gRPC) or with protocol version gRPC
  • Setting healthcheck to / or to /health
  • Registering my EC2 instance on that Target Group

The problem is that the Target Group is marking my instance as unhealthy (failed healthcheck), so it’s never sending traffic to it (the result is a 502 error, consistent with my AWS experience of a Target Group with no healthy instances).

I would greatly appreciate if someone could tell me how to perform healthchecks with a Target Group configured for HTTP2 or gRPC.

Or alternatively, if I’m completely mistaken and how I could resolve this.

For reference, this worked in ChirpStack v3, and I suspect it was because the Application server only used HTTP (I could be wrong on this).

Things that I am considering:

  • Replacing the Application Load Balancer with a Network Load Balancer. The problem is that a Network Load Balancer won’t decrypt HTTPS traffic, so if I want to expose ChirpStack through HTTPS I need to do it some other way.
2

Update: I tried changing to a Network Load Balancer, and the request that was timing out is now succeeding. I’m still not seeing any LoRaWAN Frames though, it stays loading forever:

image
image1588×560 20.9 KB

Another suspicious thing I found is that nginx is reporting some errors:

2022/10/14 21:14:25 [error] 4178#4178: *1270 upstream timed out (110: Connection timed out) while reading upstream, client: 186.12.187.113, server: , request: "POST /api.InternalService/StreamGatewayFrames HTTP/1.1", upstream: "http://172.17.0.2:8080/api.InternalService/StreamGatewayFrames", host: "chirpstack-v4-4.eba-hctng6ep.us-east-1.elasticbeanstalk.com:8080", referrer: "http://chirpstack-v4-4.eba-hctng6ep.us-east-1.elasticbeanstalk.com:8080/"

I think the key point there is that the request seems to be using protocol HTTP/1.1, but I believe it is supposed to be done through gRPC (which is compatible with HTTP/2 but not with HTTP/1.1). However, I’m at a loss as to why it could be using HTTP/1.1

3

For the streaming gRPC calls, you need to enable gRPC in the load-balancer or reverse proxy. E.g. for NGINX: Introducing gRPC Support with NGINX 1.13.10 - NGINX.

Note that the web-interface uses gRPC-web, which works over HTTP/1.1.

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.