Stuck in Context Deadline Exceeded Hell

This seems to be the most common, and least understood error that chirpstack users come across. I have read everything I can on here about it, but I’m still really confused on how to troubleshoot it.

I battled this error previously and then one day it just vanished. I never really figured out the root cause, but I was able to make dozens of images of the working setup that have been performing well in the field with no Chirpstack related issues. All the time though I had a nagging feeling that my (and apparently the communities) lack of understanding this error was going to come back and bite me in the ass. Now it has.

I have a custom gateway built around a Raspberry Pi CM3+ board, a RAK2245 concentrator, and a Nimbelink cellular modem on a Verizon VPN. All of this works great. However, we are now expanding internationally, and I needed to find an alternative cellular provider who could handle everything globally. That all went well at first.

The problems started when I tried to add a new device. My gateway receives commands through the AWS MQTT server, and translates those into API requests that are made locally. This is my programs output when it works:

2020-11-01 20:49:47,470:main-INFO:Add new device command received. GUID: , Type: 1, Name: PI0201000006, Key: ******************************
2020-11-01 20:49:47,470:main-INFO:Checking if device exists: http://localhost:8080/api/devices/778284780bfb0446
2020-11-01 20:49:47,491:main-DEBUG:<Response [404]>
2020-11-01 20:49:47,491:main-DEBUG:{‘error’: ‘object does not exist’, ‘code’: 5, ‘message’: ‘object does not exist’, ‘details’: []}
2020-11-01 20:49:47,492:main-DEBUG:Device does not exist. Adding device.
First getting application and device profile IDs.
2020-11-01 20:49:47,492:main-DEBUG:Getting Application ID: http://localhost:8080/api/applications?limit=1000
2020-11-01 20:49:47,520:main-DEBUG:Application name: PI-Data-Server id: 1
2020-11-01 20:49:47,520:main-DEBUG:Getting Device Profile ID: http://localhost:8080/api/device-profiles?limit=1000
2020-11-01 20:49:47,555:main-DEBUG:Device profile name: Position Indicator id: bb464183-94f4-49e3-a257-919f14cadb91
2020-11-01 20:49:47,557:main-DEBUG:{
“device”:{“applicationID”: “1”, “devEUI”: “778284780bfb0446”, “deviceProfileID”: “bb464183-94f4-49e3-a257-919f14cadb91”, “name”: “PI0201000006”}
}
2020-11-01 20:49:47,563:main-INFO:Creating device http://localhost:8080/api/devices
2020-11-01 20:49:47,621:main-DEBUG:<Response [200]>
2020-11-01 20:49:47,622:main-DEBUG:Setting device keys http://localhost:8080/api/devices/778284780bfb0446/keys
2020-11-01 20:49:47,645:main-DEBUG:<Response [200]>
2020-11-01 20:49:47,646:main-DEBUG:Verifying Device Profile…http://localhost:8080/api/devices/778284780bfb0446
2020-11-01 20:49:47,673:main-DEBUG:<Response [200]>
2020-11-01 20:49:47,674:main-DEBUG:{‘device’: {‘devEUI’: ‘778284780bfb0446’, ‘name’: ‘PI0201000006’, ‘applicationID’: ‘1’, ‘description’: ‘’, ‘deviceProfileID’: ‘bb464183-94f4-49e3-a257-919f14cadb91’, ‘skipFCntCheck’: False, ‘referenceAltitude’: 0, ‘variables’: {}, ‘tags’: {}}, ‘lastSeenAt’: None, ‘deviceStatusBattery’: 256, ‘deviceStatusMargin’: 256, ‘location’: None}
2020-11-01 20:49:47,674:main-INFO:Device Created
2020-11-01 20:49:47,675:main-INFO:Publishing message: {“4”: "14,
,1,PI0201000006,1
***********************;1,success"}

When I reboot with a cell module that has the new SIM card in, the command to add a new device is now resulting in a Context Deadline Exceeded error. This is what it looks like now:

2020-11-01 21:13:08,688:main-INFO:Add new device command received. GUID: , Type: 1, Name: PI0201000007, Key: ******************************
2020-11-01 21:13:08,688:main-INFO:Checking if device exists: http://localhost:8080/api/devices/778284780bfb0447
2020-11-01 21:13:08,713:main-DEBUG:<Response [404]>
2020-11-01 21:13:08,714:main-DEBUG:{‘error’: ‘object does not exist’, ‘code’: 5, ‘message’: ‘object does not exist’, ‘details’: []}
2020-11-01 21:13:08,714:main-DEBUG:Device does not exist. Adding device.
First getting application and device profile IDs.
2020-11-01 21:13:08,715:main-DEBUG:Getting Application ID: http://localhost:8080/api/applications?limit=1000
2020-11-01 21:13:08,761:main-DEBUG:Application name: PI-Data-Server id: 1
2020-11-01 21:13:08,761:main-DEBUG:Getting Device Profile ID: http://localhost:8080/api/device-profiles?limit=1000
2020-11-01 21:13:08,803:main-DEBUG:Device profile name: Position Indicator id: bb464183-94f4-49e3-a257-919f14cadb91
2020-11-01 21:13:08,804:main-DEBUG:{
“device”:{“applicationID”: “1”, “devEUI”: “778284780bfb0447”, “deviceProfileID”: “bb464183-94f4-49e3-a257-919f14cadb91”, “name”: “PI0201000007”}
}
2020-11-01 21:13:08,804:main-INFO:Creating device http://localhost:8080/api/devices
2020-11-01 21:13:09,476:main-DEBUG:<Response [500]>
2020-11-01 21:13:09,477:main-DEBUG:500
2020-11-01 21:13:09,478:main-DEBUG:{‘error’: ‘context deadline exceeded’, ‘code’: 2, ‘message’: ‘context deadline exceeded’, ‘details’: []}
2020-11-01 21:13:09,483:main-ERROR:Failed to Create Device: {“4”: "14,
,1,PI0201000007,******************************;0,Unknown Failure:500"}

It appears as if the requests to the Application server are going through fine, but when an API command requires the Network server, something goes wrong.
Devices that were previously added all communicate just fine.

How can I further troubleshoot this issue? If I go back to the other module everything works fine, but that is not an option. I have a very clear way to recreate this bug, and I need to figure it out.

I do not want to access the API or web interface remotely, the API will be accessed locally. Can I just turn this off in settings?
If API commands that do not involve the Network server seem to be fine, how can I further narrow down the cause of this?
How does the Application server find the Network server to begin with? Is this stored in a DB somewhere that I can manually modify?
Is there a way to add devices without using the API that might not have this issue?

I’m willing to dig pretty deep to figure this out, but I don’t know where else to look at this point.

Usually once you do the initial setup you add a network-server to the application server.
They are stored in the database.

1 Like