Azure VM Debian 9 Runs out of memory

I have an Azure VM running Debian 9 with Network Server, application server, postgresql and redis installed. I used to have Application server 3.9.0 and Network Server 3.10.0 and it worked OK. i recently upgraded to Application server 3.15.0 and network server 3.13.0. Now about every 15 days my VM runs out of memory. Is it a memory leak or is it perhaps redis that is the problem? It is the smallest cheapest VM you can get on Azure.

Here is an image of the memory usage over the last month. I rebooted the VM to get it running again.

Best way to debug this further is by using the Prometheus metrics endpoints for each service, that gives you detailed information about the memory-usage of each service (+ a lot of other useful metrics).

1 Like

OK Thanks.

I will do this and comment here once I have some metrics to show.

1 Like

Ok. I added prometheus metrics and didn’t see anything strange. I then took a snapshot of the memory usage using htop and let it run for a couple of days and checked again. Both the application server and the network server memory usage stays pretty consistent. The culprit seems to be python azure agent python3 -u /usr/sbin/waagent -run-exthandlers. Any idea why this is happening and if there is a way to fix it?

Pyhton 3.5.3
waagent 2.2.34

That might be a question for Azure support, as this is not related to ChirpStack.

Can I safely upgrade to the latest version of waagent? Chirpstack will still work?

Again, I don’t know. waagent is not provided by ChirpStack, it is provided by Azure. See GitHub - Azure/WALinuxAgent: Microsoft Azure Linux Guest Agent for more info.