Systemctl application server error?

barryjump · September 29, 2020, 7:35pm

Hi, I’m having a strange issue when trying to start the application server as a service.

Specs are a vanilla Debian 9 VM on GCP with only the network server and application server installed per the Chirpstack/GCP quickstart.

Getting this error when running sudo systemctl status chirpstack-application-server.

instance-1:~$ sudo systemctl status chirpstack-application-server
● chirpstack-application-server.service - ChirpStack Application Server
   Loaded: loaded (/lib/systemd/system/chirpstack-application-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-09-29 19:30:38 UTC; 3s ago
     Docs: https://www.chirpstack.io/
  Process: 2389 ExecStart=/usr/bin/chirpstack-application-server (code=exited, status=1/FAILURE)
 Main PID: 2389 (code=exited, status=1/FAILURE)
Sep 29 19:30:37 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
Sep 29 19:30:37 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.
Sep 29 19:30:38 instance-1 systemd[1]: chirpstack-application-server.service: Service hold-off time over, scheduling restart.
Sep 29 19:30:38 instance-1 systemd[1]: Stopped ChirpStack Application Server.
Sep 29 19:30:38 instance-1 systemd[1]: chirpstack-application-server.service: Start request repeated too quickly.
Sep 29 19:30:38 instance-1 systemd[1]: Failed to start ChirpStack Application Server.
Sep 29 19:30:38 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
Sep 29 19:30:38 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.

Everything runs fine when I run it in the foreground with sudo chirpstack-application-server. Strangely the network server has no issue running with systemctl.

brocaar · September 30, 2020, 6:51am

systemctl indicated that it failed to start chirpstack-application-server. To find out why, you have to check the logs

barryjump · September 30, 2020, 10:38am

@brocaar Sorry I should have included this with the original post.

It appears to be a problem with systemd, but I’m not clear on what that problem is. Was hoping someone would have some clues to share. Curious about the failed state error towards the bottom and the “Start request repeated too quickly”.

Heres what the logs show:

    instance-1:~$ journalctl -u chirpstack-application-server -f -n 50

    -- Logs begin at Tue 2020-09-29 19:58:27 UTC. --
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2037]: time="2020-09-30T10:31:02Z" level=info msg="storage: PostgreSQL data migrations applied" count=0
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Main process exited, code=exited, status=1/FAILURE
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Service hold-off time over, scheduling restart.
    Sep 30 10:31:02 instance-1 systemd[1]: Stopped ChirpStack Application Server.
    Sep 30 10:31:02 instance-1 systemd[1]: Started ChirpStack Application Server.
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="starting ChirpStack Application Server" docs="https://www.chirpstack.io/" version=3.12.2
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up storage package"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: setup metrics"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up Redis client"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: connecting to PostgreSQL database"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: applying PostgreSQL data migrations"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2044]: time="2020-09-30T10:31:02Z" level=info msg="storage: PostgreSQL data migrations applied" count=0
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Main process exited, code=exited, status=1/FAILURE
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Service hold-off time over, scheduling restart.
    Sep 30 10:31:02 instance-1 systemd[1]: Stopped ChirpStack Application Server.
    Sep 30 10:31:02 instance-1 systemd[1]: Started ChirpStack Application Server.
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="starting ChirpStack Application Server" docs="https://www.chirpstack.io/" version=3.12.2
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up storage package"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: setup metrics"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up Redis client"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: connecting to PostgreSQL database"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: applying PostgreSQL data migrations"
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="storage: PostgreSQL data migrations applied" count=0
    Sep 30 10:31:02 instance-1 chirpstack-application-server[2049]: time="2020-09-30T10:31:02Z" level=info msg="integration: configuring global integrations"
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Main process exited, code=exited, status=1/FAILURE
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.
    Sep 30 10:31:02 instance-1 systemd[1]: chirpstack-application-server.service: Service hold-off time over, scheduling restart.
    Sep 30 10:31:02 instance-1 systemd[1]: Stopped ChirpStack Application Server.
Sep 30 10:31:02 instance-1 systemd[1]: Started ChirpStack Application Server.
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="starting ChirpStack Application Server" d
ocs="https://www.chirpstack.io/" version=3.12.2
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up storage package"
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="storage: setup metrics"
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="storage: setting up Redis client"
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="storage: connecting to PostgreSQL database"
Sep 30 10:31:02 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:02Z" level=info msg="storage: applying PostgreSQL data migrations"
Sep 30 10:31:03 instance-1 chirpstack-application-server[2054]: time="2020-09-30T10:31:03Z" level=info msg="storage: PostgreSQL data migrations applied" count=0
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered failed state.
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Service hold-off time over, scheduling restart.
Sep 30 10:31:03 instance-1 systemd[1]: Stopped ChirpStack Application Server.
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Start request repeated too quickly.
Sep 30 10:31:03 instance-1 systemd[1]: Failed to start ChirpStack Application Server.
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Unit entered faile dstate.
Sep 30 10:31:03 instance-1 systemd[1]: chirpstack-application-server.service: Failed with result 'exit-code'.

barryjump · September 30, 2020, 11:04am

@brocaar I think I’ve found the problem. I was encountering another error yesterday:

FATA[0000] setup integration error: new integration error: credentials from json error: unexpected end of JSON input

Which lead me to this thread which suggests that creating this file is missing from the GCP quickstart docs:

So I created the json credential file and loaded it into the etc/chripstack-application-server directory and pointed to it in the toml file.

But I’m not clear if that is the correct spot when running systemctl. I think it’s looking in a different location?

The reason I can run the app server with “sudo chirpstack-application-server” from my root directory is because for testing I also have that file in my root directory which it finds when starting.

Any clarity on how it should be configured? Thanks!

barryjump · September 30, 2020, 11:15am

For clarity here’s my config within etc/chripstack-application-server/chirpstack-application-server.toml

I’m certain part (or all) of the problem is the path to credentials_file=“lora-gateway.json” which is in the same directory as the toml, but it appears systemctl is not finding it.

[postgresql]
dsn="postgres://chirpstack_as:[redacted]@[redacted]/chirpstack_as?sslmode=disable"
[redis]
url="redis://[redacted]:6379"

[application_server]

    [application_server.integration]
    enabled=["gcp_pub_sub"]

    [application_server.integration.gcp_pub_sub]
    # Path to the IAM service-account credentials file.
    #
    # Note: this service-account must have the following Pub/Sub roles:
    #  * Pub/Sub Editor

    credentials_file="lora-gateway.json"
    project_id="lora-gateway"
    topic_name="chirpstack-application-server"

    [application_server.external_api]
    bind="0.0.0.0:8080"
    jwt_secret="[redacted]"

barryjump · October 2, 2020, 10:35am

Bumping this. Eager to get this figured out. Any help greatly appreciated! thanks

Muhammad_Fahad_Baig · October 3, 2020, 6:27am

You should add full path to the JSON file and not just the filename.

barryjump · October 3, 2020, 10:42am

Thanks @Muhammad_Fahad_Baig that did the trick!

In case anyone is curious, a nice trick for getting the full system file path you can run:

readlink -f your-credential-file.json