TLS Certificates Failed to Fetch

Setup and configuration
1

I tried to add the tls certificates and keys to the network server in the web interface however it is showing failed to fetch when i update the network server
Then i tried adding cert and key for network and application servers separately the same error occurred for network server cert and key but when i tried with only application server cert and key the network server was updated in the web interface

Could this be problem with the toml of network server??? if not what could be the reason??

1 Like
2

I am also facing the same issue. @brocaar any idea why this is happening ?

3

The certificates you configure through the web-interface must match the certificates which are used to serve the API interfaces. Thus first configure the interfaces through the TOML files (NS and AS), restart the services and then update the certificate config through the web-interface.

4

@brocaar i added the certificates to the toml and restarted both NS and AS , i am attaching the journals and webinterface snaps

app journal
app journal1261×356 26.9 KB
ns journal
ns journal1259×292 22.9 KB
wi
wi1010×462 17 KB

5

WI cert
WI cert987×478 16.7 KB

as you can see once i add certificates in the toml in webinterface the @ in the network server is not showing the frequency band and when i add the certificates i get context deadline code 2 error.

6

There are two steps:

  1. Configuring the server certificates
  2. Configuring the client certificates

Step 1 is done in the TOML config files. Step 2. must be done through the web-interface (TLS certificates) tab. If both are setup correctly, then the AS should be able to connect to the NS again (and the NS to the AS).

7

@brocaar
I am attaching the steps i followed please spot the mistake i am making.
https://drive.google.com/file/d/1hzmKfXgHcShjy0j_Ghf6FSIq2mPiEwek/view?usp=sharing

1 Like
8

@brocaar I also followed the steps @KANNANS showed in the video. I am also getting error. Kindly help.

9

@brocaar i am getting the following error while adding tls certificates
Feb 24 05:18:31 localhost.localdomain chirpstack-application-server[211254]: time=“2021-02-24T05:18:31-05:00” level=warning msg=“grpc: addrConn.createTransport failed to connect to {10.10.20.58:8000 0 }. Err: connection error: desc = “transport: authentication handshake failed: x509: certificate is valid for 127.0.0.1, not 10.10.20.58”. Reconnecting…”

Is it due to mistakes in toml or the config files in certificate folder???

10

@brocaar

Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: 2021/02/26 01:28:09 http: TLS handshake error from 10.10.20.51:49907: remote error: tls: unknown certificate
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: time=“2021-02-26T01:28:09-05:00” level=info msg=“creating network-server client” server=“10.10.20.58:8000”
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: panic: runtime error: invalid memory address or nil pointer dereference
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x188 pc=0xcc1347]
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: goroutine 1734 [running]:
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-application-server/internal/storage.UpdateNetworkServer(0x17be5a0, 0xc000a9fef0, 0x7f37c3cba1d8, 0xc00010c778, 0xc00064d2e0, 0x40ccd8, 0x8)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /chirpstack-application-server/internal/storage/network_server.go:203 +0x907
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-application-server/internal/api/external.(*NetworkServerAPI).Update.func1(0x17cafc0, 0xc00010c778, 0x0, 0x0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /chirpstack-application-server/internal/api/external/network_server.go:167 +0x96
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-application-server/internal/storage.Transaction(0xc0003512c0, 0xc000a9fef0, 0x17b4f60)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /chirpstack-application-server/internal/storage/db.go:129 +0xa9
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-application-server/internal/api/external.(*NetworkServerAPI).Update(0xc000236a00, 0x17be5a0, 0xc000a9fef0, 0xc000a9fd40, 0xc000236a00, 0x7f37c3cf41e8, 0xc00024ba00)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /chirpstack-application-server/internal/api/external/network_server.go:166 +0x3d8
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-api/go/v3/as/external/api._NetworkServerService_Update_Handler.func1(0x17be5a0, 0xc000a9fef0, 0x138a040, 0xc000a9fd40, 0x20, 0xc000348b40, 0xc0006b3478, 0xaa4fd3)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/brocaar/chirpstack-api/go/v3@v3.7.8/as/external/api/networkServer.pb.go:882 +0x86
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x17be5a0, 0xc000a9fef0, 0x138a040, 0xc000a9fd40, 0xc0007a6900, 0xc0007a6920, 0xc0000dbec8, 0xc0006b3578, 0x383433362d373562, 0x3232333562643637)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xad
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x17be5a0, 0xc000a9fef0, 0x138a040, 0xc000a9fd40, 0x0, 0xc000a8fd00, 0x17be5a0, 0xc000a9fef0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-application-server/internal/logging.UnaryServerCtxIDInterceptor(0x17be5a0, 0xc000a9fef0, 0x138a040, 0xc000a9fd40, 0xc0007a6900, 0xc0007a6940, 0x0, 0x0, 0x0, 0x0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /chirpstack-application-server/internal/logging/logging.go:50 +0x2fd
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x17be5a0, 0xc000a9fec0, 0x138a040, 0xc000a9fd40, 0x20, 0xc006431e663f88ee, 0x216323843b, 0x26a8500)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.UnaryServerInterceptor.func1(0x17be5a0, 0xc000a9fdd0, 0x138a040, 0xc000a9fd40, 0xc0007a6900, 0xc0007a6960, 0x17be5a0, 0xc000a9fdd0, 0xc000584128, 0xc000a9fda0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/logging/logrus/server_interceptors.go:32 +0x110
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x17be5a0, 0xc000a9fdd0, 0x138a040, 0xc000a9fd40, 0x20, 0x138a040, 0xc000a9fd40, 0x0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware/tags.UnaryServerInterceptor.func1(0x17be5a0, 0xc000a9fd10, 0x138a040, 0xc000a9fd40, 0xc0007a6900, 0xc0007a6980, 0xb3784a, 0x1341220, 0xc0007a69a0, 0xc0007a6900)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/tags/interceptors.go:22 +0x86
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x17be5a0, 0xc000a9fd10, 0x138a040, 0xc000a9fd40, 0xc0006ef500, 0x0, 0xc0006b3b18, 0x40ccd8)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x17be5a0, 0xc000a9fd10, 0x138a040, 0xc000a9fd40, 0xc0007a6900, 0xc0007a6920, 0xc0006f4b88, 0x4f8558, 0x1370bc0, 0xc000a9fd10)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:34 +0xd5
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: github.com/brocaar/chirpstack-api/go/v3/as/external/api._NetworkServerService_Update_Handler(0x13011a0, 0xc000236a00, 0x17be5a0, 0xc000a9fd10, 0xc0005b0e40, 0xc000404d80, 0x17be5a0, 0xc000a9fd10, 0xc000847400, 0x23c9)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/github.com/brocaar/chirpstack-api/go/v3@v3.7.8/as/external/api/networkServer.pb.go:884 +0x14b
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000a4b60, 0x17ceda0, 0xc0001fd680, 0xc0006ef500, 0xc000405350, 0x2696290, 0x0, 0x0, 0x0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x50a
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: google.golang.org/grpc.(*Server).handleStream(0xc0000a4b60, 0x17ceda0, 0xc0001fd680, 0xc0006ef500, 0x0)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xcc9
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00075bbc0, 0xc0000a4b60, 0x17ceda0, 0xc0001fd680, 0xc0006ef500)
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xa1
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Feb 26 01:28:09 localhost.localdomain chirpstack-application-server[2946]: /go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
Feb 26 01:28:09 localhost.localdomain systemd[1]: chirpstack-application-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 26 01:28:09 localhost.localdomain systemd[1]: chirpstack-application-server.service: Failed with result ‘exit-code’.

This is the app server journal when i am updating the certificates in web interface

11

@brocaar

The go version shown in system environment and that shown when doing “cfssl version” are different. Is this causing the issue we are facing ? Please help

image

12

@brocaar
I deleted te network server and tied to add again with the certificates and now i am getting
Mar 01 06:09:03 localhost.localdomain chirpstack-application-server[1886]: time=“2021-03-01T06:09:03-05:00” level=error msg=“finished unary call with code Unknown” ctx_id=092607fa-16a1-402c-b911-b4a4ecfae4c4 error=“rpc error: code = Unknown desc = tls: failed to find any PEM data in certificate input”
what could be the reason
pls help