Upgrade 4.6 to 4.7, SSL Problem with MQTT Integration

ralphmu · March 26, 2024, 3:11pm

Today i upgraded a Chirpstack Server (Debian based) from Chirpstack Version 4.6 to 4.7. The MQTT Integration is working with MQTT Mosquitto Broker in the same Subnet over SSL. Since the update I get the following error message when starting the Chirpstack server:

Configuring client with TLS certificate, ca_cert: /etc/chirpstack/certs/ca.pem, tls_cert: /etc/chirpstack/certs/mqtt-broker.pem, tls_key: /etc/chirpstack/certs/mqtt-broker-key.pem
chirpstack[4306]: Error: Setup MQTT integration
chirpstack[4306]: Caused by:
chirpstack[4306]:     No private key found
systemd[1]: chirpstack.service: Main process exited, code=exited, status=1/FAILURE

I could not see any breaking changes in the changelog for version 4.7, except the switch from openssl to rustls.

Does Anybody have the same/similar problems?

ralphmu · March 26, 2024, 8:39pm

It seems that the private keys should be now in PKCS#8 format. As this commit from @brocaar suggests: e96e828

They can be converted with openssl like this:

openssl pkcs8 -in mykey.pem -topk8 -nocrypt -out mykey-pkcs8.pem

brocaar · March 27, 2024, 8:57am

To add some context to this, ChirpStack v4.7 migrated from OpenSSL to Rustls. The big advantage is that it is no longer needed to (cross)compile OpenSSL such that ChirpStack can be linked against it.

Unfortunately, Rustls is more restrictive in the certificate formats that it accept than OpenSSL. As far as I understand PKCS#8 is in general the preferred method. For RSA private-keys, there is an easy fix and ChirpStack will automatically do the conversion to PKCS#8:

github.com

chirpstack/chirpstack/blob/master/chirpstack/src/certificate.rs#L102


      
          
              Ok((
                  not_after,
                  ca_cert.serialize_pem()?,
                  app_cert.serialize_pem_with_signer(&ca_cert)?,
                  app_cert.serialize_private_key_pem(),
              ))
          }
          
          fn private_key_to_pkcs8(pem: &str) -> Result<String> {
              if pem.contains("RSA PRIVATE KEY") {
                  let pkey = RsaPrivateKey::from_pkcs1_pem(pem).context("Read RSA PKCS#1")?;
                  let pkcs8_pem = pkey.to_pkcs8_pem(LineEnding::default())?;
                  Ok(pkcs8_pem.as_str().to_owned())
              } else {
                  Ok(pem.to_string())
              }
          }

Unfortunate for EC keys I haven’t found an easy solution yet, see also: sec1: Trait DecodeEcPrivateKey not implemented for EcPrivateKey · Issue #747 · RustCrypto/formats · GitHub

Contributions are welcome

ralphmu · March 27, 2024, 9:36am

Thanks for the additional context and the explanations. I’m not very familiar with Rust, nor with PKCS#8 Key format. I’m a bit confused, because the failing key is a RSA key, generated with the tools from the chirpstack-certificates repository:

chirpstack-4:/etc/chirpstack/certs# file mqtt-broker-key.pem mqtt-broker-key.pem: PEM RSA private key

After conversion it is “just” ASCII Text:

chirpstack-4:/etc/chirpstack/certs# file mqtt-broker-key-pkcs8.pem mqtt-broker-key-pkcs8.pem: ASCII text

The RSA Key does not work, the “ASCII Text” does.

brocaar · March 27, 2024, 9:40am

How does the first line of the private-key look like? E.g.

-----BEGIN EC PRIVATE KEY-----

ralphmu · March 27, 2024, 9:47am

It is a RSA Private Key:

chirpstack-4:/etc/chirpstack/certs# cat mqtt-broker-key.pem 
-----BEGIN RSA PRIVATE KEY-----
MI...

The ASCII Text one:

chirpstack-4:/etc/chirpstack/certs# cat mqtt-broker-key-pkcs8.pem 
-----BEGIN PRIVATE KEY-----
MI...

brocaar · March 27, 2024, 10:30am

Ah, I see… The auto-conversion to PKCS#8 was not used for all the private-key reads, the commit below fixes this:

This should fix your issue.

ralphmu · March 27, 2024, 10:32am

Thanks for your fast reaction and the fix!