Login issue with Chirpstack V4

amac · August 15, 2022, 8:02am

Hi, Thanks for all the work to release Chirpstack v4.
I have been using V3 OK for a while and tried to install V4 on a new VPS running both Debian 11 and Ubuntu 22.04 and tried the migration process with a clone of my running Chirpstack V3 server but I am getting the same issue with the login on all install attempts.

I followed the doc to install and configure OK and completed with no errors but when I try to login it gets a “Incomplete Response” error and makes Chirpstack crash and restart.

If I try to login with an invalid user, the web interface correctly says “invalid username or password” and does not cause the crash and restart but any valid user login attempt (including the default admin/admin) causes the same “incomplete response” error

The log shows

Aug 15 17:55:59 npchirp2 kernel: [ 562.895964] traps: tokio-runtime-w[1029] trap invalid opcode ip:55b7348d9ef0 sp:7f947b9fa040 error:0 in chirpstack[55b733b51000+1064000]
Aug 15 17:55:59 npchirp2 systemd[1]: chirpstack.service: Main process exited, code=killed, status=4/ILL
Aug 15 17:55:59 npchirp2 systemd[1]: chirpstack.service: Failed with result ‘signal’.
Aug 15 17:56:00 npchirp2 systemd[1]: chirpstack.service: Service RestartSec=100ms expired, scheduling restart.
Aug 15 17:56:00 npchirp2 systemd[1]: chirpstack.service: Scheduled restart job, restart counter is at 1.
Aug 15 17:56:00 npchirp2 systemd[1]: Stopped ChirpStack open-source LoRaWAN Network Server.
Aug 15 17:56:00 npchirp2 systemd[1]: Started ChirpStack open-source LoRaWAN Network Server.

Given this does not occur with an invalid user/pwd, I am guessing it is not the DB lookup but something that happens directly afterwards with a valid login

Can you give me any clues what may be going on with this?

Regards
Andrew.

brocaar · August 15, 2022, 9:14am

That is very odd. Which hosting / cloud provider are you using?

amac · August 15, 2022, 9:24am

Thanks for the reply
Network Presence which is an Australian company which I am also part of and very standard KVM VPS system which is also where I run the Chirpstack V3 server OK and a wide range of other Linux VPS machines on different distros over many years.
The same issue happens on several clean Debian 11 and Ubuntu22.04 installs and on a clone of the older V3 server I tried to migrate to migrate on Debian 10. The existing V3 server was upgraded OK as well before trying to migrate. I was not able to find much on searching the trap error either
Andrew

amac · August 15, 2022, 9:34am

I forgot to add that the V3 server clone work OK with the chirpstack-application-server but when I install V4 and stop the application-server and launch the V4 chirpstack that the issue appears again
Andrew

brocaar · August 15, 2022, 9:50am

The reason why I’m asking is that I can not reproduce this issue locally or on my own testing VM instance. I just setup the latest ChirpStack v4 build on a Debian 11 VM and it runs fine.

Would you be able to provide me with a test VM on Network Presence so that I can reproduce this issue and find out what triggers this? If I understand correctly, the issue already exists before the data migration, when logging in using admin / admin, so it doesn’t require any of your device data on it.

If this is possible, please send me a private message to exchange the hostname & credentials and I’ll look into this asap.

Thanks for reporting!

amac · August 15, 2022, 10:09am

I am happy to reimage my test instance with debian 11 so you have a clean start and send you credentials but I am unsure how to send a private message on this forum system

The issue exists from the start after installing V4 and trying to login for the first time. I may be just missing something in the install but did follow the doc for the install.

Please advise how I can send a private message and I will forward you login credentials.

amac · August 15, 2022, 10:13am

Sorry, I found the private message function but it said I was not allowed to send a message to your user

Andrew

brocaar · August 15, 2022, 11:02am

I just sent you a PM

brocaar · August 15, 2022, 8:16pm

Update:

Thanks to the VM that @amac provided, I have been able to find what has caused this issue. The error happens at the execution of this line:

github.com

chirpstack/chirpstack/blob/master/chirpstack/src/storage/user.rs#L275

      
        
            }
            
            
fn verify_password(pw: &str, hash: &str) -> bool {
                let parsed = match PasswordHash::new(hash) {
                    Ok(v) => v,
                    Err(_) => {
                        return false;
                    }
                };
            
            
    Pbkdf2.verify_password(pw.as_bytes(), &parsed).is_ok()
            }
            
            
#[cfg(test)]
            pub mod test {
                use super::*;
                use crate::test;
            
            
    pub async fn create_user() -> User {
                    let mut user = User {
                        is_admin: true,

The pbkdf2 crate depends on sha2, which if available (runtime check) uses CPU instructions for crypto operations or else uses a software implementation. My assumption is that it thinks the CPU instructions are available, but once these are executed it fails.

I have confirmed that once the software implementation is forced, all works fine

github.com

RustCrypto/hashes/blob/master/sha2/Cargo.toml#L34

      
        
            
            
[dev-dependencies]
            digest = { version = "0.10.3", features = ["dev"] }
            hex-literal = "0.2.2"
            
            
[features]
            default = ["std"]
            std = ["digest/std"]
            asm = ["sha2-asm"] # WARNING: this feature SHOULD NOT be enabled by library crates
            compress = [] # Expose compress functions
            force-soft = [] # Force software implementation
            asm-aarch64 = ["asm"] # DEPRECATED: use `asm` instead
            
            
[package.metadata.docs.rs]
            all-features = true
            rustdoc-args = ["--cfg", "docsrs"]

brocaar · August 16, 2022, 8:14am

An other update, this issue happened on a VM using a Qemu CPU. The sha2 library does a runtime check if the avx2 CPU extension is available so that the the library can use these CPU instructions over a software implementation of these instructions:

github.com

RustCrypto/hashes/blob/master/sha2/src/sha512/x86.rs#L14

      
        
            
            
use core::mem::size_of;
            
            
#[cfg(target_arch = "x86")]
            use core::arch::x86::*;
            #[cfg(target_arch = "x86_64")]
            use core::arch::x86_64::*;
            
            
use crate::consts::K64;
            
            
cpufeatures::new!(avx2_cpuid, "avx2");
            
            
pub fn compress(state: &mut [u64; 8], blocks: &[[u8; 128]]) {
                // TODO: Replace with https://github.com/rust-lang/rfcs/pull/2725
                // after stabilization
                if avx2_cpuid::get() {
                    unsafe {
                        sha512_compress_x86_64_avx2(state, blocks);
                    }
                } else {
                    super::soft::compress(state, blocks);

github.com

RustCrypto/utils/blob/master/cpufeatures/src/x86.rs#L76

      
        
                ("ssse3", 0, ecx, 9),
                ("fma", 0, ecx, 12),
                ("sse4.1", 0, ecx, 19),
                ("sse4.2", 0, ecx, 20),
                ("popcnt", 0, ecx, 23),
                ("aes", 0, ecx, 25),
                ("avx", 0, ecx, 28),
                ("rdrand", 0, ecx, 30),
                ("sgx", 1, ebx, 2),
                ("bmi1", 1, ebx, 3),
                ("avx2", 1, ebx, 5),
                ("bmi2", 1, ebx, 8),
                ("rdseed", 1, ebx, 18),
                ("adx", 1, ebx, 19),
                ("sha", 1, ebx, 29),
            }

It looks like the (emulated) CPU does advertise the presence of avx2 as documented here (which is checked by the cpufeatures library):

(https://www.intel.com/content/dam/develop/external/us/en/documents/36945)

But it doesn’t actually support the avx2 extension instructions, thus when sha2 starts using these instructions, the execution panics because of the unsupported instructions.

This was a rather deep dive

amac · August 16, 2022, 8:53am

Many thanks for all the work to debug this rather esoteric issue.

Regards
Andrew

brocaar · August 16, 2022, 9:39am

Actually, I believe it is a bug and I created a pull-request to fix this:

github.com/RustCrypto/hashes

To use avx2, both avx and avx2 must be checked.

RustCrypto:master ← brocaar:fix_avx

opened 09:37AM - 16 Aug 22 UTC

brocaar

+2 -1

Please see the Intel note about detecting the `avx2` extension: ![image](http…s://user-images.githubusercontent.com/165497/184846210-97148b99-318f-4491-83ac-a211f7117c53.png) https://www.intel.com/content/dam/develop/external/us/en/documents/36945 It states that both the support for AVX and AVX2 must be detected. --- Some context: I had a bug report that trying to login caused the application (ChirpStack) to panic. After a deep dive, it turned out that the host (Qemu CPU) does not support these instructions, but `avx2_cpuid::get()` returns true. Then I noticed that the Intel docs state that both `AVX` and `AVX2` must be checked. Running this application on this specific Qemu CPU returns: ``` Supports avx: false Supports avx2: true ``` ```rust cpufeatures::new!(avx_cpuid, "avx"); cpufeatures::new!(avx2_cpuid, "avx2"); fn main() { println!("Supports avx: {}", avx_cpuid::get()); println!("Supports avx2: {}", avx2_cpuid::get()); } ``` As reference, this is the content of `/proc/cpuinfo`: ```text processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 13 model name : QEMU Virtual CPU version (cpu64-rhel6) stepping : 3 microcode : 0x1000065 cpu MHz : 3393.624 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm sse4a 3dnowprefetch vmmcall bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass bogomips : 6787.24 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 13 model name : QEMU Virtual CPU version (cpu64-rhel6) stepping : 3 microcode : 0x1000065 cpu MHz : 3393.624 cache size : 512 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm sse4a 3dnowprefetch vmmcall bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass bogomips : 6787.24 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ```

fouriq · September 28, 2022, 11:46am

hi @brocaar ,

I seem to have ran into this problem on a fresh docker install.
The installation runs fine but the application server crashes and restarts the moment I try to log in.
I think the docker images are still on 4.0.0. Can you update to 4.0.1 or is there another way for me to update?

system · December 27, 2022, 11:47am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.