Like swatting a real mosquitto

Hi, I have been trying to set up ChirpStack in a Kubernetes space for a while now. Seem to have app-server, net-server, gateway-bridge etc done, but mosquitto doesn’t seem to be working for me, and I can’t find any resources online that answer this particular question(and I’m not very sure how exactly to ask it).

Above are some of the pods I have atm. App-server, net-server, gateway-bridge all spin up and run, however the Mosquitto broker moves to ‘Complete’ and goes right into the CrashLoopBackOff. This obviously causes the mqtt integration part of the components to fail, and a colleague who’s a bit more savvy with K8s has told me the ports configured are not listening, meaning I can’t even access the app-server UI atm. While I might be wrong (and if I am, I would be happy to be corrected so long as I can reach the answer) it appears it’s down to Mosquitto.

I have figured it might have been something to do with a lack of config, so I’ve spent a few days putting together the mosquitto.conf file with “allow_anonymous true” hoping to get a connection from any of my ChirpStack components, but the logs just indicate an mqtt connection refused error(now gone to timeout since mqtt broker always his Crash loop).

output of kubectl logs chirpstack-application-server

Because no connection could be made, I tried to add in the password_file and make allow_anonymous false. Below is my current config if anyone might have a general idea what is wrong.

Thanks in advance.

configMap-1.yml

kind: ConfigMap
metadata:
  name: mosquitto-password
  namespace: ****
  labels:
    app: chirpstack-mosquitto  
data:
  password_file.txt: |
    admin:admin
    user:user
    app-server:app-server
    net-server:net-server
    gateway-bridge:gateway-bridge

configMap.yml

kind: ConfigMap
metadata:
  name: mosquitto-config
  namespace: ****
  labels:
    app: chirpstack-mosquitto  
data:
  mosquitto.conf: |    
    persistence true
    persistence_location /mosquitto/data/
    # per_listener_settings false
    log_dest stdout
    # listener 1886
    listener 1883
    protocol mqtt
    # Defaults to false, unless there are no listeners defined in the configuration
    # file, in which case it is set to true, but connections are only allowed from
    # the local machine.
    allow_anonymous false
    password_file /.config/mosquitto/auth/password_file.txt
    #    cafile: /mosquitto/config/certs/ca.crt
    #    certfile: /mosquitto/config/certs/server.crt
    #    keyfile: /mosquitto/config/certs/server.key
    require_certificate false
    use_identity_as_username false

deployment.yml

kind: Deployment
metadata:
  name: chirpstack-mosquitto
  namespace: ****  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: chirpstack-mosquitto
  template:
    metadata:
      labels:
        app: chirpstack-mosquitto
    spec:
      containers:
      - name: chirpstack-mosquitto
        image: ****/chirpstack/eclipse-mosquitto:1.6.12
        ports:
        - containerPort: 1883
        volumeMounts:
        - name: password-file
          mountPath: /.config/mosquitto/auth/password_file.txt
          subPath: password_file.txt
        - name: mosquitto-data
          mountPath: /mosquitto/data
        - name: mosquitto-log
          mountPath: /mosquitto/log
        - name: config-file
          mountPath: /.config/mosquitto/mosquitto.conf
          subPath: mosquitto.conf         
      securityContext:
        runAsNonRoot: true
        fsGroup: 1
        runAsGroup: 1000
        runAsUser: 1000
        supplementalGroups:
        - 1    
      volumes:
      - name: config-file
        configMap:
          name: mosquitto-config
      - name: password-file
        configMap:
          name: mosquitto-password        
      - name: mosquitto-data
        emptyDir: {}
      - name: mosquitto-log
        emptyDir: {}   

service.yml

kind: Service
metadata:
  name: chirpstack-mosquitto
  namespace: 186215-poc  
spec:
  type: ClusterIP
  ports:
    - name: mqtt 
      port: 1883
      targetPort: 1883
      protocol: TCP  
  selector:
    app: chirpstack-mosquitto    
1 Like

I would start with the crash loop back off / errors with the mosquitto container. You could take a look at the out put of kubectl describe pod <pod_name>. It will list events at the bottom, and a last exit code somewhere in the middle of the output. I would also look at the logs for that container kubectl logs -f <pod_name>.

I am not super familiar with mosquitto, but I think the password needs to be generated with their command line tool mosquitto_passwd -c <file> <username>. When I run that with admin as the username and password it outputs an encoded string value for the password.

1 Like

I also just published a blog I’ve been working on to guide the process of deploying ChirpStack to Kubernetes - maybe this will help! https://johnroesler.medium.com/deploying-chirpstack-to-kubernetes-f55b0dc9d12e

2 Likes

The solution to the problem for this was the version. As soon as I dropped from Mosquitto version 1.6.12 to 1.5 the pod began working. So thank you for the repository with a very significant hint.

shameless plug: https://gitlab.com/wobcom/chirpstack-helm

2 Likes

might try the headless service route for loadbalancing. :slight_smile:
Overall nice post.

oooh, nice helm charts!