Hi All
I am trying to deploy 3.15 version in k8s with an external database(not container-based DB).
All components deployed fine but only application server is giving below error
time=“2021-06-24T15:35:11.694077275Z” level=fatal msg=“setup storage error: storage: migrate up error: Dirty database version 1. Fix and force version.”
Any suggestion on this?
Thanks
Thank you for your quick response.
Is this a config issue, I compared the config from the link you gave it looks to be the same?
One thing I am not using public schema instead I use search_path in dsn to pass my schema from external DB also I haven’t created chirpstack_as role as my db login user is different which I used in dsn
level=fatal msg=“setup storage error: storage: migrate up error: Dirty database version 1. Fix and force version.”
Sometimes it changed the Dirty database version to 27.
Can you please point me to what is wrong in the config if this is the case for app server 3.15?
Hi @brocaar any suggestion on this.
I tried changing automigrate=false in application server config but it start the server fine but while running now it is giving
time=“2021-06-25T16:18:45.128169093Z” level=error msg=“finished unary call with code Unknown” ctx_id=2446d3ef-32cb-4abb-8c64-6da19dfac7c5 error=“rpc error: code = Unknown desc = missing destination name username in *storage.User” grpc.code=Unknown grpc.method=Login grpc.service=api.InternalService grpc.start_time=“2021-06-25T16:18:45Z” grpc.time_ms=4.815 peer.address=“127.0.0.1:49562” span.kind=server system=grp
and if I set to true then I will get this and server wont start
level=fatal msg=“setup storage error: storage: migrate up error: Dirty database version 1. Fix and force version.”
Any suggestion if any issue with 3.15 version?
I still dont know which K8s-deployment you are using or if you built your own. This seems more of an issue with your particular configuration and not something to do with k8s.
Here is my app server config… do you see any issue with this?
apiVersion: v1
kind: ConfigMap
metadata:
name: chirpstack-application-server
namespace: my-test
labels:
app: chirpstack-application-server
data:
chirpstack-application-server.toml: |
[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=3
log_to_syslog=false# The number of times passwords must be hashed. A higher number is safer as # an attack takes more time to perform. password_hash_iterations=100000 # PostgreSQL settings # # Please note that PostgreSQL 9.5+ is required. [postgresql] # PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable). # # Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable') # it is also possible to use the following format: # 'user=chirpstack_as dbname=chirpstack_as sslmode=disable'.# # The following connection parameters are supported: # # * dbname - The name of the database to connect to # * user - The user to sign in as # * password - The user's password # * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) # * port - The port to bind to. (default is 5432) # * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq) # * fallback_application_name - An application_name to fall back to if one isn't provided. # * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. # * sslcert - Cert file location. The file must contain PEM encoded data. # * sslkey - Key file location. The file must contain PEM encoded data. # * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data. # # Valid values for sslmode are: # # * disable - No SSL # * require - Always SSL (skip verification) # * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA) # * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate) dsn="postgres://si_my-test_d:mypasswrd@myserver.abc.com:5432/mydb?sslmode=disable&search_path=myschema" # Automatically apply database migrations. //if i set this to false then i get //time=“2021-06-25T16:18:45.128169093Z” level=error msg=“finished unary call with code Unknown” ctx_id=2446d3ef-32cb-4abb-8c64-6da19dfac7c5 error=“rpc error: code = Unknown desc = missing destination name username in *storage.User” grpc.code=Unknown grpc.method=Login grpc.service=api.InternalService grpc.start_time=“2021-06-25T16:18:45Z” grpc.time_ms=4.815 peer.address=“127.0.0.1:49562” span.kind=server system=grp // if true then i get //level=fatal msg=“setup storage error: storage: migrate up error: Dirty database version 1. Fix and force version.” automigrate=false # Max open connections. # # This sets the max. number of open connections that are allowed in the # PostgreSQL connection pool (0 = unlimited). max_open_connections=4 # Max idle connections. # # This sets the max. number of idle connections in the PostgreSQL connection # pool (0 = no idle connections are retained). max_idle_connections=2 # Redis settings # # Please note that Redis 2.6.0+ is required. [redis] # Server address or addresses. # # Set multiple addresses when connecting to a cluster. servers=[ "redis.my-test.svc:6379", ] # Password. # # Set the password when connecting to Redis requires password authentication. password="root" # Database index. # # By default, this can be a number between 0-15. database=0 # Redis Cluster. # # Set this to true when the provided URLs are pointing to a Redis Cluster # instance. cluster=false # Master name. # # Set the master name when the provided URLs are pointing to a Redis Sentinel # instance. master_name="" # Connection pool size. # # Default (when set to 0) is 10 connections per every CPU. pool_size=0 # Application-server settings. [application_server] # Application-server identifier. # # Random UUID defining the id of the application-server installation (used by # ChirpStack Network Server as routing-profile id). # For now it is recommended to not change this id. id="6d5db27e-4ce2-4b2b-b5d7-91f069397978" # JavaScript codec settings. [application_server.codec.js] # Maximum execution time. max_execution_time="200ms" # Integration configures the data integration. # # This is the data integration which is available for all applications, # besides the extra integrations that can be added on a per-application # basis. [application_server.integration] # Payload marshaler. # # This defines how the MQTT payloads are encoded. Valid options are: # * protobuf: Protobuf encoding # * json: JSON encoding (easier for debugging, but less compact than 'protobuf') # * json_v3: v3 JSON (will be removed in the next major release) # marshaler="json" # Enabled integrations. # # Enabled integrations are enabled for all applications. Multiple # integrations can be configured. # Do not forget to configure the related configuration section below for # the enabled integrations. Integrations that can be enabled are: # * mqtt - MQTT broker # * amqp - AMQP / RabbitMQ # * aws_sns - AWS Simple Notification Service (SNS) # * azure_service_bus - Azure Service-Bus # * gcp_pub_sub - Google Cloud Pub/Sub # * postgresql - PostgreSQL database enabled=["mqtt"] # MQTT integration backend. [application_server.integration.mqtt] # MQTT topic templates for the different MQTT topics. # # The meaning of these topics are documented at: # https://www.chirpstack.io/application-server/integrate/data/ # # The following substitutions can be used: # * "{{ .ApplicationID }}" for the application id. # * "{{ .DevEUI }}" for the DevEUI of the device. # # Note: the downlink_topic_template must contain both the application id and # DevEUI substitution! uplink_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/rx" downlink_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/tx" join_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/join" ack_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/ack" error_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/error" status_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/status" location_topic_template="application/{{ .ApplicationID }}/device/{{ .DevEUI }}/location" # Retained messages configuration. # # The MQTT broker will store the last publised message, when retained message is set # to true. When a client subscribes to a topic with retained message set to true, it will # always receive the last published message. uplink_retained_message=false join_retained_message=false ack_retained_message=false error_retained_message=false status_retained_message=false location_retained_message=false # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws) server="tcp://mosquitto.my-test.svc:1883" # Connect with the given username (optional) username="admin" # Connect with the given password (optional) password="" # admin = YWRtaW4K # Maximum interval that will be waited between reconnection attempts when connection is lost. # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'. # max_reconnect_interval="1m0s" # Quality of service level # # 0: at most once # 1: at least once # 2: exactly once # # Note: an increase of this value will decrease the performance. # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels # EMQX is another viable alternative qos=0 # Clean session # # Set the "clean session" flag in the connect message when this client # connects to an MQTT broker. By setting this flag you are indicating # that no messages saved by the broker for this client should be delivered. clean_session=true # Client ID # # Set the client id to be used by this client when connecting to the MQTT # broker. A client id must be no longer than 23 characters. When left blank, # a random id will be generated. This requires clean_session=true. client_id="" # CA certificate file (optional) # # Use this when setting up a secure connection (when server uses ssl://...) # but the certificate used by the server is not trusted by any CA certificate # on the server (e.g. when self generated). ca_cert="" # TLS certificate file (optional) tls_cert="" # TLS key file (optional) tls_key="" # Settings for the "internal api" # # This is the API used by ChirpStack Network Server to communicate with ChirpStack Application Server # and should not be exposed to the end-user. [application_server.api] # ip:port to bind the api server bind="0.0.0.0:8001" # ca certificate used by the api server (optional) ca_cert="" # tls certificate used by the api server (optional) tls_cert="" # tls key used by the api server (optional) tls_key="" # Public ip:port of the application-server API. # # This is used by ChirpStack Network Server to connect to ChirpStack Application Server. When running # ChirpStack Application Server on a different host than ChirpStack Network Server, make sure to set # this to the host:ip on which ChirpStack Network Server can reach ChirpStack Application Server. # The port must be equal to the port configured by the 'bind' flag # above. public_host="chirpstack-application-server.my-test.svc:8001" # Settings for the "external api" # # This is the API and web-interface exposed to the end-user. [application_server.external_api] # ip:port to bind the (user facing) http server to (web-interface and REST / gRPC api) bind="0.0.0.0:8080" # http server TLS certificate (optional) tls_cert="" # http server TLS key (optional) tls_key="" # JWT secret used for api authentication / authorization # You could generate this by executing 'openssl rand -base64 32' for example jwt_secret="verysecret" # Allow origin header (CORS). # # Set this to allows cross-domain communication from the browser (CORS). # Example value: https://example.com. # When left blank (default), CORS will not be used. cors_allow_origin="" # when set, existing users can't be re-assigned (to avoid exposure of all users to an organization admin)" disable_assign_existing_users=false # Settings for the remote multicast setup. [application_server.remote_multicast_setup] # Synchronization interval. sync_interval="1s" # Synchronization retries. sync_retries=3 # Synchronization batch-size. sync_batch_size=100 # Settings for the fragmentation-session setup. [application_server.fragmentation_session] # Synchronization interval. sync_interval="1s" # Synchronization retries. sync_retries=3 # Synchronization batch-size. sync_batch_size=100 # Join-server configuration. # # ChirpStack Application Server implements a (subset) of the join-api specified by the # LoRaWAN Backend Interfaces specification. This API is used by ChirpStack Network Server # to handle join-requests. [join_server] # ip:port to bind the join-server api interface to bind="0.0.0.0:8003" # CA certificate (optional). # # When set, the server requires a client-certificate and will validate this # certificate on incoming requests. ca_cert="" # TLS server-certificate (optional). # # Set this to enable TLS. tls_cert="" # TLS server-certificate key (optional). # # Set this to enable TLS. tls_key="" # Key Encryption Key (KEK) configuration. # # The KEK mechanism is used to encrypt the session-keys sent from the # join-server to the network-server. # # The ChirpStack Application Server join-server will use the NetID of the requesting # network-server as the KEK label. When no such label exists in the set, # the session-keys will be sent unencrypted (which can be fine for # private networks). # # Please refer to the LoRaWAN Backend Interface specification # 'Key Transport Security' section for more information. [join_server.kek] # Application-server KEK label. # # This defines the KEK label used to encrypt the AppSKey (note that the # AppSKey is signaled to the NS and on the first received uplink from the # NS to the AS). # # When left blank, the AppSKey will be sent unencrypted (which can be fine # for private networks). as_kek_label="" # KEK set. # # Example (the [[join_server.kek.set]] can be repeated): # [[join_server.kek.set]] # # KEK label. # label="000000" # # Key Encryption Key. # kek="01020304050607080102030405060708" # Metrics collection settings. [metrics] # Timezone # # The timezone is used for correctly aggregating the metrics (e.g. per hour, # day or month). # Example: "Europe/Amsterdam" or "Local" for the the system's local time zone. timezone="Local" # Metrics stored in Redis. # # The following metrics are stored in Redis: # * gateway statistics [metrics.redis] # Aggregation intervals # # The intervals on which to aggregate. Available options are: # 'MINUTE', 'HOUR', 'DAY', 'MONTH'. aggregation_intervals=["MINUTE", "HOUR", "DAY", "MONTH"] # Aggregated statistics storage duration. minute_aggregation_ttl="2h0m0s" hour_aggregation_ttl="48h0m0s" day_aggregation_ttl="2160h0m0s" month_aggregation_ttl="17520h0m0s" # # Metrics stored in Prometheus. # # # # These metrics expose information about the state of the ChirpStack Network Server # # instance. # [metrics.prometheus] # # Enable Prometheus metrics endpoint. # endpoint_enabled=true # # # The ip:port to bind the Prometheus metrics server to for serving the # # metrics endpoint. # bind="0.0.0.0:8004" # # # API timing histogram. # # # # By setting this to true, the API request timing histogram will be enabled. # # See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms # api_timing_histogram=false # Monitoring settings. # # Note that this replaces the metrics.prometheus configuration. If a # metrics.prometheus if found in the configuration then it will fall back # to that and the monitoring section is ignored. [monitoring] # IP:port to bind the monitoring endpoint to. # # When left blank, the monitoring endpoint will be disabled. bind="" # Prometheus metrics endpoint. # # When set to true, Prometheus metrics will be served at '/metrics'. prometheus_endpoint=false # Prometheus API timing histogram. # # By setting this to true, the API request timing histogram will be enabled. # See also: https://github.com/grpc-ecosystem/go-grpc-prometheus#histograms prometheus_api_timing_histogram=false # Health check endpoint. # # When set to true, the healthcheck endpoint will be served at '/health'. # When requesting, this endpoint will perform the following actions to # determine the health of this service: # * Ping PostgreSQL database # * Ping Redis database healthcheck_endpoint=false
Thanks, My issue is resolved but errors were a bit misleading I feel.
could you post here how the issue resolved please, otherwise people might look in here and wonder if they are having the same Issue or made the same mistake. 
I reverted the changes you suggested for adding the search_path in DSN and instead set the current_schema to my schema instead of the public in the DB permission.