Upgrading Keycloak 13 to 23 (Kubernetes)

Hey all,

I’m trying to upgrade my keycloak image in my deployment from quay io/keycloak/keycloak:13.0.1 to quay io/keycloak/keycloak:23.0.
However, when I push my image to the cluster the pod STATUS changes from completed to crashloopbackoff.
My KC connects to a db in Azure.

My deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: development
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
        date: '1650525506'
    spec:
      imagePullSecrets:
        - name: mysecret
      containers:
        - name: keycloak
          imagePullPolicy: Always
          image: quay.io/keycloak/keycloak:23.0
          resources:
            requests:
              cpu: 50m
              memory: 256Mi
            limits:
              cpu: 250m
              memory: 1024Mi
          env:
            - name: KEYCLOAK_USER
              value: 'admin'
            - name: KEYCLOAK_PASSWORD
              value: 'admin'
            - name: PROXY_ADDRESS_FORWARDING
              value: 'true'
            - name: DB_VENDOR
              value: 'mssql'
            - name: DB_USER
              value: db-user
            - name: DB_PASSWORD
              value: 'db-pass123'
            - name: DB_ADDR
              value: 'this.is.my.address'
            - name: DB_DATABASE
              value: 'keycloak-db'
            - name: KEYCLOAK_FRONTEND_URL
              value: 'https://keycloak.mydomain.com'
          ports:
            - name: http
              containerPort: 8080
            - name: https
              containerPort: 8443

After it goes into crashloopbackoff the only logs I see are.

Keycloak - Open Source Identity and Access Management

Find more information at: https://www.keycloak.org/docs/latest

Usage:

kc.sh [OPTIONS] [COMMAND]

Use this command-line tool to manage your Keycloak cluster.
Make sure the command is available on your "PATH" or prefix it with "./" (e.g.:
"./kc.sh") to execute from the current folder.

Options:

-cf, --config-file <file>
                     Set the path to a configuration file. By default, configuration properties are
                       read from the "keycloak.conf" file in the "conf" directory.
-h, --help           This help message.
-v, --verbose        Print out error details when running this command.
-V, --version        Show version information

Commands:

  build                   Creates a new and optimized server image.
  start                   Start the server.
  start-dev               Start the server in development mode.
  export                  Export data from realms to a file or directory.
  import                  Import data from a directory or a file.
  show-config             Print out the current configuration.
  tools                   Utilities for use and interaction with the server.
    completion            Generate bash/zsh completion script for kc.sh.

Examples:

  Start the server in development mode for local development or testing:

      $ kc.sh start-dev

  Building an optimized server runtime:

      $ kc.sh build <OPTIONS>

  Start the server in production mode:

      $ kc.sh start <OPTIONS>

  Enable auto-completion to bash/zsh:

      $ source <(kc.sh tools completion)

  Please, take a look at the documentation for more details before deploying in
production.

Use "kc.sh start --help" for the available options when starting the server.
Use "kc.sh <command> --help" for more information about other commands.

Could anyone help me with what I might be doing wrong, or not doing here?

Thanks in advance.

You missed reading the upgrade notes. With Keycloak 17 came the switch from wildfly to quarkus.
The environment variables for the quarkus distribution have an entirely different naming scheme.

Check Migrating to Quarkus distribution - Keycloak and the other guides provided on Guides - Keycloak

2 Likes

Hey ThoreKr, thanks for the notes. These helped me quite a bit and I think I finally got it working. But I’m not sure why it’s been trying to update the DB for over 8 hours now.

My pod status is 1/1 Running,

The last log I see from Keycloak is

ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user ‘admin’ to realm ‘master’: user with username exists

But on verbose it keeps repeating

2024-01-31 15:17:43,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) new JtaTransactionWrapper
2024-01-31 15:17:43,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) was existing? false
2024-01-31 15:17:43,834 DEBUG [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-0) Executed scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$1403/0x0000000100befcf0
2024-01-31 15:17:43,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper  commit
2024-01-31 15:17:43,835 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper end
2024-01-31 15:17:48,044 DEBUG [org.jgroups.protocols.dns.DNS_PING] (jgroups-81,keycloak-9dcf977d-2bfp8-5822) keycloak-9dcf977d-2bfp8-5822: entries collected from DNS (in 5 ms): [10.244.1.14:0]
2024-01-31 15:17:48,045 DEBUG [org.jgroups.protocols.dns.DNS_PING] (jgroups-81,keycloak-9dcf977d-2bfp8-5822) keycloak-9dcf977d-2bfp8-5822: sending discovery requests to hosts [10.244.1.14:0] on ports [7800 .. 7800]
2024-01-31 15:17:48,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) new JtaTransactionWrapper
2024-01-31 15:17:48,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) was existing? false
2024-01-31 15:17:48,836 DEBUG [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-0) Executed scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$1403/0x0000000100befcf0
2024-01-31 15:17:48,836 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper  commit
2024-01-31 15:17:48,837 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper end
2024-01-31 15:17:53,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) new JtaTransactionWrapper
2024-01-31 15:17:53,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) was existing? false
2024-01-31 15:17:53,833 DEBUG [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-0) Executed scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$1403/0x0000000100befcf0
2024-01-31 15:17:53,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper  commit
2024-01-31 15:17:53,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) JtaTransactionWrapper end
2024-01-31 15:17:58,833 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) new JtaTransactionWrapper
2024-01-31 15:17:58,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-0) was existing? false
2024-01-31 15:17:58,834 DEBUG [org.keycloak.services.scheduled.ScheduledTaskRunner] (Timer-0) Executed scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$1403/0x0000000100befcf0

I’ve read that the ERROR log isn’t an actual error in keycloak and it appears in the new version, but from there it doesn’t really do anything. I’m trying to understand if this is something to do with my config for KC or if it’s because I’m using a basic tier db from azure where the compute power might not be enough.

Resolved:

The final issues I’ve had with it was my service was using ClusterIP, so I change it to NodePort.
and the Path for KC_HOSTNAME_URL & KC_HOSTNAME_ADMIN_URL had to be updated to /auth instead of /keycloak/auth.

Thank You :slight_smile:

1 Like

Any chance you can post your updated YAML file (and any configmaps you had to change) for your converted deployment? I’m going through the same thing you discussed in your original post and seeing the changes you had to make would be very helpful

Hey sorry about the late reply.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: myns
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port: '8080'
      prometheus.io/path: '/metrics'
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:23.0.5
        args: 
          - "start"
          - "--cache-stack=kubernetes"
          - "--transaction-xa-enabled=false"
          - "--log-level=INFO"
        envFrom:
        - configMapRef:
            name: keycloak-config
        env:
        - name: KEYCLOAK_ADMIN
          valueFrom:
            secretKeyRef:
              name: keycloak-secrets
              key: KEYCLOAK_ADMIN
        - name: KEYCLOAK_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secrets
              key: KEYCLOAK_ADMIN_PASSWORD
        - name: KC_DB_URL
          valueFrom:
            secretKeyRef:
              name: keycloak-secrets
              key: KC_DB_URL
        - name: KC_DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: keycloak-secrets
              key: KC_DB_USERNAME
        - name: KC_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secrets
              key: KC_DB_PASSWORD
        ports:
        - name: http
          containerPort: 8080
        - name: jgroups
          containerPort: 7600

This is my deployment and the variables had all been changed.

So I had some of them divided between a configmap and a secret.

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-secrets
  namespace: myns
type: Opaque
data:
  KEYCLOAK_ADMIN: <my b64 ecoded value>
  KEYCLOAK_ADMIN_PASSWORD: <my b64 ecoded value>
  KC_DB_URL: <my b64 ecoded value>
  KC_DB_USERNAME: <my b64 ecoded value>
  KC_DB_PASSWORD: <my b64 ecoded value>
apiVersion: v1
kind: ConfigMap
metadata:
  name: keycloak-config
  namespace: myns
data:
  KC_PROXY: "edge"
  jgroups.dns.query: "keycloak"
  PROXY_ADDRESS_FORWARDING: "true"
  KC_HEALTH_ENABLED: "true"
  KC_METRICS_ENABLED: "true"
  KC_HTTP_ENABLED: "true"
  KC_HTTP_RELATIVE_PATH: "/auth"
  KC_HOSTNAME_URL: "https://keycloak.example.dev/auth"
  KC_HOSTNAME_ADMIN_URL: "https://keycloak.example.dev/auth"
  KC_DB: "mssql"
  KC_DB_URL_HOST: "mssql"
  KC_DB_URL_PORT: "1433" //port for microsoft azure sql server
1 Like