Slow response times for a while when running keycloak 17 (quarkus) on kubernetes and clustered

Configuring the server
,
1

Hi!

We are new to keycloak and just about to start configuring it for a HA production environment.
We will run it in kubernetes and start with keycloak 17.

While everything seem to work, response times are extremely slow at startup, upgrades and if a pod dies. After that it takes a couple of minutes and response times are reasonable again. Is something missing in the configuration?

The Dockerfile to build keycloak 17:

FROM quay.io/keycloak/keycloak:17.0.0 as builder

ENV KC_METRICS_ENABLED=true \
    KC_DB=mysql \
    KC_CACHE_STACK=kubernetes \
    KC_STACK=ispn

RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:17.0.0
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/


ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

Statefulset + Service:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  serviceName: keycloak-headless
  replicas: 2
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: # image-built-from-dockerfile
        args: ["-Djgroups.dns.query=keycloak-headless", "start"]
        env:
        - name: KEYCLOAK_ADMIN
          value: "admin"
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: # admin password
        - name: KC_HOSTNAME
          value: # host
        - name: KC_HOSTNAME_STRICT_BACKCHANNEL
          value: "true"
        - name: KC_PROXY
          value: "edge"
        - name: KC_DB_USERNAME
          value: # db username
        - name: KC_DB_PASSWORD
          value: # db password
        - name: KC_DB_URL_HOST
          value: # db host
        - name: KC_DB_URL_PROPERTIES
          value: "?characterEncoding=UTF-8"
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "500m"
          limits:
            memory: "4Gi"
            cpu: "1000m"
        readinessProbe:
          httpGet:
            path: /realms/master
            port: 8080

---

apiVersion: v1
kind: Service
metadata:
  name: keycloak-headless
  labels:
    app: keycloak
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak

And the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "KC_SC"
    nginx.ingress.kubernetes.io/affinity-mode: "balanced"
spec:
  rules:
  - host: # host
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: keycloak-headless
            port:
              number: 8080