Cannot login Administrator console when running Keycloak with Replicas >1 on Kubernetes

I deployed Keycloak on a Kubernetes cluster.
Keycloak will use an external MySQL Database.

The service points to the DB as below:

apiVersion: v1
kind: Service
metadata:
  name: external-mysql-db
  namespace: iam-keycloak
spec:
  clusterIP: None
  ports:
  - name: mysql
    protocol: TCP
    port: 3306
    targetPort: 3306
  selector: {}
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-mysql-db
  namespace: iam-keycloak
subsets:
  - addresses:
    - ip: "172.19.11.60"
    ports:
    - port: 3306
      name: mysql

When running Keycloak with replicas = 1, everything works smoothly.
But when I increased the replicas = 2 or more, I can not login to the Administrator console.

Here is the Keycloak deployment:

keycloak-deploy.yaml
apiVersion: v1
kind: Service
metadata:
  name: keycloak-svc
  namespace: iam-keycloak
  labels:
    app: keycloak
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: iam-keycloak
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:12.0.4
        env:
        - name: KEYCLOAK_USER
          valueFrom:
            secretKeyRef:
              name: keycloak-secret
              key: keycloak_user
        - name: KEYCLOAK_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secret
              key: keycloak_pass
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        - name: DB_VENDOR
          value: mysql
        - name: DB_ADDR
          value: external-mysql-db
        - name: DB_DATABASE
          value: keycloak_db
        - name: DB_USER
          value: keycloak_user
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secret
              key: db_password
        - name: JGROUPS_DISCOVERY_PROTOCOL
          value: dns.DNS_PING
        - name: JGROUPS_DISCOVERY_PROPERTIES
          value: dns_query=keycloak-svc
        - name: CACHE_OWNERS_COUNT
          value: "2"
        - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
          value: "2"
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        readinessProbe:
          httpGet:
            path: /auth/realms/master
            port: 8080
          initialDelaySeconds: 10

Anyone has a solution for this issues, please.
Thank you.

Simply multiplying the count of Keycloak instances is not enough, you have to configure a cluster - a Keycloak/Wildfly cluster, not related to K8s:

1 Like

Thank you for your help.
Currently, I’m using DNS_PING

    - name: JGROUPS_DISCOVERY_PROTOCOL
      value: dns.DNS_PING
    - name: JGROUPS_DISCOVERY_PROPERTIES
      value: dns_query=keycloak-svc
    - name: CACHE_OWNERS_COUNT
      value: "2"
    - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
      value: "2"

And I have to add this annotation to Ingress:
nginx.ingress.kubernetes.io/affinity: "cookie"

So I can run multi-replicas pods of Keycloak.

@truongnh was this enough?

thanks :slight_smile:

@pierreozoux @dasniko Finally, I found the solution.

I have to insert the following script to the Keycloak Image: (keycloak.cli)

embed-server --server-config=standalone-ha.xml --std-out=echo
batch

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:1})
echo Configuring node identifier
/subsystem=transactions:write-attribute(name=node-identifier, value=${jboss.node.name})

run-batch
stop-embedded-server

The Dockerfile:

FROM quay.io/keycloak/keycloak:12.0.4
RUN mkdir /opt/jboss/startup-scripts/
COPY keycloak.cli /opt/jboss/startup-scripts/

The environment for Keycloak container:
If you deploy Keycloak with Replicas = 5

name: CACHE_OWNERS_AUTH_SESSIONS
value: "5"
name: JGROUPS_DISCOVERY_PROTOCOL
value: dns.DNS_PING
name: JGROUPS_DISCOVERY_PROPERTIES
value: 'dns_query=${keycloak-svc}.${namespace}.svc.cluster.local'
name: CACHE_OWNERS_COUNT
value: "5"
name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
value: "5"

For more detail:

2 Likes

Wow, super! Thanks a lot for sharing all these details :slight_smile:

Ok, it is maybe beyond the scope of this thread, but now I’m curious, why didn’t you use the officail operator?

Have a nice week!

Hi @truongnh I am wondering how to add the following script to keycloak image(cli) could you please tell me how to do that? really appreciate it, thanks! :slightly_smiling_face:

Hi @hafidz

I used Dockerfile to build the Keycloak Image.

For detailed:

Let create a folder then push the Dockerfile and keycloak.cli in it:

The content of Dockerfile:

FROM quay.io/keycloak/keycloak:12.0.4
RUN mkdir /opt/jboss/startup-scripts/
COPY keycloak.cli /opt/jboss/startup-scripts/

Change directory to the folder contains Dockerfile and keycloak.cli. In order to build a new Image, run the following command:

docker build -t newkeycloak .

Done:)

Hi @truongnh

Is this setup did not require kubernetes service to connecting jgroups/infinispan between each pods ?
what kind of PING did you use ?

Sorry for my late reply.
The JGROUPS_DISCOVERY_PROTOCOL is DNS_PING.