Readiness probe failed: Get "http://x.x.x.x:8080/auth/realms/master": dial tcp x.x.x.x:8080: connect: connection refused

Configuring the server
#1

I am attempting to install 15.0.2 in minikube. Everything loads fine (minikube), but the container fails to start as I get the following issue which I can not get past:

Name:         keycloak-8556878c5b-bq6w6
Namespace:    default
Priority:     0
Node:         keycloke.devXX.domain.org/XXX.XXX.XXX.XXX
Start Time:   Tue, 08 Feb 2022 09:30:36 -0800
Labels:       app=keycloak
              pod-template-hash=8556878c5b
Annotations:  <none>
Status:       Running
IP:           XXX.XXX.XXX.XXX
IPs:
  IP:           XXX.XXX.XXX.XXX
Controlled By:  ReplicaSet/keycloak-8556878c5b
Containers:
  keycloak:
    Container ID:   docker://839fe523ada2711e79f3b707058bf5f9e9a802bfcad282b66c97d937470248fc
    Image:          quay.io/keycloak/keycloak:15.0.2
    Image ID:       docker-pullable://quay.io/keycloak/keycloak@sha256:64fb81886fde61dee55091e6033481fa5ccdac62ae30a4fd29b54eb5e97df6a9
    Ports:          8080/TCP, 8443/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 08 Feb 2022 09:32:30 -0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 08 Feb 2022 09:31:38 -0800
      Finished:     Tue, 08 Feb 2022 09:32:00 -0800
    Ready:          False
    Restart Count:  3
    Readiness:      http-get http://:8080/auth/realms/master delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DB_VENDOR:                 POSTGRES
      DB_ADDR:                   XXX.XXX.XXX.XXX
      DB_DATABASE:               keycloak
      DB_USER:                   KEYCLOAK_USER
      DB_PASSWORD:              KEYCLOAK_PASSWORD
      KEYCLOAK_USER:             KEYCLOAK_USER
      KEYCLOAK_PASSWORD:         KEYCLOAK_PASSWORD
      PROXY_ADDRESS_FORWARDING:  false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-642c8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-642c8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m4s                 default-scheduler  Successfully assigned default/keycloak-8556878c5b-bq6w6 to keycloke.dev30.apollo.e911.io
  Normal   Pulled     62s (x3 over 2m3s)   kubelet            Container image "quay.io/keycloak/keycloak:15.0.2" already present on machine
  Normal   Created    62s (x3 over 2m3s)   kubelet            Created container keycloak
  Normal   Started    62s (x3 over 2m3s)   kubelet            Started container keycloak
  Warning  Unhealthy  54s (x11 over 2m2s)  kubelet            Readiness probe failed: Get "http://XXX.XXX.XXX.XXX:8080/auth/realms/master": dial tcp XXX.XXX.XXX.XXX:8080: connect: connection refused
  Warning  Unhealthy  43s (x2 over 103s)   kubelet            Readiness probe failed: Get "http://XXX.XXX.XXX.XXX:8080/auth/realms/master": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    24s (x3 over 74s)    kubelet            Back-off restarting failed container

My deployment.yaml file is this:

apiVersion: v1
kind: Service
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: default
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:15.0.2
        env:
        - name: DB_VENDOR
          value: POSTGRES
        - name: DB_ADDR
          value: XXX.XXX.XXX.XXX
        - name: DB_DATABASE
          value: KEYCLOAK_USER
        - name: DB_USER
          value: KEYCLOAK_PASSWORD
        - name: DB_PASSWORD
          value: KEYCLOAK_PASSWORD
        - name: KEYCLOAK_USER
          value: "KEYCLOAK_USER"
        - name: KEYCLOAK_PASSWORD
          value: "KEYCLOAK_PASSWORD"
        - name: PROXY_ADDRESS_FORWARDING
          value: "false"
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        readinessProbe:
          httpGet:
            path: /auth/realms/master
            port: 8080

Based on all documentation I could find, I am doing everything right. Help is greatly appreciated!

#2

You need to give Keycloak some time to load and start responding in port 8080, even more on the first run, because of database creation.

This is done by adjusting readinessProbe and livenessProbe (and startupProbe). You can experiment with a 120 seconds delay with

readinessProbe:
  httpGet:
    path: /auth/realms/master
    port: 8080
  initialDelaySeconds: 120

The proper way to do that are startup probes, but it is generally available on Kubernetes 1.20+.

There is a nice documentation about startupProbes at Kubernetes Startup Probes - Examples & Common Pitfalls

#3

An update to everyone that might land here:
This issue was caused by a few factors!

  1. My database had not been allowed access outside of localhost. As keycloak was being deployed on a standalone system, this caused it to be unable to connect to it.
  2. I was not using the community repo files. My company hosts their own secure repo, which evidently did not have all the proper RPM’s needed. After adding back in the community repo files and re-deploying minikube and Keycloak, I was able to get the application to start up properly!
1 Like