Failed to verify identity token

Configuring the server
1

Hello! I was trying to use keycloak 20 cluster configuration with 3 pods in kubernetes.

My config:

       - name: JAVA_OPTS
          value: -Xms128m -Xmx384m -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256m
            -Djava.net.preferIPv4Stack=true
        - name: JAVA_OPTS_APPEND
          value: -Djboss.as.management.blocking.timeout=300 -Djgroups.dns.query=keycloak-headless
        - name: JDBC_PARAMS
          value: socketTimeout=0&connectTimeout=10000&autoReconnect=true
        - name: KC_CACHE
          value: ispn
        - name: KC_CACHE_STACK
          value: kubernetes
        - name: KC_DB_ADDR
          value: mysqlendpoint:3306
        - name: KC_DB_DATABASE
          value: keycloak
        - name: KC_DB_POOL_MAX_SIZE
          value: "15"
        - name: KC_DB_POOL_MIN_SIZE
          value: "5"
        - name: KC_DB_USER
          value: keycloak
        - name: KC_DB_VENDOR
          value: mariadb
        - name: KC_HEALTH_ENABLED
          value: "true"
        - name: KC_HOSTNAME_STRICT
          value: "false"
        - name: KC_HOSTNAME_STRICT_BACKCHANNEL
          value: "true"
        - name: KC_HOSTNAME_STRICT_HTTPS
          value: "false"
        - name: KC_HTTP_ENABLED
          value: "true"
        - name: KC_HTTP_RELATIVE_PATH
          value: /auth
        - name: KC_LOG_LEVEL
          value: DEBUG
        - name: KC_METRICS_ENABLED
          value: "true"
        - name: KC_PROXY
          value: edge

kubectl get svc -n keycloak
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
keycloak            ClusterIP   10.16.8.228   <none>        8080/TCP,7800/TCP   25h
keycloak-headless   ClusterIP   None          <none>        8080/TCP,7800/TCP   25h

I then try to check list of users by following these instructions Using OpenID Connect (OIDC) and Keycloak to Centralize Authorization - Quarkus

The application is using bearer token authorization and the first thing to do is obtain an access token from the Keycloak Server in order to access the application resources:

export access_token=$(\
curl --insecure -X POST https://localhost:8543/realms/quarkus/protocol/openid-connect/token \
-H 'Accept: application/json' \ 
-H 'Content-Type: application/x-www-form-urlencoded' \ 
-H 'cache-control: no-cache' \ 
-d 'grant_type=password&username=admin&password=strongpass&client_id=admin-cli' \
| jq -r '.access_token' \
)

 curl -v -X GET \
   http://localhost:8080/api/admin \
   -H "Authorization: Bearer "$access_token

I get

{
  "error": "HTTP 401 Unauthorized"
}

In logs i find

2023-04-12 21:52:45,834 DEBUG [io.quarkus.vertx.http.runtime.ForwardedParser] (vert.x-eventloop-thread-1) Recalculated absoluteURI to https://keycloak-instance.com/auth/admin/realms/cluster-realm/users
2023-04-12 21:52:45,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-3) new JtaTransactionWrapper
2023-04-12 21:52:45,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-3) was existing? false
2023-04-12 21:52:45,836 DEBUG [org.keycloak.services.managers.AuthenticationManager] (executor-thread-3) Failed to verify identity token: Key not found
2023-04-12 21:52:45,836 DEBUG [org.keycloak.services.resources.admin.AdminRoot] (executor-thread-3) Token not valid
2023-04-12 21:52:45,837 DEBUG [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-3) Error response 401: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized

Why my token is invalid? Distributed (ISPN) cache seems works fine:

2023-04-12 22:19:08,750 INFO  [org.infinispan.LIFECYCLE] (jgroups-10,keycloak-0-7341) [Context=offlineClientSessions] ISPN100002: Starting rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], phase READ_OLD_WRITE_ALL, topology id 78
2023-04-12 22:19:08,764 INFO  [org.infinispan.LIFECYCLE] (jgroups-11,keycloak-0-7341) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], topology id 78
2023-04-12 22:19:08,848 INFO  [org.keycloak.quarkus.runtime.storage.legacy.liquibase.QuarkusJpaUpdaterProvider] (main) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
2023-04-12 22:19:08,854 INFO  [org.infinispan.LIFECYCLE] (jgroups-11,keycloak-0-7341) [Context=offlineSessions] ISPN100002: Starting rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], phase READ_OLD_WRITE_ALL, topology id 78
2023-04-12 22:19:08,863 INFO  [org.infinispan.LIFECYCLE] (jgroups-10,keycloak-0-7341) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], topology id 78
2023-04-12 22:19:08,977 INFO  [org.infinispan.LIFECYCLE] (jgroups-10,keycloak-0-7341) [Context=sessions] ISPN100002: Starting rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], phase READ_OLD_WRITE_ALL, topology id 78
2023-04-12 22:19:09,031 INFO  [org.infinispan.LIFECYCLE] (jgroups-10,keycloak-0-7341) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], topology id 78
2023-04-12 22:19:09,139 INFO  [org.infinispan.LIFECYCLE] (jgroups-11,keycloak-0-7341) [Context=work] ISPN100002: Starting rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], phase READ_OLD_WRITE_ALL, topology id 74
2023-04-12 22:19:09,145 INFO  [org.infinispan.LIFECYCLE] (non-blocking-thread--p2-t3) [Context=work] ISPN100010: Finished rebalance with members [keycloak-2-34507, keycloak-1-32332, keycloak-0-7341], topology id 74
2023-04-12 22:19:11,541 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-0-7341, Site name: null
2 
export access_token=$(\
    curl --insecure -X POST https://localhost:8543/realms/quarkus/protocol/openid-connect/token \
    --user backend-service:secret \
    -H 'content-type: application/x-www-form-urlencoded' \
    -d 'username=alice&password=alice&grant_type=password' | jq --raw-output '.access_token' \
 )

And here are the logs for token issuance, the token itself seems to be valid

2023-04-12 22:35:21,036 DEBUG [io.quarkus.vertx.http.runtime.ForwardedParser] (vert.x-eventloop-thread-1) Recalculated absoluteURI to https://keycloak-instance.com/auth/realms/master/protocol/openid-connect/token
2023-04-12 22:35:21,037 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) new JtaTransactionWrapper
2023-04-12 22:35:21,037 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) was existing? false
2023-04-12 22:35:21,040 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) new JtaTransactionWrapper
2023-04-12 22:35:21,040 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) was existing? true
2023-04-12 22:35:21,041 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (executor-thread-1) AUTHENTICATE CLIENT
2023-04-12 22:35:21,041 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (executor-thread-1) client authenticator: client-secret
2023-04-12 22:35:21,041 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (executor-thread-1) client authenticator SUCCESS: client-secret
2023-04-12 22:35:21,041 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (executor-thread-1) Client admin-cli authenticated by client-secret
2023-04-12 22:35:21,042 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (executor-thread-1) AUTHENTICATE ONLY
2023-04-12 22:35:21,042 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) processFlow: direct grant
2023-04-12 22:35:21,042 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) check execution: 'direct-grant-validate-username', requirement: 'REQUIRED'
2023-04-12 22:35:21,042 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) authenticator: direct-grant-validate-username
2023-04-12 22:35:21,042 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (executor-thread-1) Going through the flow 'direct grant' for adding executions
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (executor-thread-1) Selections when trying execution 'direct-grant-validate-username' : [ authSelection - direct-grant-validate-username]
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) invoke authenticator.authenticate: direct-grant-validate-username
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) authenticator SUCCESS: direct-grant-validate-username
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) check execution: 'direct-grant-validate-password', requirement: 'REQUIRED'
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) authenticator: direct-grant-validate-password
2023-04-12 22:35:21,043 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (executor-thread-1) Selections when trying execution 'direct-grant-validate-password' : [ authSelection - direct-grant-validate-password]
2023-04-12 22:35:21,044 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) invoke authenticator.authenticate: direct-grant-validate-password
2023-04-12 22:35:21,080 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) authenticator SUCCESS: direct-grant-validate-password
2023-04-12 22:35:21,080 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-1) Authentication successful of the top flow 'direct grant'
2023-04-12 22:35:21,083 DEBUG [org.keycloak.services.managers.AuthenticationSessionManager] (executor-thread-1) Removing authSession 'a9779347-bd20-4a73-9812-79a289b72313'. Expire restart cookie: true
2023-04-12 22:35:21,088 DEBUG [org.keycloak.services.resources.Cors] (executor-thread-1) Added CORS headers to response
2023-04-12 22:35:21,088 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) JtaTransactionWrapper  commit
2023-04-12 22:35:21,088 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) JtaTransactionWrapper end
2023-04-12 22:35:21,088 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-1) JtaTransactionWrapper resuming suspended
2023-04-12 22:35:21,088 DEBUG [org.keycloak.events] (executor-thread-1) type=LOGIN, realmId=3adb9858-a673-4ef3-be97-588a10d22f46, clientId=admin-cli, userId=3c1caa36-8d45-4ef1-9735-42844432de85, ipAddress=10.30.4.5, auth_method=openid-connect, token_id=f0220c8d-e758-400c-acda-3f8d10408e7e, grant_type=password, refresh_token_type=Refresh, scope='email profile', refresh_token_id=0e368b41-e1ed-423b-8ded-3eb5c19aa016, client_auth_method=client-secret, username=admin, authSessionParentId=a9779347-bd20-4a73-9812-79a289b72313, authSessionTabId=EPHMcMZDyV
3

I dived into debugging the problem and realized that the cache is NOT WORKING after all. I tried to create a new realms cluster-realm through UI and then request it directly in the pods - the realm was created only in the 1st pod!

[
  {
    "id": "e2269ee3-bb92-4271-9fb4-a8857990ac00",
    "createdTimestamp": 1681374339352,
    "username": "user@two.com",
    "enabled": true,
    "totp": false,
    "emailVerified": false,
    "email": "user@two.com",
    "disableableCredentialTypes": [],
    "requiredActions": [],
    "notBefore": 0,
    "access": {
      "manageGroupMembership": true,
      "view": true,
      "mapRoles": true,
      "impersonate": true,
      "manage": true
    }
  }
]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1926  100  1846  100    80  15603    676 --:--:-- --:--:-- --:--:-- 16322
try on 10.220.1.36
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    28  100    28    0     0    975      0 --:--:-- --:--:-- --:--:--  1000
{
  "error": "Realm not found."
}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1929  100  1849  100    80  15765    682 --:--:-- --:--:-- --:--:-- 16487
try on 10.220.1.101
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    28  100    28    0     0   1179      0 --:--:-- --:--:-- --:--:--  1166
{
  "error": "Realm not found."
}

It’s very strange because pods logs with trace on org.jgroups.protocols looks fine

2023-04-13 11:28:01,804 DEBUG [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery requests to hosts [10.220.1.226:0, 10.220.1.36:0, 10.220.1.101:0] on ports [7800 .. 7800]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.226:7800
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.226:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: looping back message [keycloak-0-55692 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL, transient_flags=DONT_LOOPBACK], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.36:7800
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.36:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.101:7800
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received [keycloak-0-55692 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL, transient_flags=DONT_LOOPBACK], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_REQ from keycloak-0-55692, sending response keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-0-55692, src=keycloak-0-55692, size=111, headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: looping back message [keycloak-0-55692 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,806 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-0-55692 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,806 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-0-55692: keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-2-24966, physical_addr=10.220.1.101:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-2-24966: keycloak-2-24966, name=keycloak-2-24966, addr=10.220.1.101:7800, coord
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-1-48535, physical_addr=10.220.1.36:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-1-48535: keycloak-1-48535, name=keycloak-1-48535, addr=10.220.1.36:7800, server
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.36:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:02,676 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_REQ from keycloak-2-24966, sending response keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=111, headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:02,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-2-24966, physical_addr=10.220.1.101:7800, TP: [cluster=ISPN]
2023-04-13 11:28:03,388 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:03,388 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:04,015 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:04,015 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:04,646 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending stable msg to keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:04,646 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=143, headers are STABLE: [STABLE_GOSSIP] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:04,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:04,677 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:05,389 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:05,389 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:06,016 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:06,016 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:06,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:06,678 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:07,389 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:07,390 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:08,016 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:08,016 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:08,566 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 65 bytes, flags=OOB|NO_RELIABILITY|INTERNAL], headers are STABLE: [STABILITY] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stability msg from keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stable digest keycloak-2-24966: [446 (446)], keycloak-1-48535: [276 (276)], keycloak-0-55692: [184 (184)]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 446 from keycloak-2-24966
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 276 from keycloak-1-48535
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 184 from keycloak-0-55692
2023-04-13 11:28:09,390 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:09,390 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:10,017 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:10,017 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:10,678 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:10,678 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:11,391 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:11,391 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:11,863 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending stable msg to keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:11,863 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=143, headers are STABLE: [STABLE_GOSSIP] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:12,017 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:12,018 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:12,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:12,679 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:13,391 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:13,392 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:14,018 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:14,018 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 65 bytes, flags=OOB|NO_RELIABILITY|INTERNAL], headers are STABLE: [STABILITY] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stability msg from keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stable digest keycloak-2-24966: [446 (446)], keycloak-1-48535: [276 (276)], keycloak-0-55692: [184 (184)]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 446 from keycloak-2-24966
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 276 from keycloak-1-48535
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 184 from keycloak-0-55692
2023-04-13 11:28:15,392 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:15,392 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:16,018 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:16,019 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:16,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:16,679 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:17,393 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:17,393 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:18,019 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:18,019 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:18,680 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:18,680 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:19,394 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:19,394 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
4

After starting the pod, I found this line

2023-04-13 15:06:26,905 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled

Maybe that’s exactly the problem with the cache.