Keycloak Cross Datacenter Replication with external Infinispan server

Configuring the server
1

Hi, hopefully someone has experience of how to setup keycloak with
infinispan as a remote cache.

My goal is to run keycloak across multiple data centres with replicated
database and cache, however i am struggling to get anything working.

I am running keycloak 12.0.4 and infinispan 9.4.20

I have 2 unclustered keycloak instances (A and B), sharing a remote cache
and H2 backend.

When i log in to the admin console on one of the instances (lets say A),
through the shared remote store i’m expecting to see the session on the
other keycloak instance (B). Using the infinispan manager console i can
see that the sessions are being cached as the cache content entries value
increments, from both directions (A + B), however, the values never appear
within the other instance.

Here are snippets of my clustered.xml and standalone-ha.xml

clustered.xml

<replicated-cache-configuration name=“sessions-cfg” mode=“SYNC”
start=“EAGER” batching=“false”>
<transaction mode=“NON_XA” locking=“PESSIMISTIC”/>
</replicated-cache-configuration>
<transaction mode=“NON_XA” locking=“PESSIMISTIC”/>
<backups>
<backup site=“site1” failure-policy=“FAIL” strategy=“SYNC” enabled=“true”/>
</backups>
<replicated-cache name=“work” configuration=“sessions-cfg” />
<replicated-cache name=“sessions” configuration=“sessions-cfg” />
<replicated-cache name=“clientSessions” configuration=“sessions-cfg”/>
<replicated-cache name=“offlineSessions” configuration=“sessions-cfg” />
<replicated-cache name=“offlineClientSessions”
configuration=“sessions-cfg”/>
<replicated-cache name=“actionTokens” configuration=“sessions-cfg” />

<replicated-cache name=“loginFailures” configuration=“sessions-cfg” />

standalone-ha.xml

<replicated-cache name=“work” mode=“SYNC”>
<remote-store cache=“work” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</replicated-cache>

<distributed-cache name=“sessions” mode=“SYNC” owners=“1”>
<remote-store cache=“sessions” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“offlineSessions” mode=“SYNC” owners=“1”>
<remote-store cache=“offlineSessions” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“clientSessions” mode=“SYNC” owners=“1”>
<remote-store cache=“clientSessions” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“offlineClientSessions” mode=“SYNC” owners=“1”>
<remote-store cache=“offlineClientSessions” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“loginFailures” mode=“SYNC” owners=“1”>
<remote-store cache=“loginFailures” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“false”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“actionTokens” mode=“SYNC” owners=“2”>
<eviction max-entries="-1" strategy=“NONE”/>
<expiration max-idle="-1" interval=“300000”/>
<remote-store cache=“actionTokens” remote-servers=“remote-cache”
passivation=“false” fetch-state=“false” purge=“false” preload=“true”
shared=“true”>
<property name=“rawValues”>true
<property
name=“marshaller”>org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory
</remote-store>
</distributed-cache>

<distributed-cache name=“authenticationSessions” mode=“SYNC” owners=“1”/>

Backend log from keycloak server
2021-04-19 20:24:46,516 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionListener] (Thread-0) Received event from remote store. Event ‘CLIENT_CACHE_ENTRY_MODIFIED’, key ‘30abf6ae-cdc6-4691-a6e7-2dff27198178’, skip ‘true’

I guess the “skip” parameter should be false according to official keycloak documentation .

any ideas on what i might be missing?

Thanks
Yogi

3

I don’t have experience with a cross dc deployment of keycloak, but I have read a bit about the architecture.
From what I understood the ‘Data Grid’ is responsible for infinispan replication between the clusters, so I guess something is wrong there.
You haven’t mentioned anything about the Data Grid so maybe you have missed it.

4

@Cyben Thanks for the response,

I have replicated the exact steps from Server Installation and Configuration Guide , hence the only issues i am facing is, if i login from 1 instance of keycloak , i should be able to see the session details on another keycloak instance with the help of external infinispan cluster, i can see the session count is perfectly incremented on the remote cache of both the infinispan cluster but for some reason the same is not reflecting on keycloak UI, Any help is much appreciated, i am struck for last two weeks :frowning:

5

Maybe it is just a bug from the UI, try to check out the sessions with an api(I think there is one for that but not sure).
The reason I’m saying that is because I have already faced some problems of that kind, for example keycloak ui didn’t show me all the users from the db.
(The bugs I faced are in an old keycloak version and I ‘abused’ kinda the keycloak and just then there was some bugs, and I don’t have any experience with the cross dc deployment yet so I can just suggest what else you could check)