Persist infinispan cache

[Icatream]

Hi,
I’m running keycloak-18 with the guide.
I want to know, it’s possible to persist the keycloak’s infinispan cache.

[wb256]

I’m trying to figure this out too. According to this documentation it’s possible: Server Installation and Configuration Guide

However, the documentation talks about the command “standalone.sh”, which doesn’t exist in the current distribution so I’m stuck

[wb256]

I found way to persist infinispan caches by following this documentation instead: Configuring distributed caches - Keycloak.

My setup:

• 2 keycloak instances running on the same network
• Set environment variable KC_CACHE=ispn for both instances
• KC_CACHE_STACK=kubernetes might be necessary when running in kubernetes too

If everything works you should see these log messages:

• Infinispan cache persistence enabled: 2022-08-01 12:28:45,567 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
• Second node joined: 2022-08-01 12:28:58,664 INFO [org.infinispan.CLUSTER] (jgroups-8,4ed11815318d-60515) ISPN100000: Node fe02127deddf-59930 joined the cluster

This way both instances communicate with each other and synchronize sessions. You will be able to restart node 1 while node 2 keeps running and all sessions will stay after node 1 turns back on.

[Icatream]

Thanks wb256,
These can let the the keycloak instances sharing infinispan cache with each other.
But if all keycloak instance shutdown, the cache will be lost.

I found something here infinispan_configuring#write-behind_persistence

But seems, keycloak does not opening the cache persistence. I got Unable to persist Infinispan internal caches as no global state enabled at starting

[Icatream]

jGroups/Infinispan Bedevilling: No DB Table Created (Unable to persist Infinispan internal caches as no global state enabled)

@xgp Thank you! The Keycloak v18.0.1 runs on AWS EC2 Linux (no containers); load balanced on ALB (running on 1 node pending figuring out clustering). Please find the build, start, cache-ispn-jdbc-ping.xml, logs (INFO; with TRACE for org.jgroups.protocols). The DB Vendor is ‘oracle’. Very much appreciate your and @dasniko’s help. Thanks. 1- Build export KC_BUILD_OPTS="$KC_BUILD_OPTS --db=${KEYCLOAK_DB_VENDOR}" export KC_BUILD_OPTS="$KC_START_OPTS --http-relative-path=/auth/" export KC_BUI…

this discussion can help

[anand-kulk]

Yeah, I am also looking for the same…

[nishada]

I was able to create an external infinispan cluster and connect Keycloak to that using hotrod client.
I used this infinispan configuration file in keycloak (keycloak-project-example/cache-ispn-remote.xml at main · thomasdarimont/keycloak-project-example · GitHub)

Infinispan cluster was created on EC2 instances by using tcp and s3_ping (s3_ping is deprecated now).
Keycloak cluster was on EKS.

references

• keycloak-project-example/cache-ispn-remote.xml at main · thomasdarimont/keycloak-project-example · GitHub
• How to configure Infinispan for AWS S3_PING discovery | Connect2id

[anand-kulk]

@nishada, you didn’t consider spinning up infinispan in EKS as you already have Keycloak there too? any specific reason?

[Icatream]

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright 2019 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:13.0 http://www.infinispan.org/schemas/infinispan-config-13.0.xsd"
        xmlns="urn:infinispan:config:13.0">

    <jgroups>
        <stack name="jdbc-ping-tcp" extends="tcp">
            <JDBC_PING connection_driver="com.mysql.cj.jdbc.Driver"
                       connection_url="${env.KC_DB_URL}"
                       connection_username="${env.KC_DB_USERNAME}"
                       connection_password="${env.KC_DB_PASSWORD}"
                       initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data VARBINARY(255), constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
                       info_writer_sleep_time="500"
                       remove_all_data_on_view_change="true"
                       stack.combine="REPLACE"
                       stack.position="MPING"/>
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <global-state>
            <persistent-location path="ispn/"/>
        </global-state>
        <transport lock-timeout="60000" stack="jdbc-ping-tcp"/>

        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
<!--            <persistence>-->
<!--                <file-store WARNING="DO NOT persist local cache as file-store, it will cause error"/>-->
<!--            </persistence>-->
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
            <persistence>
                <file-store preload="true" fetch-state="true"/>
            </persistence>
        </distributed-cache>
    </cache-container>
</infinispan>

this will persist cache as file, due to global-stage

        <global-state>
            <persistent-location path="ispn/"/>
        </global-state>

it use jgroups jdbc-ping-tcp to communicate with each other