Preserve login session over KeyCloak restart with Infinispan local file-based cache

Issue

I’m running a single KeyCloak instance in a docker container (Quay). When KeyCloak is restarted all users have to login again. I want to preserve the logins over KeyCloak restart.

My approach to a solution

Configure the included/embedded infinispan cache to use file-based persistence. The cache file is written to a location where a Docker volume has been mounted. A custom image is used to give write permission for this file location, because without it I got ‘Permission denied’ errors. With the configuration below KeyCloak starts fine and shows no errors, however logins are NOT preserved. Users still have to login again after restart. Did I miss something?

Docker compose

services:
    keycloak:
        image: keycloak-with-chmod:latest # based on quay.io/keycloak/keycloak:19.0.3
        command: >-
            start
            --db=postgres
            --http-relative-path=/kc
            --cache=ispn --cache-config-file=cache-ispn.xml
            --proxy=edge
        volumes:
            - infinispan_data:/opt/keycloak/cache
            - ./cache-ispn.xml:/opt/keycloak/conf/cache-ispn.xml:ro
volumes:
    infinispan_data:
        driver: local

Custom docker image

FROM quay.io/keycloak/keycloak:19.0.3
RUN mkdir /opt/keycloak/cache
RUN chmod 777 /opt/keycloak/cache

cache-ispn.xml

<infinispan xmlns="urn:infinispan:config:11.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd">

    <cache-container name="keycloak">
        <transport lock-timeout="60000"/>

        <global-state>
            <persistent-location path="/opt/keycloak/cache"/>
        </global-state>

        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="sessions">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <file-store path="/opt/keycloak/cache" shared="false" purge="false"/>
            </persistence>
        </local-cache>
        <local-cache name="clientSessions">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <file-store path="/opt/keycloak/cache" shared="false" purge="false"/>
            </persistence>
        </local-cache>
        <local-cache name="authenticationSessions">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <file-store path="/opt/keycloak/cache" shared="false" purge="false"/>
            </persistence>
        </local-cache>
        <local-cache name="offlineSessions">
            <expiration lifespan="-1"/>
        </local-cache>
        <local-cache name="offlineClientSessions">
            <expiration lifespan="-1"/>
        </local-cache>
        <local-cache name="loginFailures">
            <expiration lifespan="-1"/>
        </local-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <local-cache name="actionTokens">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </local-cache>
    </cache-container>
</infinispan>
1 Like

First guess: --cache-config-file= is not a runtime param anymore, it needs to be given during build, so you need to rebuild your container accordingly.

I also can’t manage to make sessions persistent and survive service restart.
Even with building again.

@pte did you have any luck with it?

Giannis

The problem seems to be

passivation="false"

https://groups.google.com/g/keycloak-dev/c/sOBzG76f2FE

Managed to make it work like this:

        <local-cache name="sessions">
            <expiration lifespan="-1"/>
            <memory max-count="10000"/>
            <persistence passivation="true">
                <file-store shared="false" preload="true" purge="false" fetch-state="true"/>
            </persistence>
        </local-cache>
1 Like

@bilias You’re right, with

passivation="true"

it also works on my side! Thanks!

@bpedersen2 It is indeed a parameter for the ‘kc.sh build’ command. But I noticed that the container image actually re-runs the build with this parameter when running non-optimized. I’m gonna take this into account when creating an optimized build. Thanks!

After reading your post even i tried to configure keycloak’s infinispan cache in external volume (File Share)… when i start the server with 1 replicas in Kubernetes it is working fine but when i spin up another replica for High Availability and Zero Downtime i can see application is failing to start with below error

2022-12-08 11:55:36,762 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-12-08 11:55:37,112 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final
2022-12-08 11:55:37,124 WARN  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
2022-12-08 11:55:37,343 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2022-12-08 11:55:37,343 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start caches
2022-12-08 11:55:37,343 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/keycloak/cache/___global.lck' for persistent global state
2022-12-08 11:55:37,343 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/keycloak/cache/___global.lck' for persistent global state
2022-12-08 11:55:37,344 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ISPN000512: Cannot acquire lock '/opt/keycloak/cache/___global.lck' for persistent global state
2022-12-08 11:55:37,344 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: /opt/keycloak/cache/___global.lck (No such file or directory)
2022-12-08 11:55:37,344 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

Can you please let me know if you are able to run keycloak with multiple replicas

I would probably go with the recommended setup of external infinispan (cluster).
I’m only running a single instance of keycloak (as the OP does).