EncodingException: Keycloak to Infinispan

Hi
I have configured Infinispan Cluster version 14.0 with Keycloak version 21.0.2-debian-11-r0 on EKS Cluster.
When I bring up Keycloak, I am seeing below error:

org.infinispan.commons.dataconversion.EncodingException: ISPN000492: Cannot find transcoder between 'application/x-protostream' to 'application/x-jboss-marshalling'

Infinispan config:

apiVersion: v1
data:
  infinispan.yml: |-
    infinispan:
      cacheContainer:
        caches:
          actionTokens:
            distributedCache:
              encoding:
                key:
                  mediaType: application/x-jboss-marshalling
                value:
                  mediaType: application/x-jboss-marshalling
              expiration:
                interval: 300000
                max-idle: "-1"
              memory:
                max-count: "-1"
              mode: SYNC
              owners: 2
              statistics: "true"
          authenticationSessions:
            distributedCache:
              encoding:
                key:
                  mediaType: application/x-jboss-marshalling
                value:
                  mediaType: application/x-jboss-marshalling
              mode: SYNC
              owners: "2"
              statistics: "true"
          clientSessions:
            distributedCache:
              encoding:
                key:
                  mediaType: application/x-jboss-marshalling
                value:
                  mediaType: application/x-jboss-marshalling
              mode: SYNC
              owners: 2
              statistics: "true"
          distributed-cache-cfg:
            distributedCacheConfiguration:
              encoding:
                mediaType: application/x-jboss-marshalling
              expiration:
                lifespan: "900000000000000000"
          loginFailures:
            distributedCache:
              configuration: distributed-cache-cfg
              owners: 2
          offlineClientSessions:
            distributedCache:
              configuration: distributed-cache-cfg
              owners: 2
          offlineSessions:
            distributedCache:
              configuration: distributed-cache-cfg
              owners: 2
          realms:
            replicatedCache:
              encoding:
                key:
                  mediaType: application/x-jboss-marshalling
                value:
                  mediaType: application/x-jboss-marshalling
              locking:
                acquireTimeout: "15000"
                concurrencyLevel: "1000"
                isolation: REPEATABLE_READ
              mode: SYNC
              stateTransfer:
                timeout: "60000"
              statistics: "true"
          replicated-cache-cfg:
            replicatedCacheConfiguration:
              encoding:
                mediaType: application/x-jboss-marshalling
          sessions:
            distributedCache:
              configuration: distributed-cache-cfg
              owners: 2
          users:
            replicatedCache:
              encoding:
                key:
                  mediaType: application/x-jboss-marshalling
                value:
                  mediaType: application/x-jboss-marshalling
              locking:
                acquireTimeout: "15000"
                concurrencyLevel: "1000"
                isolation: REPEATABLE_READ
              mode: SYNC
              stateTransfer:
                timeout: "60000"
              statistics: "true"
          work:
            replicatedCache:
              configuration: replicated-cache-cfg
        name: default
        security:
          authorization: {}
        statistics: "true"
        transport:
          cluster: ${infinispan.cluster.name:cluster}
          node-name: ${infinispan.node.name:}
          stack: kubernetes
      server:
        endpoints:
        - connectors:
            hotrod:
              hotrodConnector: null
            rest:
              restConnector: null
          securityRealm: default
          socketBinding: default
        - connectors:
            rest:
              restConnector:
                authentication:
                  mechanisms: BASIC
          securityRealm: metrics
          socketBinding: metrics
        interfaces:
        - inetAddress:
            value: ${infinispan.bind.address:127.0.0.1}
          name: public
        security:
          credentialStores:
          - clearTextCredential:
              clearText: secret
            name: credentials
            path: credentials.pfx
          securityRealms:
          - name: default
            propertiesRealm:
              groupProperties:
                path: groups.properties
              groupsAttribute: Roles
              userProperties:
                path: users.properties
          - name: metrics
            propertiesRealm:
              groupProperties:
                path: metrics-groups.properties
                relativeTo: infinispan.server.config.path
              groupsAttribute: Roles
              userProperties:
                path: metrics-users.properties
                relativeTo: infinispan.server.config.path
        socketBindings:
          defaultInterface: public
          portOffset: ${infinispan.socket.binding.port-offset:0}
          socketBinding:
          - name: default
            port: 11222
          - name: metrics
            port: 11223
  log4j2.xml: |-
    <Configuration name="InfinispanServerConfig" monitorInterval="60" shutdownHook="disable">
      <Properties>
        <Property name="path">${sys:infinispan.server.log.path}</Property>
        <Property name="accessLogPattern">%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] &quot;%X{method} %m %X{protocol}&quot; %X{status} %X{requestSize} %X{responseSize} %X{duration}%n</Property>
      </Properties>
      <Appenders>
        <!-- Colored output on the console -->
        <Console name="STDOUT">
          <PatternLayout pattern="%highlight{%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable}{INFO=normal, DEBUG=normal, TRACE=normal}%n"/>
        </Console>

        <!-- Rolling file -->
        <RollingFile name="FILE" createOnDemand="true"
                     fileName="${path}/server.log"
                     filePattern="${path}/server.log.%d{yyyy-MM-dd}-%i">
          <Policies>
            <OnStartupTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="100 MB" />
            <TimeBasedTriggeringPolicy />
          </Policies>
          <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable%n"/>
        </RollingFile>

        <!-- Rolling file -->
        <RollingFile name="AUDIT-FILE" createOnDemand="true"
                     fileName="${path}/audit.log"
                     filePattern="${path}/audit.log.%d{yyyy-MM-dd}-%i">
          <Policies>
            <OnStartupTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="100 MB" />
            <TimeBasedTriggeringPolicy />
          </Policies>
          <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %m%n"/>
        </RollingFile>

        <!-- Rolling JSON file, disabled by default -->
        <RollingFile name="JSON-FILE" createOnDemand="true"
                     fileName="${path}/server.log.json"
                     filePattern="${path}/server.log.json.%d{yyyy-MM-dd}-%i">
          <Policies>
            <OnStartupTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="100 MB" />
            <TimeBasedTriggeringPolicy />
          </Policies>
          <JsonLayout compact="true" eventEol="true" stacktraceAsString="true">
            <KeyValuePair key="time" value="$${date:yyyy-MM-dd'T'HH:mm:ss.SSSZ}" />
          </JsonLayout>
        </RollingFile>

        <!-- Rolling HotRod access log, disabled by default -->
        <RollingFile name="HR-ACCESS-FILE" createOnDemand="true"
                     fileName="${path}/hotrod-access.log"
                     filePattern="${path}/hotrod-access.log.%i">
          <Policies>
            <SizeBasedTriggeringPolicy size="100 MB" />
          </Policies>
          <PatternLayout pattern="${accessLogPattern}"/>
        </RollingFile>
        <!-- Rolling REST access log, disabled by default -->
        <RollingFile name="REST-ACCESS-FILE" createOnDemand="true"
                     fileName="${path}/rest-access.log"
                     filePattern="${path}/rest-access.log.%i">
          <Policies>
            <SizeBasedTriggeringPolicy size="100 MB" />
          </Policies>
          <PatternLayout pattern="${accessLogPattern}"/>
        </RollingFile>
      </Appenders>

      <Loggers>
        <Root level="INFO">
          <AppenderRef ref="STDOUT"/>

          <!-- Uncomment just one of the two lines bellow to use alternatively JSON logging or plain-text logging to file-->
          <AppenderRef ref="FILE"/>
    <!--      <AppenderRef ref="JSON-FILE"/>-->
        </Root>

        <!-- Set to INFO to enable audit logging -->
        <Logger name="org.infinispan.AUDIT" additivity="false" level="ERROR">
          <AppenderRef ref="AUDIT-FILE"/>
        </Logger>

        <!-- Set to TRACE to enable access logging for Hot Rod requests -->
        <Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="INFO">
          <AppenderRef ref="HR-ACCESS-FILE"/>
        </Logger>

        <!-- Set to TRACE to enable access logging for REST requests -->
        <Logger name="org.infinispan.REST_ACCESS_LOG" additivity="false" level="INFO">
          <AppenderRef ref="REST-ACCESS-FILE"/>
        </Logger>
        <Logger name="com.arjuna" level="WARN" />
        <Logger name="io.netty.handler.ssl.ApplicationProtocolNegotiationHandler" level="ERROR" />

      </Loggers>
    </Configuration>

Content of ispn-cache.xml

    <?xml version="1.0" encoding="UTF-8"?>
    <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-13.0.xsd"
            xmlns="urn:infinispan:config:14.0">

        <cache-container name="keycloak">
            <transport lock-timeout="60000"/>

            <replicated-cache name="realms">
                <expiration lifespan="-1"/>
                <remote-store cache="realms" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </replicated-cache>

            <local-cache name="users">
                <encoding>
                    <key media-type="application/x-jboss-marshalling"/>
                    <value media-type="application/x-jboss-marshalling"/>
                </encoding>
                <memory max-count="10000"/>
            </local-cache>


            <local-cache name="keys">
                <encoding>
                    <key media-type="application/x-jboss-marshalling"/>
                    <value media-type="application/x-jboss-marshalling"/>
                </encoding>
                <expiration max-idle="3600000"/>
                <memory max-count="1000"/>
            </local-cache>

            <local-cache name="authorization">
                <encoding>
                    <key media-type="application/x-jboss-marshalling"/>
                    <value media-type="application/x-jboss-marshalling"/>
                </encoding>
                <memory max-count="10000"/>
            </local-cache>

            <distributed-cache name="sessions" owners="2">
                <expiration lifespan="-1"/>
                <remote-store cache="sessions" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
    <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
    <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
    <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>

            <distributed-cache name="authenticationSessions" owners="2">
                <expiration lifespan="-1"/>
            </distributed-cache>

            <distributed-cache name="offlineSessions" owners="2">
                <expiration lifespan="-1"/>
                <remote-store cache="offlineSessions" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>

            <distributed-cache name="clientSessions" owners="2">
                <expiration lifespan="-1"/>
                <remote-store cache="clientSessions" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>

            <distributed-cache name="offlineClientSessions" owners="2">
                <expiration lifespan="-1"/>
                <remote-store cache="offlineClientSessions" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>

            <distributed-cache name="loginFailures" owners="2">
                <expiration lifespan="-1"/>
                <remote-store cache="loginFailures" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>

            <replicated-cache name="work">
                <expiration lifespan="-1"/>
                <remote-store cache="work" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>

                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </replicated-cache>
            <distributed-cache name="actionTokens" owners="2">
                <encoding>
                    <key media-type="application/x-jboss-marshalling"/>
                    <value media-type="application/x-jboss-marshalling"/>
                </encoding>
                <expiration max-idle="-1" lifespan="-1" interval="300000"/>
                <memory max-count="-1"/>
                <remote-store cache="actionTokens" xmlns="urn:infinispan:config:store:remote:14.0"
                              purge="false"
                              preload="false"
                              shared="true" segmented="false"
                              connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                    <remote-server host="${env.INFINISPAN_SERVER}" port="${infinispan.bind.port:11222}"/>

                    <security>
                        <authentication>
                            <!--                        <plain username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"-->
                            <!--                               password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"-->
                            <!--                        />-->
                            <digest username="${env.KEYCLOAK_REMOTE_ISPN_USERNAME:keycloak}"
                                    password="${env.KEYCLOAK_REMOTE_ISPN_PASSWORD:password}"
                                    realm="default"/>
                        </authentication>
                    </security>
                    <property name="rawValues">true</property>
                    <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
                </remote-store>
            </distributed-cache>
        </cache-container>
    </infinispan>

What can be the issue here?

I could fix this issue by demoting the version to 20.0.5-debian-11-r13 from 21.0.2-debian-11-r3.

Any idea how can fix this issue in 21.0.X version?