Convert JGROUPS config to Quarkus

Hello,
I’m trying to convert my jboss configuration from keycloak 16 to the new quarkus configuration with keycloak 17.

I’m using a HA-Cluster with JGROUPS Unicast TCP Mechanism secured with AES Encryption.
But I have no Idea how this configuration should look like with quarkus.

I already know that you can provide an xml containing the infinispan configuration. But I have no clue how this config file should be formatted and what options I do have.

This is the current configuration (extracted from standalone-ha.xml)

        <subsystem xmlns="urn:jboss:domain:jgroups:8.0">
            <channels default="ee">
                <channel name="ee" stack="unicast-tcp" cluster="ejb"/>
            </channels>
            <stacks>
                <stack name="unicast-tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp">
                        <property name="bind_port">${env.JGROUPS_PORT:7800}</property>
                        <property name="external_addr">${env.JGROUPS_EXTERNAL_IP:127.0.0.1}</property>
                    </transport>
                    <protocol type="org.jgroups.protocols.TCPPING">
                        <property name="initial_hosts">${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}</property>
                        <property name="port_range">0</property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="org.jgroups.protocols.SYM_ENCRYPT">
                        <property name="provider">SunJCE</property>
                        <property name="sym_algorithm">AES</property>
                        <property name="encrypt_entire_message">true</property>
                        <property name="keystore_name">${env.JGROUPS_KEYSTORE:/etc/secrets/jgroups.jceks}</property>
                        <property name="alias">${env.JGROUPS_KEYSTORE_ALIAS:jgroups}</property>
                        <property name="store_password">${env.JGROUPS_JKS_PASSWORD:changeme}</property>
                    </protocol>
                    <protocol type="pbcast.NAKACK2">
                        <property name="use_mcast_xmit">false</property>
                    </protocol>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="org.jgroups.protocols.AUTH">
                        <property name="auth_class">org.jgroups.auth.MD5Token</property>
                        <property name="token_hash">SHA</property>
                        <property name="auth_value">${env.JGROUPS_AUTH_SECRET:changeme2}</property>
                    </protocol>
                    <protocol type="pbcast.GMS">
                        <property name="print_physical_addrs">true</property>
                        <property name="print_local_addr">true</property>
                    </protocol>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG2"/>
                    <protocol type="RSVP"/>
                </stack>
            </stacks>
        </subsystem>

Hello,

I am kinda having the same issue. I managed to get the clustering to work, based on my understand, the embedded ISPN will read this file by default infinispan/default-jgroups-udp.xml at main · infinispan/infinispan · GitHub " which is bundled inside the infinispan jar file " once the shared caching option has been enabled in Keycloak, my question here, how can I override some of those configurations, like for example switch the local listen port to be 22222 instead of 0 " which is random", I did try to add the section from the mentioned file into conf/cache-ispn.xml " same idea as in here Keycloak 17 example using JGroups JDBC_PING discovery protocol for Infinispan · GitHub " however, those configurations are being ignored, what would be the correct way to override those settings?

Regards,
Sherif

Put your <stack>...</stack> section inside the <infinispan><jgroups>... element. Here is an example of doing it for JDBC_PING: Keycloak 17 example using JGroups JDBC_PING discovery protocol for Infinispan · GitHub

That what I did :slight_smile: I did read your link and that what I have followed, I have the following inside <infinispan>

  <jgroups>
   <stack name="jdbc-udp" extends="udp">
     <UDP bind_addr="${jgroups.bind.address,jgroups.udp.address:SITE_LOCAL}"
        bind_port="${jgroups.bind.port,jgroups.udp.port:55200}"
        mcast_addr="${jgroups.mcast_addr:230.0.0.4}"
        mcast_port="${jgroups.mcast_port:45688}"
        tos="0"
        ucast_send_buf_size="1m"
        mcast_send_buf_size="1m"
        ucast_recv_buf_size="20m"
        mcast_recv_buf_size="25m"
        ip_ttl="${jgroups.ip_ttl:2}"
        thread_naming_pattern="pl"
        enable_diagnostics="false"
        bundler_type="transfer-queue"
        max_bundle_size="8500"

        thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
        thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
        thread_pool.keep_alive_time="60000"

        thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}" />
   </stack>
  </jgroups>

not sure if the that is correct or not…

Ok, so I ended up with the following content in /opt/keycloak/current/conf/jgroups.xml and I set KC_CACHE_CONFIG_FILE=jgroups.xml as environment varaible.
But in the startup log I get the error:

Apr 13 10:47:32 SETEST16 kc.sh[124016]: 2022-04-13 10:47:32,890 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.configuration.io.ConfigurationReaderException: Unexpected element 'property' encountered[11,32]
Apr 13 10:47:32 SETEST16 kc.sh[124016]: 2022-04-13 10:47:32,891 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unexpected element 'property' encountered[11,32]
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
    xmlns="urn:infinispan:config:11.0">

	<!-- custom stack goes into the jgroups element -->
	<jgroups>
		<stack name="unicast-tcp">
			<transport type="TCP" socket-binding="jgroups-tcp">
				<property name="bind_port">${env.JGROUPS_PORT:7800}</property>
				<property name="external_addr">${env.JGROUPS_EXTERNAL_IP:127.0.0.1}</property>
			</transport>
			<protocol type="org.jgroups.protocols.TCPPING">
				<property name="initial_hosts">${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}</property>
				<property name="port_range">0</property>
			</protocol>
			<protocol type="MERGE3"/>
			<protocol type="FD_SOCK"/>
			<protocol type="FD_ALL"/>
			<protocol type="VERIFY_SUSPECT"/>
			<protocol type="org.jgroups.protocols.SYM_ENCRYPT">
				<property name="provider">SunJCE</property>
				<property name="sym_algorithm">AES</property>
				<property name="encrypt_entire_message">true</property>
				<property name="keystore_name">${env.JGROUPS_KEYSTORE:/etc/secrets/jgroups.jceks}</property>
				<property name="alias">${env.JGROUPS_KEYSTORE_ALIAS:jgroups}</property>
				<property name="store_password">${env.JGROUPS_JKS_PASSWORD:changeme}</property>
			</protocol>
			<protocol type="pbcast.NAKACK2">
				<property name="use_mcast_xmit">false</property>
			</protocol>
			<protocol type="UNICAST3"/>
			<protocol type="pbcast.STABLE"/>
			<protocol type="org.jgroups.protocols.AUTH">
				<property name="auth_class">org.jgroups.auth.MD5Token</property>
				<property name="token_hash">SHA</property>
				<property name="auth_value">${env.JGROUPS_AUTH_SECRET:changeme2}</property>
			</protocol>
			<protocol type="pbcast.GMS">
				<property name="print_physical_addrs">true</property>
				<property name="print_local_addr">true</property>
			</protocol>
			<protocol type="UFC"/>
			<protocol type="MFC"/>
			<protocol type="FRAG2"/>
			<protocol type="RSVP"/>
		</stack>
	</jgroups>
	<cache-container name="keycloak">
		<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
		<transport lock-timeout="60000" stack="unicast-tcp"/>
		<local-cache name="realms">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<local-cache name="users">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<distributed-cache name="sessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="authenticationSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="offlineSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="clientSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="offlineClientSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="loginFailures" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<local-cache name="authorization">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<replicated-cache name="work">
			<expiration lifespan="-1"/>
		</replicated-cache>
		<local-cache name="keys">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<expiration max-idle="3600000"/>
			<memory max-count="1000"/>
		</local-cache>
		<distributed-cache name="actionTokens" owners="2">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<expiration max-idle="-1" lifespan="-1" interval="300000"/>
			<memory max-count="-1"/>
		</distributed-cache>
	</cache-container>
</infinispan>

Oh, I guess I knew what I was doing wrong, I missed adding the new stack name into the container line below this section

Update:
I tried a slightly different config format and now possibly ran into an infinispan bug? ( [ISPN-13765] Unable to start full JGroups stacks without parent - Red Hat Issue Tracker )

Config:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:11.0 jgroups.xsd"
    xmlns="urn:infinispan:config:11.0">

	<!-- custom stack goes into the jgroups element -->
	<jgroups>
		<stack name="unicast-tcp">
            <RSVP />
            <FRAG2 />
            <MFC />
            <UFC />
            <pbcast.GMS print_physical_addrs="true" print_local_addr="true"/>
            <AUTH auth_class="org.jgroups.auth.MD5Token" token_hash="SHA" auth_value="changeme2"/>
            <pbcast.STABLE />
            <UNICAST3 />
            <pbcast.NAKACK2 use_mcast_xmit="false"/>
            <SYM_ENCRYPT provider="SunJCE" sym_algorithm="AES" encrypt_entire_message="true" keystore_name="/etc/secrets/jgroups/jceks" alias="jgroups" store_password="changeme"/>
            <VERIFY_SUSPECT />
            <FD_ALL />
            <FD_SOCK />
            <MERGE3 />
            <TCPPING initial_hosts="${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}"/>
            <TCP bind_port="7800" bind_addr="SITE_LOCAL"/>
		</stack>
	</jgroups>
	<cache-container name="keycloak">
		<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
		<transport lock-timeout="60000" stack="unicast-tcp"/>
		<local-cache name="realms">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<local-cache name="users">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<distributed-cache name="sessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="authenticationSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="offlineSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="clientSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="offlineClientSessions" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<distributed-cache name="loginFailures" owners="2">
			<expiration lifespan="-1"/>
		</distributed-cache>
		<local-cache name="authorization">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<memory max-count="10000"/>
		</local-cache>
		<replicated-cache name="work">
			<expiration lifespan="-1"/>
		</replicated-cache>
		<local-cache name="keys">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<expiration max-idle="3600000"/>
			<memory max-count="1000"/>
		</local-cache>
		<distributed-cache name="actionTokens" owners="2">
			<encoding>
				<key media-type="application/x-java-object"/>
				<value media-type="application/x-java-object"/>
			</encoding>
			<expiration max-idle="-1" lifespan="-1" interval="300000"/>
			<memory max-count="-1"/>
		</distributed-cache>
	</cache-container>
</infinispan>

Log-Output:

ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start caches
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.CacheConfigurationException: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
1 Like