klr
April 12, 2022, 11:49am
1
Hello,
I’m trying to convert my jboss configuration from keycloak 16 to the new quarkus configuration with keycloak 17.
I’m using a HA-Cluster with JGROUPS Unicast TCP Mechanism secured with AES Encryption.
But I have no Idea how this configuration should look like with quarkus.
I already know that you can provide an xml containing the infinispan configuration. But I have no clue how this config file should be formatted and what options I do have.
This is the current configuration (extracted from standalone-ha.xml)
<subsystem xmlns="urn:jboss:domain:jgroups:8.0">
<channels default="ee">
<channel name="ee" stack="unicast-tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="unicast-tcp">
<transport type="TCP" socket-binding="jgroups-tcp">
<property name="bind_port">${env.JGROUPS_PORT:7800}</property>
<property name="external_addr">${env.JGROUPS_EXTERNAL_IP:127.0.0.1}</property>
</transport>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="org.jgroups.protocols.SYM_ENCRYPT">
<property name="provider">SunJCE</property>
<property name="sym_algorithm">AES</property>
<property name="encrypt_entire_message">true</property>
<property name="keystore_name">${env.JGROUPS_KEYSTORE:/etc/secrets/jgroups.jceks}</property>
<property name="alias">${env.JGROUPS_KEYSTORE_ALIAS:jgroups}</property>
<property name="store_password">${env.JGROUPS_JKS_PASSWORD:changeme}</property>
</protocol>
<protocol type="pbcast.NAKACK2">
<property name="use_mcast_xmit">false</property>
</protocol>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="org.jgroups.protocols.AUTH">
<property name="auth_class">org.jgroups.auth.MD5Token</property>
<property name="token_hash">SHA</property>
<property name="auth_value">${env.JGROUPS_AUTH_SECRET:changeme2}</property>
</protocol>
<protocol type="pbcast.GMS">
<property name="print_physical_addrs">true</property>
<property name="print_local_addr">true</property>
</protocol>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
</stacks>
</subsystem>
sherif
April 12, 2022, 4:52pm
2
Hello,
I am kinda having the same issue. I managed to get the clustering to work, based on my understand, the embedded ISPN will read this file by default infinispan/default-jgroups-udp.xml at main · infinispan/infinispan · GitHub " which is bundled inside the infinispan jar file " once the shared caching option has been enabled in Keycloak, my question here, how can I override some of those configurations, like for example switch the local listen port to be 22222 instead of 0 " which is random", I did try to add the section from the mentioned file into conf/cache-ispn.xml " same idea as in here Keycloak 17 example using JGroups JDBC_PING discovery protocol for Infinispan · GitHub " however, those configurations are being ignored, what would be the correct way to override those settings?
Regards,
Sherif
xgp
April 12, 2022, 5:59pm
3
Put your <stack>...</stack>
section inside the <infinispan><jgroups>...
element. Here is an example of doing it for JDBC_PING: Keycloak 17 example using JGroups JDBC_PING discovery protocol for Infinispan · GitHub
sherif
April 12, 2022, 7:46pm
4
That what I did I did read your link and that what I have followed, I have the following inside <infinispan>
<jgroups>
<stack name="jdbc-udp" extends="udp">
<UDP bind_addr="${jgroups.bind.address,jgroups.udp.address:SITE_LOCAL}"
bind_port="${jgroups.bind.port,jgroups.udp.port:55200}"
mcast_addr="${jgroups.mcast_addr:230.0.0.4}"
mcast_port="${jgroups.mcast_port:45688}"
tos="0"
ucast_send_buf_size="1m"
mcast_send_buf_size="1m"
ucast_recv_buf_size="20m"
mcast_recv_buf_size="25m"
ip_ttl="${jgroups.ip_ttl:2}"
thread_naming_pattern="pl"
enable_diagnostics="false"
bundler_type="transfer-queue"
max_bundle_size="8500"
thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
thread_pool.keep_alive_time="60000"
thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}" />
</stack>
</jgroups>
not sure if the that is correct or not…
klr
April 13, 2022, 8:49am
5
Ok, so I ended up with the following content in /opt/keycloak/current/conf/jgroups.xml and I set KC_CACHE_CONFIG_FILE=jgroups.xml as environment varaible.
But in the startup log I get the error:
Apr 13 10:47:32 SETEST16 kc.sh[124016]: 2022-04-13 10:47:32,890 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.configuration.io.ConfigurationReaderException: Unexpected element 'property' encountered[11,32]
Apr 13 10:47:32 SETEST16 kc.sh[124016]: 2022-04-13 10:47:32,891 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unexpected element 'property' encountered[11,32]
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<!-- custom stack goes into the jgroups element -->
<jgroups>
<stack name="unicast-tcp">
<transport type="TCP" socket-binding="jgroups-tcp">
<property name="bind_port">${env.JGROUPS_PORT:7800}</property>
<property name="external_addr">${env.JGROUPS_EXTERNAL_IP:127.0.0.1}</property>
</transport>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="org.jgroups.protocols.SYM_ENCRYPT">
<property name="provider">SunJCE</property>
<property name="sym_algorithm">AES</property>
<property name="encrypt_entire_message">true</property>
<property name="keystore_name">${env.JGROUPS_KEYSTORE:/etc/secrets/jgroups.jceks}</property>
<property name="alias">${env.JGROUPS_KEYSTORE_ALIAS:jgroups}</property>
<property name="store_password">${env.JGROUPS_JKS_PASSWORD:changeme}</property>
</protocol>
<protocol type="pbcast.NAKACK2">
<property name="use_mcast_xmit">false</property>
</protocol>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="org.jgroups.protocols.AUTH">
<property name="auth_class">org.jgroups.auth.MD5Token</property>
<property name="token_hash">SHA</property>
<property name="auth_value">${env.JGROUPS_AUTH_SECRET:changeme2}</property>
</protocol>
<protocol type="pbcast.GMS">
<property name="print_physical_addrs">true</property>
<property name="print_local_addr">true</property>
</protocol>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
</jgroups>
<cache-container name="keycloak">
<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
<transport lock-timeout="60000" stack="unicast-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>
1 Like
sherif
April 13, 2022, 11:10am
6
Oh, I guess I knew what I was doing wrong, I missed adding the new stack name into the container line below this section
klr
April 13, 2022, 11:23am
7
Update:
I tried a slightly different config format and now possibly ran into an infinispan bug? ( [ISPN-13765] Unable to start full JGroups stacks without parent - Red Hat Issue Tracker )
Config:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 jgroups.xsd"
xmlns="urn:infinispan:config:11.0">
<!-- custom stack goes into the jgroups element -->
<jgroups>
<stack name="unicast-tcp">
<RSVP />
<FRAG2 />
<MFC />
<UFC />
<pbcast.GMS print_physical_addrs="true" print_local_addr="true"/>
<AUTH auth_class="org.jgroups.auth.MD5Token" token_hash="SHA" auth_value="changeme2"/>
<pbcast.STABLE />
<UNICAST3 />
<pbcast.NAKACK2 use_mcast_xmit="false"/>
<SYM_ENCRYPT provider="SunJCE" sym_algorithm="AES" encrypt_entire_message="true" keystore_name="/etc/secrets/jgroups/jceks" alias="jgroups" store_password="changeme"/>
<VERIFY_SUSPECT />
<FD_ALL />
<FD_SOCK />
<MERGE3 />
<TCPPING initial_hosts="${env.JGROUPS_REMOTE_HOSTS:localhost[7800]}"/>
<TCP bind_port="7800" bind_addr="SITE_LOCAL"/>
</stack>
</jgroups>
<cache-container name="keycloak">
<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
<transport lock-timeout="60000" stack="unicast-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>
Log-Output:
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start caches
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.CacheConfigurationException: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ISPN000540: No such JGroups stack 'null'
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
1 Like