16.x --> 17.0.1 Infinispan and Clustering on AWS ECS

modetry · June 23, 2022, 9:14am

Hello,

We currently run keycloak 16 Wildfly in a docker across 4 tasks on an ECS cluster. This cluster and cache is configured in the standalone-ha.xml file. Within this, we use the following blocks to set the TCP and socket binding. We then have terraform to create the cluster on ECS. This works.

                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp">
                        <property name="external_addr">${env.INSTANCE_IP}</property>
                    </transport>
                    <protocol type="org.jgroups.protocols.JDBC_PING">
                        <property name="datasource_jndi_name">java:jboss/datasources/KeycloakDS</property>
                        <property name="initialize_sql">
                            CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name)); DELETE FROM JGROUPSPING WHERE created &lt; (current_timestamp - interval '2' day);
                        </property>
                        <property name="insert_single_sql">
                            INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)
                        </property>
                        <property name="insert_single_sql">
                            INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data)                                                                                                                                                                                                                                                            values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)
                        </property>
                        <property name="delete_single_sql">DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?</property>
                        <property name="select_all_pingdata_sql">SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?</property>
                    </protocol>
		</stack>

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}">
            <client-mapping destination-address="${jboss.host.name}"/>
        </socket-binding>
        <socket-binding name="https" port="${jboss.https.port:8443}"/>
        <socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" interface="public" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
        <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="${jboss.mail.server.host:localhost}" port="${jboss.mail.server.port:25}"/>
        </outbound-socket-binding>
    </socket-binding-group>

The problem….
We have recently started moving to 17.0.1 Quarkus and part of that includes setting the cache config in the keycloak.conf file as follows.

cache=ispn
cache-stack=tcp
cache-config-file=clusterConfig.xml

And then the clusterConfig.xml

  <jgroups>
    <stack name="jdbc-ping-tcp" extends="tcp">
      <TCP external_addr="${env.INSTANCE_IP}"
                bind_port="7800"
      />
      <JDBC_PING connection_driver="org.postgresql.Driver"
                 connection_username="${env.POSTGRES_USER}"
                 connection_password="${env.POSTGRES_PASSWORD}"
                 connection_url="jdbc:postgresql://${env.POSTGRES_PORT_5432_TCP_ADDR}:5432/${env.POSTGRES_DATABASE}"
                 initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name)); DELETE FROM JGROUPSPING WHERE created &lt; (current_timestamp - interval '2' day);"
                 insert_single_sql="INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)"
                 delete_single_sql="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?"
                 select_all_pingdata_sql="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?"
                 info_writer_sleep_time="500"
                 remove_all_data_on_view_change="true"
                 stack.combine="REPLACE"
                 stack.position="MPING" />
    </stack>
  </jgroups>

  <cache-container name="keycloak">
    <transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
    <local-cache name="realms">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="authorization">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="keys">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="3600000"/>
      <memory max-count="1000"/>
    </local-cache>
    <replicated-cache name="work">
        <expiration lifespan="900000000000000000"/>
    </replicated-cache>
    <distributed-cache name="sessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="actionTokens" owners="4">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
  </cache-container>

The above configuration works to create a “cluster” of 1…

Keycloak 17.0.1 seems to use infinispan 13.0.6.Final at some built in level and this XML schema refers to the JGROUPS one which allows the block to work. It is only the infinispan:config XML schema though so adding any information into the clusterConfig.xml file, so that I can add the socket-binding and interfaces, is impossible.

The Question.

How do I configure Keycloak 17 Quarkus to work with an ECS cluster. WHERE in WHAT config can I set the socket-binding instructions?

Please please please. Any ideas considered. Official documentation around this is minimal at best.

Thanks
Tom.

modetry · June 28, 2022, 10:57am

To add a quick TLDR to the above there are two main questions here:

Now that KC is using infinispan properly with quarkus there appears to be nowhere for us to set the socket bindings that we had previously within the standalone XML. Where do these now go?
We have an ECS set up on AWS and have clustering set with TCP and JGROUPS / JDBC_PING on the transport and this isn’t working with more than one task. I have now tried adding this to my docker-entrypoint.sh script with no joy.

echo "INSTANCE_IP: ${INSTANCE_IP}"
echo "INSTANCE_HOSTNAME: ${INSTANCE_HOSTNAME}"
echo "JGROUPS_DISCOVERY_EXTERNAL_IP: ${JGROUPS_DISCOVERY_EXTERNAL_IP}"
export JAVA_OPTS="${JAVA_OPTS} -Djgroups.dns.query=${INSTANCE_HOSTNAME}"
echo "JAVA_OPTS: ${JAVA_OPTS}"

Really getting a little desperate as nothing seems to work…
Thanks
Tom

UtkarshBhavsar · July 27, 2022, 6:20am

Hi, @modetry, Did you get the solution for this? we are facing the same issue

modetry · July 27, 2022, 7:24am

I’m afraid not.

There are a lot of other threads out there with similar issues and potential solutions but none of them seem to work for me…

nikel123 · August 3, 2022, 12:03pm

Here is my post on the issue with a solution at the end for TCPPING, probably some parts can be reused for JDBCPING: Infinispan TCPPING discovery in Quarkus distribution - #3 by dasniko

Basically the clustering config is going to cache-ispn.xml:

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

    <jgroups>
      <stack name="tcpping" extends="tcp">
        <TCP
          external_addr="${env.EXTERNAL_IP}"
          stack.combine="COMBINE"
          />
        <TCPPING
          initial_hosts="${env.INITIAL_HOSTS}"
          port_range="0"
          stack.combine="REPLACE"
          stack.position="MPING"
          />
      </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="tcpping"/>
...

Which you put into the container under: ADD cache-ispn.xml /opt/keycloak/conf/cache-ispn.xml

And set environment variable: ENV KC_CACHE_CONFIG_FILE="cache-ispn.xml" and the other custom configuration variables you put into the config (like INITIAL_HOSTS and EXTERNAL_IP in my example)

You have to adapt the jgroup stack configuration to JDBC_PING instead of TCPPING and then it should work.

Topic		Replies	Views
Keycloak Cluster JGroups/Infinispan on EC2 Configuring the server	9	4648	May 11, 2023
Keycloak 17 cluster running on docker swarm Configuring the server clustering , container	2	3442	August 17, 2022
Keycloak 20.0.3 Infinispan not starting with correct transport stack for EC2 Configuring the server	0	138	March 24, 2023
Adventures with Docker Swarm and Keycloak Clustering Configuring the server clustering , container	13	7816	January 17, 2024
JDBC_PING stack for keycloak-19 quarkus	3	1324	October 27, 2022

16.x --> 17.0.1 Infinispan and Clustering on AWS ECS

Related Topics