16.x --> 17.0.1 Infinispan and Clustering on AWS ECS

Hello,

We currently run keycloak 16 Wildfly in a docker across 4 tasks on an ECS cluster. This cluster and cache is configured in the standalone-ha.xml file. Within this, we use the following blocks to set the TCP and socket binding. We then have terraform to create the cluster on ECS. This works.

                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp">
                        <property name="external_addr">${env.INSTANCE_IP}</property>
                    </transport>
                    <protocol type="org.jgroups.protocols.JDBC_PING">
                        <property name="datasource_jndi_name">java:jboss/datasources/KeycloakDS</property>
                        <property name="initialize_sql">
                            CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name)); DELETE FROM JGROUPSPING WHERE created &lt; (current_timestamp - interval '2' day);
                        </property>
                        <property name="insert_single_sql">
                            INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)
                        </property>
                        <property name="insert_single_sql">
                            INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data)                                                                                                                                                                                                                                                            values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)
                        </property>
                        <property name="delete_single_sql">DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?</property>
                        <property name="select_all_pingdata_sql">SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?</property>
                    </protocol>
		</stack>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}">
            <client-mapping destination-address="${jboss.host.name}"/>
        </socket-binding>
        <socket-binding name="https" port="${jboss.https.port:8443}"/>
        <socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" interface="public" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
        <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="${jboss.mail.server.host:localhost}" port="${jboss.mail.server.port:25}"/>
        </outbound-socket-binding>
    </socket-binding-group>

The problem….
We have recently started moving to 17.0.1 Quarkus and part of that includes setting the cache config in the keycloak.conf file as follows.

cache=ispn
cache-stack=tcp
cache-config-file=clusterConfig.xml

And then the clusterConfig.xml

  <jgroups>
    <stack name="jdbc-ping-tcp" extends="tcp">
      <TCP external_addr="${env.INSTANCE_IP}"
                bind_port="7800"
      />
      <JDBC_PING connection_driver="org.postgresql.Driver"
                 connection_username="${env.POSTGRES_USER}"
                 connection_password="${env.POSTGRES_PASSWORD}"
                 connection_url="jdbc:postgresql://${env.POSTGRES_PORT_5432_TCP_ADDR}:5432/${env.POSTGRES_DATABASE}"
                 initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name)); DELETE FROM JGROUPSPING WHERE created &lt; (current_timestamp - interval '2' day);"
                 insert_single_sql="INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.tcp.address:127.0.0.1}',NOW(), ?, ?)"
                 delete_single_sql="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?"
                 select_all_pingdata_sql="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?"
                 info_writer_sleep_time="500"
                 remove_all_data_on_view_change="true"
                 stack.combine="REPLACE"
                 stack.position="MPING" />
    </stack>
  </jgroups>

  <cache-container name="keycloak">
    <transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
    <local-cache name="realms">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="authorization">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="keys">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="3600000"/>
      <memory max-count="1000"/>
    </local-cache>
    <replicated-cache name="work">
        <expiration lifespan="900000000000000000"/>
    </replicated-cache>
    <distributed-cache name="sessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="4">
        <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
    <distributed-cache name="actionTokens" owners="4">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration interval="18000000" lifespan="900000000000000000" max-idle="-1"/>
    </distributed-cache>
  </cache-container>

The above configuration works to create a “cluster” of 1…

Keycloak 17.0.1 seems to use infinispan 13.0.6.Final at some built in level and this XML schema refers to the JGROUPS one which allows the block to work. It is only the infinispan:config XML schema though so adding any information into the clusterConfig.xml file, so that I can add the socket-binding and interfaces, is impossible.

The Question.

How do I configure Keycloak 17 Quarkus to work with an ECS cluster. WHERE in WHAT config can I set the socket-binding instructions?

Please please please. Any ideas considered. Official documentation around this is minimal at best.

Thanks
Tom.

To add a quick TLDR to the above there are two main questions here:

  1. Now that KC is using infinispan properly with quarkus there appears to be nowhere for us to set the socket bindings that we had previously within the standalone XML. Where do these now go?

  2. We have an ECS set up on AWS and have clustering set with TCP and JGROUPS / JDBC_PING on the transport and this isn’t working with more than one task. I have now tried adding this to my docker-entrypoint.sh script with no joy.

echo "INSTANCE_IP: ${INSTANCE_IP}"
echo "INSTANCE_HOSTNAME: ${INSTANCE_HOSTNAME}"
echo "JGROUPS_DISCOVERY_EXTERNAL_IP: ${JGROUPS_DISCOVERY_EXTERNAL_IP}"
export JAVA_OPTS="${JAVA_OPTS} -Djgroups.dns.query=${INSTANCE_HOSTNAME}"
echo "JAVA_OPTS: ${JAVA_OPTS}"

Really getting a little desperate as nothing seems to work… :frowning:
Thanks
Tom