Infinispan TCPPING discovery in Quarkus distribution

Configuring the server
1

In older version could just set JGROUPS_DISCOVERY_PROTOCOL, JGROUPS_DISCOVERY_PROPERTIES and JGROUPS_TRANSPORT_STACK.
In the quarkus verision (18.0) seems not to work anymore. Also other discovery protocols doesn’t suite me: it’s running in docker on dedicated servers, so no possibility for multicast.

2

Seems that the default JGroups stack configurations are coming from Infinispan-core (tcp, udp, google, etc). Also don’t see easy way to inject an “default-configs/default-jgroups-tcpping.xml” file into classpath. Bummer, seems, that I will have to wire custom cache config file with JGroups section.

For those, who are interested: Embedding Infinispan in Java Applications

3

Of course it‘s mentioned in the Keycloak docs. Also there are already examples mentioned here in the forum.
See Configuring distributed caches - Keycloak and e.g. Use of JDBC_PING with Keycloak 17 (Quarkus distro) (whole thread, do not just read the first post!)

3 Likes
4

Doesn’t seems to work unfortunately. Probably I’m missing something.

<infinispan                                                                                                                        
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                                                                                                                        
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"                                                                                                                        
        xmlns="urn:infinispan:config:11.0">                                                                                                                        
                                                                                                                        
    <jgroups>                                                                                                                        
      <stack name="tcpping" extends="tcp">                                                                                                                        
        <TCPPING                                                                                                                        
          timeout="3000"                                                                                                                        
          initial_hosts="${env.KC_INITIAL_HOSTS}"                                                                                                                        
          port_range="1"                                                                                                                        
          num_initial_members="0"                                                                                                                        
          stack.combine="REPLACE"                                                                                                                        
          stack.position="MPING"                                                                                                                        
          />                                                                                                                        
      </stack>                                                                                                                        
    </jgroups>                                                                                                                        
                                                                                                                        
    <cache-container name="keycloak">                                                                                                                        
        <transport lock-timeout="60000" stack="tcpping"/>                                                                                                                        
        <local-cache name="realms">
...

Container log output:

2022-05-05 15:19:10,088 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2022-05-05 15:19:10,088 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2022-05-05 15:19:10,161 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-05-05 15:19:10,161 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
2022-05-05 15:19:10,162 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-05-05 15:19:10,162 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB

Seems, that transport stack property is ignored, because it still try to do UDP instead of custom tcpping stack.

5

Found, need to explicitly specify KC_CACHE_CONFIG_FILE as file name in /opt/keycloak/conf folder for the custom config to load.

2 Likes
6

Hello @nikel123 i have having same issue with keycloak-18. i have below build in docker
FROM docker.artifactory…/keycloak/keycloak:18.0 as builder

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange,preview,scripts
ENV KC_DB=postgres
ENV KC_CACHE=ispn
ENV KC_CACHE_STACK=kubernetes
ENV KC_HTTP_RELATIVE_PATH=“/auth”
ENV KC_PROXY=“edge”

Install custom providers

RUN curl -sL https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar -o /opt/keycloak/providers/keycloak-metrics-spi-2.5.3.jar
COPY --from=intermediate ./target/keycloak_auth-1.0-SNAPSHOT-jar-with-dependencies.jar /opt/keycloak/providers
RUN /opt/keycloak/bin/kc.sh build
AND

  • name: JGROUPS_DISCOVERY_PROTOCOL
    value: “dns.DNS_PING”
    - name: JGROUPS_TRANSPORT_STACK
    value: “tcp”
    - name: JGROUPS_DISCOVERY_PROPERTIES
    value: “dns_query=keycloak-headless”
    - name: CACHE_OWNERS_COUNT
    value: “2”
    - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
    value: “2”
    here is my headleass-service
    apiVersion: v1
    kind: Service
    metadata:
    name: keycloak-headless
    spec:

Important parameter to discover every instance even before its complete startup

publishNotReadyAddresses: true
clusterIP: None
ports:
- name: ping
port: 7600
targetPort: 7600
selector:
app: keycloak

Would you please take a look and suggest? Please help with you working solution.