Infinispan 'tcp': Does it require specific port to be opened on each server?

v18.0.1

Infinispan is throwing me off and I’m beginning to question my sanity.

Listed below are (1) the log output w.r.t infinispan and (2) the infinispan configuration file.

The build and start are set as the expected. However, I can’t figure out why the error message occurs and no database table created.

Have I missed something obvious? Does a port required to be specified for each server in the cluster?

1 - LOG OUTPUT

022-06-27 13:33:33,342 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: broker.dev.id.ubc.ca, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: true
2022-06-27 13:33:35,407 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-06-27 13:33:35,419 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-06-27 13:33:35,461 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-06-27 13:33:36,028 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final
2022-06-27 13:33:36,365 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2022-06-27 13:33:36,370 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2022-06-27 13:33:36,572 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB

2 - CONFIGURATION FILE

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
    xmlns="urn:infinispan:config:11.0">

  <!-- custom stack goes into the jgroups element -->
  <jgroups>
    <stack name="jdbc-ping-tcp" extends="tcp">
      <JDBC_PING connection_driver="oracle.jdbc.driver.OracleDriver" 
                 connection_username="${env.KC_DB_USERNAME}" 
                 connection_password="${env.KC_DB_PASSWORD}" 
                 connection_url="${env.KC_DB_URL}" 
                 initialize_sql="CREATE TABLE JGROUPSPING (own_addr varchar2(200) NOT NULL, cluster_name varchar2(200) NOT NULL,ping_data BLOB,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
                 info_writer_sleep_time="500"
                 remove_all_data_on_view_change="true"
                 stack.combine="REPLACE"
                 stack.position="MPING" />
    </stack>
  </jgroups>
  
  <cache-container name="keycloak">
    <!-- custom stack must be referenced by name in the stack attribute of the transport element -->
    <transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
    <local-cache name="realms">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <distributed-cache name="sessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <local-cache name="authorization">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <replicated-cache name="work">
      <expiration lifespan="-1"/>
    </replicated-cache>
    <local-cache name="keys">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="3600000"/>
      <memory max-count="1000"/>
    </local-cache>
    <distributed-cache name="actionTokens" owners="2">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="-1" lifespan="-1" interval="300000"/>
      <memory max-count="-1"/>
    </distributed-cache>
  </cache-container>
</infinispan>

Infinispan uses default port 7800 (and this has to be open, of course).
But the log output
ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
is not related to the port, but to an error in your cache config file (no, I didn‘t see the error on a quick look, sorry), so Infinispan uses the default configuration instead your custom one.

1 Like

Hello,

I had the same trouble recently and I solved the issue ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration! by adding the right environment variable to my Docker container -e KC_CACHE_STACK="tcp"

Hope this helps,
Ulrich

1 Like

Hi Ulrich,
Thanks for your input.
I tried to configure JDBC_PING for AWS EC2, too (KC v18.0.2). I tried the suggestions mentioned here.
Unfortunately, looks like the JDBC_PING configuration cannot be found …

(sorry, don’t know how to use blockquote for logs)

cc-keycloak 2022-09-02 20:18:33,562 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: XX.XXX, Strict HTTPS: true, Path: , Strict BackChannel: false, Admin: , Port: -1, Proxied: true ││ cc-keycloak 2022-09-02 20:18:35,708 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal ││ cc-keycloak 2022-09-02 20:18:35,810 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled ││ cc-keycloak 2022-09-02 20:18:35,913 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller ‘org.infinispan.jboss.marshalling.core.JBossUserMarshaller’ ││ cc-keycloak 2022-09-02 20:18:36,475 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan ‘Triskaidekaphobia’ 13.0.9.Final ││ cc-keycloak 2022-09-02 20:18:36,701 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel ISPN ││ cc-keycloak 2022-09-02 20:18:38,849 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) cc-keycloak-52709: no members discovered after 2009 ms: creating cluster as coordinator ││ cc-keycloak 2022-09-02 20:18:38,863 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [cc-keycloak-52709|0] (1) [cc-keycloak-52709] ││ cc-keycloak 2022-09-02 20:18:38,880 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel ISPN local address is cc-keycloak-52709, physical addresses are [10.0.20.170:7800] ││ cc-keycloak 2022-09-02 20:18:39,854 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: cc-keycloak-52709, Site name: null ││ cc-keycloak 2022-09-02 20:18:40,752 INFO [io.quarkus] (main) Keycloak 18.0.2 on JVM (powered by Quarkus 2.7.5.Final) started in 13.123s. Listening on: http://0.0.0.0:8080

Hi @anowak-ct

Did you fix this issue ? I tried on keycloak19.0.1 by use docker it shown could not find cache config file.

2022-09-05 09:03:47,198 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2022-09-05 09:03:47,199 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Error details:: java.lang.RuntimeException: Failed to start caches
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000365: Could not find the specified JGroups configuration file ‘default-configs/default-jgroups-jdbc-ping-tcp.xml’

what’s solution for this ?

thanks

Could you please share the Dockerfile so we can try to reproduce because we are using keycloak 19.0.1 with jdbc_ping already on aws.

Hi !

This one for Dockerfile

FROM quay.io/keycloak/keycloak:19.0.1 as builder

ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=preview
ENV KC_DB=postgres

# specify the custom cache config file here

ENV KC_CACHE_CONFIG_FILE=jdbc-ping.xml

COPY ./jdbc-ping.xml /opt/keycloak/conf/jdbc-ping.xml

RUN /opt/keycloak/bin/kc.sh build --cache-config-file=jdbc-ping.xml

FROM quay.io/keycloak/keycloak:19.0.1
COPY --from=builder /opt/keycloak/ /opt/keycloak/

WORKDIR /opt/keycloak

# for demonstration purposes only, please make sure to use proper certificates in production instead
RUN keytool -genkeypair -storepass password -storetype PKCS12 -keyalg RSA -keysize 2048 -dname "CN=server" -alias server -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -keystore conf/server.keystore


ENTRYPOINT ["/opt/keycloak/bin/kc.sh","start"]

Can you try with this dockerfile (I know this is not the best way but works for dev environment :wink:) ? Also don’t pass --cache if you are anyways env variable.

FROM quay.io/keycloak/keycloak:19.0.1

ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV KC_FEATURES_DISABLED=admin2

# copy the custom cache config file into the keycloak conf dir
COPY ./cache-ispn-jdbc-ping.xml /opt/keycloak/conf/cache-ispn-jdbc-ping.xml

# specify the custom cache config file here
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml

# Provide JDBC connection String
ENV KC_DB=postgres
ENV JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING
ENV QUARKUS_HTTP_LIMITS_MAX_INITIAL_LINE_LENGTH=16384000

ENV KC_HTTPS_PORT=9443
ENV KC_PROXY=passthrough

#Build
RUN /opt/keycloak/bin/kc.sh build

EXPOSE 9443
EXPOSE 7800

ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]