Correct value for jgroups external_addr value when running KC 16.x in Docker Swarm

Hello,

I am stuck trying to set the proper jgroups configuration for a clustered standalone installation of KC:

/subsystem=jgroups/stack=tcp/transport=TCP/property=external_addr:add(value=${env.JGROUPS_DISCOVERY_EXTERNAL_IP})

I have 3 Docker Swarm host machines on EC2, each running potentially more than 1 KC container, to what value should I set the jgroup external_addr configuration of each container?

Does the value need to be unique per KC instance which is my understanding so far or can 2 instances running on the same host announce the same external_addr - such as the internal EC2 IP/hostname?

If the value needs to be unique what should I use in my scenario - I was thinking : JGROUPS_DISCOVERY_EXTERNAL_IP : {{.Task.Name}}

Thanks,

David

Hello, I have tried both strategies to no avail:

  • Setting JGROUPS_DISCOVERY_EXTERNAL_IP to {{.Task.Name}} - making sure tasks can ping each other
  • Modifying docker-entrypoint.sh to set JGROUPS_DISCOVERY_EXTERNAL_IP to the EC2 internal IP address something like: ip-172-xx-xx-xxx.eu-west-3.compute.internal

Do these need to be actual IPs not hostnames?

Neither strategy results in an Infispan cluster being created between nodes as per logs:

# I expect to see several nodes here...
13:13:21,135 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 56) ISPN000094: Received new cluster view for channel ejb: [699ec079bb5e|0] (1) [699ec079bb5e]

In addition I have added JDBC_PING.cli from an official blog post on the subject to my image since by looking at the KC 16.x Dockerfile repository I noticed that this setting is not being taking into account to set external_addr for the Infinispan discovery.

Has anyone actually gotten JDBC_PING to work using Docker Swarm as an orchestrator? If so a concrete example would be appreciated.

For what it’s worth I do see some join attempts (regardless of specifying JGROUPS_DISCOVERY_EXTERNAL_IP) but they fail:

18:26:31,151 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 56) 7a122481a7f2: JOIN(7a122481a7f2) sent to 70c8446ec082 timed out (after 3000 ms), on try 8

Best regards,

David

Ok, for anyone coming through here, in the end I set the BIND: "{{.Task.Name}}" environment variable in the compose file I use on Docker Swarm. This seems to do the trick, I see the Infinispan cluster forming. I believe the issue was something to do with binding adresses used by JGroups and/or Wildfly, possibly similar to what was discued here.

I do not have to set JGROUPS_DISCOVERY_EXTERNAL_IP in the end from what I see. I did not change/add parameters from the repository example apart from the the deploy part for Docker Swarm.

/David

This helped me one time: Adventures with Docker Swarm and Keycloak Clustering

1 Like