Keycloak with external multiple Infinispan replicas

I run Keycloak v17 (Quarkus) and remote Infinispan on Kubernetes. It runs fine with single Keycloak pod and single Infinispan pod. However, it does not run with single Keycloak pod and multiple Infinispan pods.

When I run single Keycloak pod and single Infinispan pod I provide remote-server host to be Infinispan Kubernetes service name (fx infinispan.keycloak.svc.cluster.local). However during the Keycloak start I see that the host is changed from Kubernetes service ip to a pod ip:

2022-03-23 12:21:47,801 INFO  [org.infinispan.HOTROD] (blocking-thread--p3-t1) ISPN004021: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.5.Final
2022-03-23 12:21:47,806 INFO  [org.infinispan.HOTROD] (Thread-0) ISPN004006: Server sent new topology view (id=1, age=0) containing 1 addresses: [100.64.148.158:11222]
2022-03-23 12:21:47,806 INFO  [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(100.64.148.158:11222), adding to the pool.
2022-03-23 12:21:47,807 DEBUG [org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory] (Thread-0) Creating new channel pool for 100.64.148.158:11222
2022-03-23 12:21:47,807 INFO  [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan.keycloak.svc.cluster.local:11222), removing from the pool.

However, when I change it to two Infinispan replicas, it looks like the Keycloak only “remembers” one Infinispan pod ip, but not the other:

2022-03-23 11:34:37,975 INFO  [org.infinispan.HOTROD] (blocking-thread--p3-t1) ISPN004021: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.5.Final
2022-03-23 11:34:37,988 WARN  [org.infinispan.HOTROD] (Thread-0) ISPN004005: Error received from the server: java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
2022-03-23 11:34:37,990 DEBUG [org.infinispan.persistence.manager.PersistenceManagerImpl] (keycloak-cache-init) PersistenceManagerImpl encountered an exception during startup of stores: java.util.concurrent.CompletionException: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=3 returned server error (status=0x85): java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=3 returned server error (status=0x85): java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:323)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:168)
	at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:139)
	at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:94)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

So how do I use Keycloak with external multiple replicas Infinispan?

Did you find the solution?

Alhamdulillah It works now (image version 17.0.1-legacy)

Could you help me with the solution. i am also having same issue.
2023-12-21 05:17:31,124 WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-2-1) ISPN004005: Error received from the server: java.lang.IllegalArgumentException: No interface address matching /172.18.1.124
2023-12-21 05:17:31,124 WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-1-1) ISPN004005: Error received from the server: java.lang.IllegalArgumentException: No interface address matching /172.18.11.46
2023-12-21 05:17:31,173 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool – 4) MSC000001: Failed to start service jboss.deployment.unit.“bundle-cache-manager.war”.component.OrganizationIdPopulationSchedulerManager.START: org.jboss.msc.service.StartException in service jboss.deployment.unit.“bundle-cache-manager.war”.component.OrganizationIdPopulationSchedulerManager.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
at org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
at org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
at org.jboss.as.ejb3@18.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:127)
at org.jboss.as.ejb3@18.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:141)
at org.jboss.as.ee@18.0.1.Final//org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
… 8 more