I run Keycloak v17 (Quarkus) and remote Infinispan on Kubernetes. It runs fine with single Keycloak pod and single Infinispan pod. However, it does not run with single Keycloak pod and multiple Infinispan pods.
When I run single Keycloak pod and single Infinispan pod I provide remote-server
host
to be Infinispan Kubernetes service name (fx infinispan.keycloak.svc.cluster.local
). However during the Keycloak start I see that the host is changed from Kubernetes service ip to a pod ip:
2022-03-23 12:21:47,801 INFO [org.infinispan.HOTROD] (blocking-thread--p3-t1) ISPN004021: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.5.Final
2022-03-23 12:21:47,806 INFO [org.infinispan.HOTROD] (Thread-0) ISPN004006: Server sent new topology view (id=1, age=0) containing 1 addresses: [100.64.148.158:11222]
2022-03-23 12:21:47,806 INFO [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(100.64.148.158:11222), adding to the pool.
2022-03-23 12:21:47,807 DEBUG [org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory] (Thread-0) Creating new channel pool for 100.64.148.158:11222
2022-03-23 12:21:47,807 INFO [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan.keycloak.svc.cluster.local:11222), removing from the pool.
However, when I change it to two Infinispan replicas, it looks like the Keycloak only “remembers” one Infinispan pod ip, but not the other:
2022-03-23 11:34:37,975 INFO [org.infinispan.HOTROD] (blocking-thread--p3-t1) ISPN004021: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.5.Final
2022-03-23 11:34:37,988 WARN [org.infinispan.HOTROD] (Thread-0) ISPN004005: Error received from the server: java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
2022-03-23 11:34:37,990 DEBUG [org.infinispan.persistence.manager.PersistenceManagerImpl] (keycloak-cache-init) PersistenceManagerImpl encountered an exception during startup of stores: java.util.concurrent.CompletionException: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=3 returned server error (status=0x85): java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=3 returned server error (status=0x85): java.lang.IllegalArgumentException: No interface address matching '/100.64.115.30' in MultiHomedServerAddress{port=11222, addresses=[/100.64.182.249/32, /127.0.0.1/8]}
at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:323)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:168)
at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:139)
at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:94)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
So how do I use Keycloak with external multiple replicas Infinispan?