Issue with Keycloak 6.0.0 Cluster Utilizing 20 Gb/s Bandwidth

Hello -

We’re running keycloak 6.0.0 and have it clustered via the default configuration that ships with the standalone-ha.xml clustering configuration (we’re not using cross-datacenter replication).

After a lengthy period of stability, we can no longer cluster the nodes together. The application logs have error messages such as:

2019-12-05 13:36:19,494 ERROR [org.keycloak.models.sessions.infinispan.util.FuturesHelper] (Timer-3) Exception when waiting for future: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 5242042 from XXXXXX
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at org.keycloak.models.sessions.infinispan.util.FuturesHelper.waitForAllToFinish(FuturesHelper.java:47)
    at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.removeExpiredOfflineUserSessions(InfinispanUserSessionProvider.java:611)
    at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.removeExpired(InfinispanUserSessionProvider.java:482)
    at org.keycloak.services.scheduled.ClearExpiredUserSessions.run(ClearExpiredUserSessions.java:37)
    at org.keycloak.services.scheduled.ScheduledTaskRunner.runTask(ScheduledTaskRunner.java:61)
    at org.keycloak.services.scheduled.ScheduledTaskRunner.run(ScheduledTaskRunner.java:45)
    at org.keycloak.timer.basic.BasicTimerProvider$1.run(BasicTimerProvider.java:51)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 5242042 from XXXXXX
    at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
    at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
    at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

When this happens, we’ve observed the keycloak nodes generating 20 Gigabits of network traffic per second.

We’ve been able to reproduce the issue in our QA environment by generating a large number of offline tokens and then having them expire from the infinispan cache. When cache is purged, we took a network capture. Each datagram is 50 kb in size, which is enormous. The traffic eventually grows so large that all network links saturate and everything fails. We examined one particular datagram and was surprised to see java stack traces inside (best seen in the following screenshot since it is difficult to format):

We haven’t confirmed it yet, but suspect the nodes are sending a storm of sorts to each other. One node sends a datagram with an exception and then an error occurs thus causing another exception until things grow in size to the point where the network packet size of ~60 kb is exceeded.

Has anybody seen anything like this? Is it a known bug? Any advice for troubleshooting or fixing?

1 Like

I posted a more detailed version of this question and answered it here: