Is it possible to replace Infinspan in Keycloak with a centralized cache store like Redis or Memcached?
It took us a while but we finally got Keycloak running in clustered mode using JDBC_PING on Amazon’s ECS cluster service. We have also played with several configurations like a single owner for the Infinisan caches or multiple owners and sticky sessions or non-sticky sessions. Even though sticky sessions has its downsides, like potentially losing sessions, we’d prefer to run this way as it is the most performant solution as caches do not have to be replicated to the entire cluster. One of the problems we’ve run into is that the AWS ALB does not seem to want to route new sessions very evenly when sticky sessions are turned on. Eventually one of the cluster nodes ends up dying under load (and all the associated sessions are lost).
All this would be much easier to configure if we could just cut out JGroups and Infinispan and have keycloak talk to Redis or Memcached while running in standalone mode. I do realize that performance will be slightly slower as objects will have to be serialized/deserialized, but the benefits of being able to dynamically scale the cluster without having to have users potentially re-login would be very beneficial to us.
Is this possible? Has anyone ever done it?
I saw this answer on SO, but I’m not interested in having to customize a bunch of SPIs to do this.
Thank you for the reply. Will Keycloak only work with the in-memory clustered version of Infinispan or can I plugin the Redis version of Infinispan?
We are running Keycloak inside of AWS ECS with autoscaling. Running in standalone-ha cluster mode and using JBCD_PING for node discovery. We have run into two problems:
when we turn on sticky sessions and set the number of owners = 1 the AWS ALB ends up sending most or all of the traffic to one cluster node and that cluster node eventually dies under load
when we turn sticky sessions off and set the number of owners = the cluster size (e.g. 8). This works for us, but we can’t get the cluster to scale any larger than 8 nodes because of memory constraints and the cluster has to synchronize data with the whole cluster so transactions start taking too long. Also we can’t allow the cluster to autoscale like this b/c once we get past 8 it is not usable anymore.
I would prefer to do some testing with a shared cache instead of a replicated cache so I could turn sticky sessions off and not have to worry about the nodes discovering or talking to each other. The nodes would only talk to the database or shared cache.
What I’d like to do is have the option to run Keycloak in different types of clustering modes. Stateless or Replication mode without the downside of possibly loosing sessions that comes with running Keycloak in sticky-session mode. Connect2ID allows for this type of clustering options as described here:
Hi @csavory,
Need quick information. I am curious to know how many total number of Realms you are handling with 8 node domain cluster ? Also per Realm how many groups you created. I just want to size it so that it works for our scenarios too. Any ball park figures would really help.
We are in the process of creating a Domain Cluster for one DC. The backup DC we will build later. We handle 1500 Realms, 12 to 15 clients per Realm. Also 75 to 100 groups per Realm ? I would like to learn any lessons from you.
Right ! But I have no idea when KeyCloak X is coming. Any help there ? Or till then, I would like to use the existing one but I want to size it, so I am looking for some sort of real world examples.
We have about 15 realms. 8 node cluster was about as high as we could get it, but I want to say that it never really worked that well. We tried to use both sticky sessions (recommeded) as well the shared infispan cache with groups of 2.
That why I was thinking a central cache option like a db or redis would be the better way to go. Even though it would sacrifice performance a little, scaling and clustering would be so much easier.
As far as I can tell from https://www.keycloak.org/2020/12/first-keycloak-x-release.adoc, not even Keycloak.X supports Redis centralized caching. @kkcmadhu I’m not sure if keycloak x will even attempt to solve this, it seems like Red Hat is dead set on just keeping infinispan (I even searched their jira).
However, this line in the blog gives me some hope:
However, the configuration is now using Infinispan’s native configuration as opposed to using an abstraction as in the Wildfly Infinispan Subsystem.
I’m hoping this means we can just copy+paste configuration for external infinispan clusters backed by redis (I’m sure there are examples out there) into keycloak.x and finally make keycloak actually stateless.
It’s kind of a pipe dream at this point given that 1. Keycloak.X basically has zero documentation at this point, 2. There seems to be basically zero documentation/StackOverflow answers/examples for plugging in external infinispan clusters.
I’m really hoping someone at RH sees this and at least tries to migrate Keycloak.X’s session management to redis/memcached, because Keycloak being (implicitly) stateful is a BIG turn-off for me currently to the point where I’m evaluating alternatives.
For the time being we are still using Infinispan and JGroups for clustering and HA deployments.
The blog says this, but I still have not yet found roadmaps/jira tickets to indicate that RH is even thinking about making Keycloak stateless…
@csavory did you managed to find working clustering solution on AWS ECS?
We struggle with same problem. From some time our 3 nodes cluster became unstable and both - killing one task and deploying new version (3 new tasks) cause it to be unresponsive and we need to kill all tasks to make it responsive again. We tried to play with CACHE_OWNERS and sticky sessions but nothing really helps and ew have another downtimes caused by cluster not being able to synchronize properly. Do you have any hints how to make it more stable?
@slaskawi you heard our problems. I can see your both Keycloak and Infinispan maintainer. What do you suggest? With stateless Keycloak and basically simple Redis as centralized cache we woudnt have current issues. And currently we’re experiencing dontime after downtime so I would really appreciate Keycloak team member help here. I asked on google group some time ago but no one responded.
Many cloud providers offers managed redis or memcached. It would be great if keycloak could ditch infinispan and use one of them but as I understand it’s not possible in near future.
The Keycloak team works currently on a “Map Storage SPI”, which, from my understanding, should provide an interface for a distributed cache backend.
I might be wrong, I didn’t focus on that yet, so perhaps I misunderstood this. But if I’m right, this would mean that one can connect a map based system (which the most caches are) to Keycloak, instead of using Infinispan.
Simply switching from one product to another wouldn’t be a solution here. As soon there would be another product hard coded/used, some other folks would complain. So, making it right, with providing an interface, is the proper way.
Has there been progress on replacing Infinispan with Redis? We are currently running Keycloak dockers containers with external Infinispan with HA configuration. We are experiencing multiple issues that are still under investigation. Since there isn’t much documentation available on infinispan to address below issue we are looking at Redis as an alternate session store.
Memory Leaks and Cross-site configuration: we can see its heap memory keeps going up until the containers themselves become unhealthy and shut down. The test is run at 100 TPS and it took about 3 days to reach this scenario. With real-world prod traffic having slightly less than 1 TPS, we can expect that it will run out of heap memory and become unhealthy in about 300 days.
This heap memory seems to be taken up by org.infinispan.container.versioning.irac.IracEntryVersion$MapEntry objects, which increase from 127k records at 06-04T16:29:51 to 7,300k records at 06-05T12:48:03. According to Infinispan sourcesthose objects are related to async cross site replication functionality, which suggest that our cross-site configuration is not set properly.
Lost Sessions when Deploying Containers: We lose sessions when deploying new containers and shutting down old ones. With how we configure Infinispan, **[a session needs to exist inside 2 tasks] (for resiliency) and we have [minimum 3 tasks per region]
Infinispan Console not Accessible when Deployed: Infinispan exposes a single port (11222) to handle HTTP traffic (for the admin console) and TCP traffic (for Keycloak via the hot rod protocol). When we run this locally in the POC project, we are able to access both with no complication. In AWS however, we had to put the Infinispan cluster behind the NLB which exposes only TCP, so no HTTP traffic is able to get to our clusters, leading to the admin console being inaccessible.
There has never been an attempt to replace Infinispan with Redis, nor is this on the roadmap. Additionally, things have changed a lot since the initial date of this post…
Infinispan is working much more reliable, support by Keycloak has become better, sessions can now be stored in the database, …