Keycloak with Centralized Cache Store

Is it possible to replace Infinspan in Keycloak with a centralized cache store like Redis or Memcached?

It took us a while but we finally got Keycloak running in clustered mode using JDBC_PING on Amazon’s ECS cluster service. We have also played with several configurations like a single owner for the Infinisan caches or multiple owners and sticky sessions or non-sticky sessions. Even though sticky sessions has its downsides, like potentially losing sessions, we’d prefer to run this way as it is the most performant solution as caches do not have to be replicated to the entire cluster. One of the problems we’ve run into is that the AWS ALB does not seem to want to route new sessions very evenly when sticky sessions are turned on. Eventually one of the cluster nodes ends up dying under load (and all the associated sessions are lost).

All this would be much easier to configure if we could just cut out JGroups and Infinispan and have keycloak talk to Redis or Memcached while running in standalone mode. I do realize that performance will be slightly slower as objects will have to be serialized/deserialized, but the benefits of being able to dynamically scale the cluster without having to have users potentially re-login would be very beneficial to us.

Is this possible? Has anyone ever done it?

I saw this answer on SO, but I’m not interested in having to customize a bunch of SPIs to do this.

2 Likes

I just saw that Infinispan introduced a Redis Cache store back in 2015, but I’m not sure if it’s possible to swap this out in the

<subsystem xmlns="urn:jboss:domain:infinispan:9.0">

section of my standalone-ha.xml file.

https://infinispan.org/blog/2015/09/14/new-redis-cache-store-introduced-in/

Keycloak has been designed to work with Infinispan. There’s no way to (without rewriting many internal bits of Keycloak) to replace it.

What is exactly your problem. To be this sounds like you’re struggling with memory tuning.

Thank you for the reply. Will Keycloak only work with the in-memory clustered version of Infinispan or can I plugin the Redis version of Infinispan?

We are running Keycloak inside of AWS ECS with autoscaling. Running in standalone-ha cluster mode and using JBCD_PING for node discovery. We have run into two problems:

  1. when we turn on sticky sessions and set the number of owners = 1 the AWS ALB ends up sending most or all of the traffic to one cluster node and that cluster node eventually dies under load
  2. when we turn sticky sessions off and set the number of owners = the cluster size (e.g. 8). This works for us, but we can’t get the cluster to scale any larger than 8 nodes because of memory constraints and the cluster has to synchronize data with the whole cluster so transactions start taking too long. Also we can’t allow the cluster to autoscale like this b/c once we get past 8 it is not usable anymore.

I would prefer to do some testing with a shared cache instead of a replicated cache so I could turn sticky sessions off and not have to worry about the nodes discovering or talking to each other. The nodes would only talk to the database or shared cache.

-Chris

What I’d like to do is have the option to run Keycloak in different types of clustering modes. Stateless or Replication mode without the downside of possibly loosing sessions that comes with running Keycloak in sticky-session mode. Connect2ID allows for this type of clustering options as described here:

https://connect2id.com/products/server/docs/config/infinispan#supported-clustering-modes

It sounds like this would be a new feature request for Keycloak.

1 Like

Hi @csavory,
Need quick information. I am curious to know how many total number of Realms you are handling with 8 node domain cluster ? Also per Realm how many groups you created. I just want to size it so that it works for our scenarios too. Any ball park figures would really help.
We are in the process of creating a Domain Cluster for one DC. The backup DC we will build later. We handle 1500 Realms, 12 to 15 clients per Realm. Also 75 to 100 groups per Realm ? I would like to learn any lessons from you.

Thank you in advance for your kind help and time.

cheers,
Krish

keycloak x will attempt to solve some of this i guess.

Right ! But I have no idea when KeyCloak X is coming. Any help there ? Or till then, I would like to use the existing one but I want to size it, so I am looking for some sort of real world examples.

We have about 15 realms. 8 node cluster was about as high as we could get it, but I want to say that it never really worked that well. We tried to use both sticky sessions (recommeded) as well the shared infispan cache with groups of 2.

That why I was thinking a central cache option like a db or redis would be the better way to go. Even though it would sacrifice performance a little, scaling and clustering would be so much easier.

2 Likes

Thank you so much @csavory for the inputs. Redis as central cache would be good. But is there a way that you know of to do it successfully ?

Like some 800 realms on the Cluster. With about an average of 10,000 users per realm.

As far as I can tell from Keycloak - Blog - Introducing Keycloak.X Distribution, not even Keycloak.X supports Redis centralized caching. @kkcmadhu I’m not sure if keycloak x will even attempt to solve this, it seems like Red Hat is dead set on just keeping infinispan (I even searched their jira).

However, this line in the blog gives me some hope:

However, the configuration is now using Infinispan’s native configuration as opposed to using an abstraction as in the Wildfly Infinispan Subsystem.

I’m hoping this means we can just copy+paste configuration for external infinispan clusters backed by redis (I’m sure there are examples out there) into keycloak.x and finally make keycloak actually stateless.

It’s kind of a pipe dream at this point given that 1. Keycloak.X basically has zero documentation at this point, 2. There seems to be basically zero documentation/StackOverflow answers/examples for plugging in external infinispan clusters.

I’m really hoping someone at RH sees this and at least tries to migrate Keycloak.X’s session management to redis/memcached, because Keycloak being (implicitly) stateful is a BIG turn-off for me currently to the point where I’m evaluating alternatives.

For the time being we are still using Infinispan and JGroups for clustering and HA deployments.

The blog says this, but I still have not yet found roadmaps/jira tickets to indicate that RH is even thinking about making Keycloak stateless…

@csavory did you managed to find working clustering solution on AWS ECS?
We struggle with same problem. From some time our 3 nodes cluster became unstable and both - killing one task and deploying new version (3 new tasks) cause it to be unresponsive and we need to kill all tasks to make it responsive again. We tried to play with CACHE_OWNERS and sticky sessions but nothing really helps and ew have another downtimes caused by cluster not being able to synchronize properly. Do you have any hints how to make it more stable?


@slaskawi you heard our problems. I can see your both Keycloak and Infinispan maintainer. What do you suggest? With stateless Keycloak and basically simple Redis as centralized cache we woudnt have current issues. And currently we’re experiencing dontime after downtime so I would really appreciate Keycloak team member help here. I asked on google group some time ago but no one responded.