We have been running a 3 instances Keycloak cluster on AWS EC2 with some modifications on the standalone-ha.xml file that comes with official Keycloak distribution. The instance type we are using is fairly large (c5.4xlarge) and the whole system is fairly stable.
Recently, our company has moved to using AWS Fargate, which means we will have much less powerful machines. During our load testing, we are now running 18 instances of Keycloak and we are seeing a small amount of errors. Some requests are taking a really long time to complete. CPU and network usage also become very unstable, even after the load testing is done.
Does anyone have any suggestions on areas of interest where we can look into to eliminate the issues? Or it would be really helpful too if anyone can point us to some information on how to configure/run a large Keycloak cluster.
Depending on your target number of users and sessions, I’d consider running an external infinispan cluster. I’ve been running ~15-20 keycloak instances on Fargate with 5 infinispan instances (r5.2xlarge) on EC2, and it’s much more stable than running everything on Fargate.
There have been some discussions here about config:
@jichen-amplify@xgp
Can you please let us know the kind of traffic that you are managing with your deployments? Just trying to understand what should be the scale of the infrastructure, when the peak load is expected to go like 1000 authentications per second.
Is that 1000 auths per second sustained? You’re really going to have almost 100 million auths per day!? That would certainly be one of the biggest deployments of Keycloak I’ve heard of.
The cluster I described above does ~200 auths per second peak load, but definitely not sustained. Most auths in one day is ~2 million.
No, its not sustained. It would be peak load that may be experienced intermittently.
So, we are integrating with a mobile app which will be used by about 15 to 20 million users. The app may send periodic notifications for some offers that it may have. During that period, we are expecting that lot of users may try to access the application, which would create a load on Keycloak to provide tokens to users who may need a new token or a new authentication.
In your scenario where you have about 200 auth / second, what kind of infrastructure did you need , interms of sizing.
OK, thanks a lot , this information helps us to decide the way forward.
Just one last question, can you please share what is the total user volume that you did cater to ?
@ashishtchaudhari, are you running Keycloak and Infinispan on Fargate or Infinispan on EC2 instance as suggested by @xgp ?
If you are running on Fargate, can you please share the container sizing details (vCPU and Memory) for Keycloak and Infinispan? and how many instances are you running all the time?
Hi Atul,
At this point we are running it over auto-scaling group of EC2 instances. We have decided to go with R5b instances with 2 CPU an 16 GB of RAM. Our performance testing so far have showed that we would need more memory to store session cache. Our Infinispan servers are embedded and not external as of now. We are still evaluating the best approach.
@xgp
We are facing issues with keycloak in certain scenarios, if the rolling restart of the server happens, there are certain issues related to refresh token is seen. We have offline tokens configured, but even then it seems that in some cases, even if the token sent is valid, keycloak returns an error stating that the token is invalid. By anychance, did you face an issue similar to this, if yes, any guidance around this will help.