Noticeable performance degradation on Keycloak 22 with a very simple Keycloak serving only client_credentials grant type

Getting advice
1

I’ve stumbled upon a very weird performance issue with Keycloak on the latest version, 22.

We have multiple deployments of Keycloak in production and we decided to upgrade to the latest version. Our current version is Keycloak 19 on Quarkus distribution, and it works great!

However, as soon as I upgraded the simplest deployment to latest version 22.0.4, the performance dropped significantly. I can provide many more details but the gist of it is this:

  • Run a couple of EC2 instances with Keycloak 19 Quarkus, running on edge mode behind a ALB. We only have between 10-20 clients and they all are used for service logins, with a very simple clients_credentials grant type. These are confidential clients with a simple secret (no JWT or certificates, just plain client_id and client_secret). Our typical load varies during the day between 30 to 90 requests per minute, we have around 30 client_credentials logins per minute (not a very big load).

However, with Keycloak 19 our p99 for request times varies between 20-30 milliseconds (a very decent result), compared to Keycloak 22 with p99 for request times varying between 100-200 milliseconds. This is a huge performance penalty, almost 10 fold worse. And the most interesting fact is that does not matter the mentioned load, the p99 was constant for Keycloak 19 and it is constant for Keycloak 22. So our load is not enough to worsen the response times.

However, the baseline is severely shifted for the worse. I am baffled and wondering if anyone noticed such a thing? Could it be the upgrade to Quarkus 3? Any help is greatly appreciated! We are afraid to upgrade the rest of the deployments as those are much heavy used with a huge variety of loads so if anything gets 10 times slower will be noticeable.

1 Like
2

Thanks for observing this with all the details! :+1:
If you really think this is an issue with Keycloak, I recommend to open at least a discussion thread here to discuss this with the Keycloak team itself. As this is a community forum, it’s not that likely that the responsible and/or important people read this here.

1 Like
3

You might also check to see if you can reproduce similar results with keycloak-benchmark. The setup is not trivial, but it would give maintainers and others the ability to test changes if the results are reproducible.

2 Likes
4

Thank you very much both for your response. I will open a discussion as there is a certain performance drop when jumping from Keycloak 21 to 22 and the only lead is the Quarkus bump

5

Hi, @stancristian88 !

Did you ever get this resolved? I’m looking to adopt KeyCloak for our application and would like to hear about your experience with this issue.

Cheers!