Hi we have setup a keycloak in kubernetes.
We extend keycloak:9.0.0 docker image to put in 3 theme jars, and 2 user-provider jars.
The only thing in the themes are css styling on top of keycloak theme.
For the providers, we only do lookup and password verification, since we intend to move them to keycloak and later unlink them and control login only with keycloak.
This is all tested and seems working fine, except that we see that CPU and Memory spikes when logging in, and when user should verify email, it seems too much to handle and we se a KILL signal in the logs.
We have added JAVA_OPTS
-server -Xms2048m -Xmx8192m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
For some reason, we see spikes at login, and specially with email_verify on the user. And after spikes, KILL signal appears.
What is the reason for these spikes?
CPU -> 100%
Memory usage to what is given ( 8GB )
How to solve this? We have use docker-hub recommended settings.
Seems related to having user-storage-providers, disabling them seems to get away from this problem.
They are basically implemented after the guidelines here: https://access.redhat.com/documentation/en-us/red_hat_single_sign-on_continuous_delivery/2/html/server_developer_guide/user-storage-spi#import_implementation_strategy
Adapted to use external rest services to lookup/verify user.
We ar trying to import users when logging in to keycloak, and it works, but with the result that after logging in, we see CPU starting to grow and eventually hit the limit, casing the pod to crash.
Eventually, I fixed this problem. I still don’t know why Keycloak went crazy though.
We are using REST apis to lookup and validate the user on login to import it to Keycloak.
For this we used Java 11 HttpClient
First I went with default
HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_1_1)
Then ops did some sniffing and then says that the service in accept env. is behind Nginx which is talking http2
So I changed to
HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_2_0)
Which made the problem go away.
Even if it worked with 1.1, that it getting and validating users, for some reason CPU went crazy in Keycloak. Why, no clue, seemed in heap dump that e got a lot objects of type ConcurrentLinkedQueue$Node when this happens