We recently upgraded from Keycloak 19 to Keycloak 22 and have observed significant performance degradation in our system. Specifically, CPU utilization has spiked to 90% under normal load. Our setup includes a 2-node cluster running in ECS Fargate, and we’ve been experiencing frequent container restarts due to the error: Terminating due to java.lang.OutOfMemoryError: Java heap space.
Here are the steps we’ve taken to address the issue:
- Increased Heap Space: We adjusted the initial and maximum memory settings using the following parameters:
-Xms1024m -Xmx3072m
However, this did not resolve the issue. The high CPU consumption and container restarts persisted.
- Memory Management Adjustments: We noticed changes in memory management parameters in the Keycloak 24
kc.sh
file and applied similar settings to our Keycloak 22 image:
-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:MaxRAMPercentage=70 -XX:MinRAMPercentage=70 -XX:InitialRAMPercentage=50
Unfortunately, these adjustments also did not mitigate the problem. The CPU usage remained high, and the container continued to restart.
- Heap Dump Analysis: We enabled heap dumps on
OutOfMemoryError
by setting-XX:+HeapDumpOnOutOfMemoryError
and analyzed the heap dumps using Eclipse Memory Analyzer. Our analysis indicated that Freemarker is consuming most of the memory.
Attached are the logs and screenshots from the Eclipse Memory Analyzer for your reference.
Class Name | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------
io.vertx.core.impl.VertxThread @ 0x69c3c4710 executor-thread-8 Thread | 136 | 1,16,72,22,944 | 31.68%
'- <Java Local> freemarker.core.Environment @ 0x69c6aaf18 | 296 | 49,05,33,824 | 13.31%
'- currentMacroContext freemarker.core.Macro$Context @ 0x69c6b0d98 | 48 | 49,04,93,592 | 13.31%
'- localVars freemarker.core.Environment$Namespace @ 0x69c6b4230 | 40 | 49,04,93,544 | 13.31%
'- map java.util.HashMap @ 0x69c6b4258 | 48 | 49,04,93,504 | 13.31%
'- table java.util.HashMap$Node[16] @ 0x69c6b4288 | 80 | 49,04,93,456 | 13.31%
|- [4] java.util.HashMap$Node @ 0x69c6b4378 | 32 | 49,04,85,912 | 13.31%
| '- value freemarker.core.AddConcatExpression$ConcatenatedSequence @ 0x76a445f70| 24 | 49,04,85,880 | 13.31%
|- [6] java.util.HashMap$Node @ 0x69c6b4358 | 32 | 7,104 | 0.00%
|- [1] java.util.HashMap$Node @ 0x69c6b6998 | 32 | 128 | 0.00%
|- [8] java.util.HashMap$Node @ 0x69c6b4338 | 32 | 104 | 0.00%
|- [15] java.util.HashMap$Node @ 0x69c6b42d8 | 32 | 32 | 0.00%
|- [11] java.util.HashMap$Node @ 0x69c6b42f8 | 32 | 32 | 0.00%
|- [10] java.util.HashMap$Node @ 0x69c6b4318 | 32 | 32 | 0.00%
|- [0] java.util.HashMap$Node @ 0x69c6b69b8 | 32 | 32 | 0.00%
'- Total: 8 entries | | |
----------------------------------------------------------------------------------------------------------------------------------------------
We would greatly appreciate any insights or suggestions on how to address these issues. Has anyone else experienced similar problems after upgrading to Keycloak 22? Are there specific configurations or optimizations that we might be missing?