hey,
we running keycloak 10 on kubernetes and have 300.000 session.
We configured distributed cache with entries living on 2 out of 6 pods.
When we deploy a new release the swrver starts in 1 minute but it then takes 4 minutes to sync the infinity sessions on the pod that is starting. During that time all pods noticeably have delays (or locks?) and users experience delays of minimal 10 seconds. This happens on every pod restarting 1 by 1. This means half an hour our site performance terrible.
Does anyone experience this like us ? we started with very old keycloak version and this effect was always there. Creating issue at keycloak redhat wasn’t realy picked up.
Does any have advice maybe there is some different configuration running better ?
Tried already less pods, distribution on all pods, more cpu etc, without any effect.
We would love to have hazelcast cache since thats what we use in other solutions but there is no option for this in keycloak. Why does it take 4 minutes to sync 300.000 sessions, hazelcast syncs a full load in 20 seconds for million entries.
1 Like
Hello
We had the same problem. We had 300 000 offline sessions and our keycloak start up after 3-4 minutes. We increased max-pool-size to Postgres connection. Default value - 20 and we increased to 500. After that our 300 000 sessions download after seconds
Hope this helps
hey,
thanks for your reply.
In our situation its (unfortunately) not the database since we do not use offline sessions. We have 300.000 user sessions that are only in memory. Actually we would like to persist them in mysql but that doesnt work correctly, a feature story for this is also not picked up (strange they do not find this important…).
What is the advange of offline sessions ? i dont think thats an alternative for us …
Anyway thats why it is strange it takes 4 minutes everything is in memory. during startup infinispan sometimes throws timeout exceptions but they clear up after running several minutes. From beginning we have had infinispan issues i think its not a good caching system, i hope with infinispan 12 (we are now on 10, keycloak 10), it will be better since they finish the blocking to nonblocking project.
Now we have 70 000 active sessions and 525 000 oflline sessions. Today we will see time of load, connection and download in cluster. We have 4 pods in cluster with CACHE_OWNERS=2
We decide, that save sessions in-memory is not very reliably and after that discover oflline sessions.
Our mobile app save offline refresh token and after our restart we get it from database and don’t use active sessions
We searched solution for in-memory sessions problem, but didn’t find anything. We thought that we can create remote Infinispan, but there aren’t docs anywhere about that
wow thats very helpfull, we also where discussing remote infinispan server.
to be honest we do not 100% understand when and how to use offline sessions, we dont use them now. We have 70.000 app users and 240.000 web users. Is it also possible to use offline sessions for users on web ? the 240.000 users is in one month time so during day i thing around 10.000. But does offline have same functionality or do you have to switch users back from offline to active in memory users ?
hey again,
is it possible for our webapplication to keep using online sessions (with much shorter idle time), but also creating offline token… So active users still use online sessions, but if user comes back next day, the online session is expired and we log ussr in with the offline token, but with this offline token we create an online session again?
Hi
As i understood, offline token is just a kind of refresh token, which expires after time you set in Admin console (like Offline Session Idle option) and saved in Persist DB
You can create online session from offline refresh token with refresh REST Endpoint
I think web cannot use offline refresh tokens, because after restart of browser you have new application without any information of previous sessions. And i think it is not safe enough
We use offline refresh token because our users can enter into app after month and we don’t believe in Inifinispan 
You can read abouth offline refresh token here
Offline Refresh Token.)
And Rvanderboom, do you know about HAL Management console? You can see there many settings and statistics of Keycloak. Database connection pools, WEB server listener workers, average response time and many other option
It is located on http://localhost:9990 on default. I suggest to enter to this admin console too
thanks a lot, we definitly going to hve look at HAL console, didnt hear of it.
but for web you can also store/keeo offline token in the web application and restore online session from this right ? Specially if we would do this only if no online session is available. So putting online session on short idle time to reduce in memory session and restore with the offline token if user is back i.e. next. day ?
Hi
If you can store offline token in web it is possible to restore session the day after (and after restart of keycloak too)
Hey,
followup – we implemented offline sessions (and use online only for very active users)…
My question is, the offline sessions are also kept in memory, and to avoid same memory or performance issues, is it possible to limit the offline sessions in the infiapan cache ??
So for example only max total 5000 offline sessions (that are about the active users a day), so the other are only in the database ??
Hi
As far as i know, you cannot limit offline tokens per realm or client (in infinispan or database)
Why do you think, you will have performance issue? Now we have 1 300 000 ffline tokens and don’t have any issues with perfomance (only startup problem )
ok, good to hear you have no performance issues. But online sessions in cache are blocked during deployment (restart of pod) because infinispan is blocking (in next upcoming releases it will be made nonblocking). So offline does not have this issue ? Is that because its loaded from database in stead of through syncing like in online sessions ? (online needs to totally reorganise via pod to pad transfer, maybe offfline isn’t ?)
But i thought with infinispan you could configure a cache with eviction LRU and set a max-entries in there ?
i beleave the user caxhe is also on 5000 max entries, meaning its caches 5000 entries and al otheres are retrieved from database (and replaced in cache bases on LRU strategy) ?
do offline sessions take same or less memory space then online sessions (so per session…)?
Sorry but i don’t know answer for this questions
I understand, there is to little documentation about which configuration setups are possible and how it can be used :-<
No offlinesessions are distributed (i have also cluster) but according to infinispan you can set to local-cache (like users are local cached with max-enteries). Thats what i want but i do not know if its supported and working … :-<
You also have same setup with distributed cache for offine sessions right ?
Yes. We have same setup, but we don’t change cache config. Cluster of 4 pods and 2 CACHE_OWNERS
We setup grace period of 2 minutes. When we deploy new version of our docker container 2 new pods startup and 1 old dies. New pods connect to cluster and download info from previous version pods.
After 2 minutes 2 new pods startup and 2 old dies and again connect to cluster. After 2 minutes las old pods dies
With this setup we don’t have downtime
ok dank, dat is positief.
Ik had voor eerst in testomgeving wel een duplicate violation exception op de primary key van de offline session (mysql) insert.
ik had wel de owners van offline op 1 staan en van online op 2. ik las ergens dat dat niet moet.
Heb ik nu aangepast hopelijk is het dit…
ik heb verder geen configuratie voor de db datasource gedaan, dus weet niet hoe keycloak vanuit meerdere pods deze inserts en updates regelt…