Paged Reading of Users via GET /{realm}/users gets Progressively Slower

Getting advice
1

Keycloak Server Version: 11.0.3

I am attempting to read all the users from the Keycloak realm into our local sever via the Admin REST API.

There are ~17k users in this test. I have read them without paging (request them all at once) and I have read them using the paging option (first, max).

I selected a pagesize of 250 for the first test, and I see terrible performance the later pages.
Page 1: 1.2 seconds
Page 2: 1.328 seconds
Page 3: 1.465 seconds
…
Page 44: 14.691 seconds
Page 45: 14.617 seconds
Page 46: 20.773 seconds
Page 47: 21.681 seconds
Page 48: 24.139 seconds
Page 49: 43.202 seconds
…
Page 66: 58.424 seconds
Page 67: 62.42 seconds
Page 68: 61.841 seconds
Page 69: 62.961 seconds
Page 70: 67 seconds
Total time was ~23 minutes

The pages get progressivly longer and longer. If I switched to a page size to 5000, it runs faster, but still gets worse.
Page 1: 18.307 seconds
Page 2: 25.281 seconds
Page 3: 37.267 seconds
Page 4: 50.657 seconds
Total time was ~2.2 minutes

If I read them all at once, it takes even less time. About 59 seconds.

However, we will have customers/IDPs that will have too many users to read at once (150k+).

Any thoughts?

Thanks,
Chris

2

Short: I had a somewhat related problem and switched away from bulk/realtime user loading.

Idea: Depending on what information you need and what you need it for, you may be able extract this info from validated bearer tokens on demand. → trading “up-to-date user info in your DB” for “speed”

3

Thanks. We do that now, updating when a user logs into the site from the token. The issue is customers want all the licensed users in the system when the system goes live so users can be available to use for sharing, groups, etc even if thy have not yet logged in. So I need to bulk add them.

I seem to see much better performance for straight local Keycloak users than a Federated user base. I am still testing that but that might give me a clue. Such as the cache settings for User Federation having an impact.

4

An update. It looks like Keycloak is hitting the AD server for every user to make sure they exist. No matter my cache settings (it seems). I found out because I got an error reading the server once and hit a NULL pointer exception and saw the stack.

It seems like there needs to be an option to use the cache without hitting the server. In this case I don’t really care if stale users are added as I will remedy the situation later if needed.