Problem with the new "...Stream" methods of the UserQueryProvider interface

Extending the server
1

Hi everyone,

Context: We are in the process of migrating our legacy authentication system to the Keycloak (KC later in the text), and we are using our custom UserStorageProvider to provide users from our legacy database to the KC.

Note: We’re talking about a million users in one realm. There’s no possibility to use many realms at this moment.

We’ve come to a few problems while migrating our custom UserStorageProvider from the KC 11.0.3 to 12.0.2 version. Old query methods from the UserQueryProvider interface are marked as deprecated and advised to be replaced with the new methods containing the “Stream” suffix.
We’ve done as instructed and the first problem that we encounter was an “out of memory” error. Digging through the KC source code we think that we found the culprit.

Here it is: UserStorageManager_12.0.2.

The arguments firstResult and maxResults are no longer passed to the searchForUserStream method of our custom provider (lines 410 and 412), so our query has no limit anymore and fetches all the data from the DB causing the memory overflow.

This was not a problem on version 11.0.3: UserStorageManager_11.0.3.

We’ve managed to circumvent this by limiting our ResultSet to fetch the 1000 records per DB roundtrip, but now the fetching takes a lot longer than before (many roundtrips to the DB) and this slows down the listing of the users on the KC Admin app to almost 10s per page.

Any ideas of what could we do to improve this? Or we should stick to version 11.0.3?

Thanks for any help,
Regards.

1 Like
2

@gmilan Did you find any solution for this problem.
I am also facing the same issue, I have a large user set and the getUserStream is invoked by keycloak without firstresult and maxresult. The screen gets hanged.
The issue is still noticed in 15.0.2 version.

I wanted to understand is this an expected behavior ? am I missing something ?

The only workaround I could think of is to store the hardcode the limit of records from query as of now.

3

We just return an empty list there and continue without that feature, since that method is only used for listing users on the admin page; going this way none of them is listed (we made that trade-off in order to move forward).

4

@gmilan
Thank you for your response.
Yes seems like the only way. We cannot show incomplete users at it would be false.
The admin can search an individual user using the search functionality from the UI and he always can refer to the database for full users list.

5

Exactly, and the important thing is that user authentication is not affected at all by this trade-off.