UserStorageProvider SPI Implementation and Caching

I have been working on a UserStorageProvider SPI implementation connected to an external store/db. Have most things working as I would expect however I am a bit confused on the general flow of calls and the idea of the user cache as it is involved here. Most importantly I see two things that I would love to better understand:

  • When a user initially auths I am seeing two calls over the provider to the remote store. A LOGIN and then a CODE_TO_TOKEN (or so it appears. Will expand on in SPI flow under this)
    • We initially authenticate via a user attribute (which ive come to understand means the user is not cached YET)
    • there appears to then be a call to get user by id, which thankfully has the user already cached in a local in memory map within the SPI so this returns quite quickly. I then see the OnCache callback so I assume keycloak has then cached the user
    • A little bit later I see another get user by id call come in (which is the CODE_TO_TOKEN event I believe) and this call again reaches out over to the remote store.
    • Am I wrong to think this later call should just use the cached user in keycloak?
  • No matter what I set the caching policy to for our provider I do not see a change in the behavior laid out above.

Any guidance or further explanation would be greatly appreciated (edited)

I was able to figure out what was happening in my case and figure I would post back here in case anyone else comes across a similar issue.

Two things were compounding to require there to be a second call in what ill just refer to as the “token” phase.

Firstly, it appears attributes and other fields of the user model are lazily loaded/cached, so depending on when they are asked for, if the current user model isnt available they will be retrieved anew. This included groups, stored credentials, roles, etc.

The second finding was that in the UserAdapter/CachedUserSession if a role is found in the user model that is not currently known to keycloak, the cached user will be marked for invalidation and the delegate will instead be used to fetched anything else about the user after that point.

What was happening for me was the auth phase would retrieve and cache the user as expected, however role retrieval would mark the user for invalidation. Upon token phase when I needed to use attributes or roles for mapping in the token, the invalidation would cause the SPI to be hit again to go pull the user. (My roles were coming only from the SPI I had not been creating/saving them in keycloak).

To get around these two findings I have made two main changes to my SPI and have found caching is now working as expected:

  • Saved the roles I know can be returned by my SPI and backend DB in keycloak. Mapping the returned strings into the now saved roles by name.

  • During the auth phase in a custom authenticator I am now pre-loading attributes, groups, and roles so that I know they have been loaded into the cached user, and will be available when needed in the token phase for attribute mapping and roles in the token.

  • Additionally, as Ive seen multiple times referenced here and in examples, I also have a data map in my SPI implementation to act as a tiny “local” cache for the life of the initialized SPI instance, to store any user models retreived during the transactions. By doing this I ensure the user model that I have fetched is quickly available for any of the “lazy” loading mentioned above.

With all of that said I now see only a single call over to my remote service to fetch a user, and can get through both an auth flow and a token phase without making a second call over, which greatly improves performance.

I hope this can be user to anyone else, as it sure took me a while to figure out what was going on! Now onto managing invalidation in a way that makes sense for our use case and service.

Thank you so much for this information PaulOJ! I have implemented the third bullet point but I’m not sure how to implement the the first two. I’m not using a custom authenticator, my user store SPI is only implementing a CredentialInputValidator. Can you provide some details on the custom authenticator SPI and how you are preloading the attributes, groups and roles? Is the first bullet point implemented in your UserAdapter ? Thank You

So I am not as sure how caching and flow will work for you if you are only implementing the CredentialInputValidator interface as I do not believe that returns a user model, but maybe I am misremembering. As for the authenticator and preloading, all that was done there was in the overridden authenticate method I did the following:

final UserModel user = context.getUser();

//pre-load data that will be needed later in token phase
user.getAttributes();
user.getRequiredActionsStream();
user.getGroupsStream();
user.getRoleMappingsStream();

Additionally, the:

    @Override
    public boolean requiresUser() {
        return true;
    }

Will be needed in that authenticator to ensure you get a user.