Adding and removing roles based on SAML attributes

I’m just getting started with Keycloak and I have what I think should be a simple use case, but I’m not sure how to implement it. I have a SAML IdP set up and it is working fine. We get an attribute from SAML called “UserCategories” which is a list of category IDs. So I set up some mappers of type “Advanced Attribute to Role”. I mapped CategoryA to RoleA, CategoryB to RoleB, etc. This works fine at first. If a user’s UserCategories of CategoryA, then that user is correctly assigned RoleA.

The problem is when a user is re-assigned from CategoryA to CategoryB. Now when that user logs on, he is correctly assigned RoleB, but he retains RoleA from before. How can I fix this situation? I want a user to have those roles and ONLY those roles that are mapped in based on his current UserCategories. And if a user loses a category attribute value, then he should lose the corresponding role. What’s the proper way to do this?