How to get SpringBoot adapter not to look at the bearer token for roles?

We have many microservices each of which may have up to 100 or more roles. When the user logs into the UI the token contains all the realm roles as well as the client roles the user has access to.
I have read many articles that having this amount of roles is not right but we can’t do anything at this stage and our system has been going on for a while.
Now, I know how to remove client roles from the access token by modifying Client Scopes > roles > client roles but if you do so, on the backend the Spring adapter fails with an error

{"error":"invalid_scope","error_description":"Requires uma_protection scope."}

Then I tried to include uma_protection role from that client in a composite realm role and map my user map to that realm role but still no luck (note that the realm roles are still appearing in the token)

Then next I added a new mapper in Client Scope > roles > Mapper and added uma_protection as hardcoded role for that client but now the error is different:

{"error":"access_denied","error_description":"not_authorized"}

What I don’t understand is that if Spring adapter tries to get RPT token why is the result different? How does it know that I meddle with the token because ultimately it gets the right scope from the Keycloak server.
Can anyone please help me to solve this issue. Have I missed anything? Any configuration I haven’t done?
Thanks

By default, spring adapter will only consider realm roles.

To also use client roles:

keycloak.use-resource-role-mappings=true

Thank you so much. I actually have a different problem. Maybe my question isn’t clear. The problem we are having is that the roles are increasing the size of the token and therefore they request/response fails because of exceeding the infrastructure buffer size. We already increased the buffer size to 6k but this won’t help.
So to fix that problem, we tried to remove the client roles from the token but then we get 403 for any request because it seems Keycloak only looks at the roles in the token instead of evaluating at runtime when it receives RPT request.
So this is the problem that I want to solve.