Get client specific roles in user-profile

I am using keycloak via pac4j-play library. I have created 2 separate clients in keycloak and each of the client has some roles defined in it. But if a user is assigned some roles in both the clients, then I am getting all the roles for that user in the user-profile, and not just the roles specific to the client I am connected to. The protocol used is SAML.
Can anyone suggest any fix?