SSO for some users in a Group fails Azure

Hey,
I have configured Azure Active Dir to login to keycloak. And the steps used to configure it - Groups from Azure AD - #3 by Shryne

And it was working fine, until suddenly some users of the same group are not able to login. And when I checked the roles were removed from the user. What can be possible error here?
Any sort of help or pointers will be helpful. Thanks

Update:
After debugging, found that users having large AD Token, due to multiple groups are not able to authenticate.

Keycloak is deployed on Kubernetes.