How to have a group with User and Admin only?

I need to create a groups which have respective admins. These admins for their own group are responsible for adding members in their group. Is this feasible? If so how can I achieve it? I tried my hand at fine grain permissions but that didn’t work out. I am still able to create users in X group when I am passing a jwt of an admin in Y group. Since the role is admin the endpoint allows it, I thought keycloak will give me an error that the admin can not create a user in X group since the admin belongs to Y group.