Restricting client roles when using a scope from different clients

I have a number of OIDC Clients, each with multiple Roles and Full Scope Allowed disabled.

For each of these Clients, there is one Client Scope, with audience resolve and role mapper for the client’s roles. The Client Scope’s Scope has all of the client’s roles assigned so only users with at least one of the roles can use the scope and get the audience. Client A has Scope A as default, but other clients may also have scope A.

If a user authenticates to client A, all of A’s roles that the user has appear in tokens.

Client B also has scope A. But when a user authenticates to B, I would like to restrict which of A’s roles appear in tokens. What is the best way to do this?

One way is to remove all the roles assigned to Scope A, and instead assign roles to each client that uses scope A, in their dedicated" scopes. So if Client A has roles A1 and A2, client B only gets A1 assigned, so only A1 appears in B’s tokens. But this 1) makes tooling more complicated, because now I have to manage role assignments in many different places, 2) users authenticating to any client with scope A get audience A even if they don’t have any of A’s roles.

I was hoping I could leave the roles assigned to scope A, and in clients where I want to restrict roles, assign only the roles needed, and hope that leaves out the others – but it doesn’t.

Is there a better way to do this? Is it possible with fine-grained authorization?