First there is a question of vocabulary. I guess that what you call “authorization server” is what is called PDP, Policy Decision Point, in OAuth. In other words, it’s where you send a token, and it evaluates policies, and returns permissions.
The term “authorization server” can also be used to describe the server that provides the token. Should it be the identity server? Not necessarily, it can just delegate identification to some other server, or take a token as input (token exchange). (to confuse matters, the endpoint for both usages is the same in Keycloak, it didn’t use to be)
So you want to use Keycloak as a PDP and PMP (Policy Management Point, where you define the policies).
I’m actually curious as to whether that’s possible, even though I tend to agree with others that it’s not the primary purpose of Keycloak. You would need to only reference stuff from the JWT in your policies, whereas Keycloak policies can actually fetch stuff from user profiles, groups etc.
I have found problems with Keycloak as a policy evaluator (PDP): oauth - Keycloak JavaScript APIs do not provide effective roles and attributes from groups - Stack Overflow
I have found problems with Keycloak as a token provider (as opposed to identity provider, see above): Token exchange (external to internal) and mapping don’t work together - Miscellanaeous - Keycloak
I have not found any problems with Keycloak as an identity provider, complete with federation.
The main policy engine that I hear about is OPA. For a comparison with Casbin, there is this… OPA vs Casbin · GitHub
(it reads a bit like a vi vs emacs page, they are very close 
)