Not possible to have user credentials in keycloak while federating to LDAP?

Is it possible to have user credentials in keycloak while federating to LDAP?
I have recently set up a keycloak server on the latest version (20.0.1 at the time or writing) and everything is working as expected. However, now I’m experimenting with LDAP federation (openLDAP) to provide legacy ldap authentication for some applications we have that do not support oidc. (mail stack for example).

The goal is to just have LDAP as a mirror for keycloak, so everything that happens in keycloak (single source of truth) should also be reflected in the LDAP directory. I have this working, and users are created in LDAP when I create them in keycloak, however the user credentials (in keycloak) seem to stop working. That is, the userPassword is updated in the LDAP directory (via a mapper on password->userPassword), and this actually works (I tested binding against the directory with that user / password combination). However, logging into keycloak with any of the accounts created in this manner is not possible as credentials in keycloak itself are missing. When adding credentials the pop up shows, it says it’s succesful, but the credentials never show up in the list.

Is this by design? Am I missing something? Is this perhaps a bug?