SAML2 Login for LDAP Users (custom flow / automatically link users)

I’ve got a user directory that is configured as LDAP (ActiveDirectory) provider in the “User Federation”.
When I open /auth/realms/master/account/#/personal-info, I can succesfully authenticate against KeyCloak with any credentials from this directory and can access my account-info which is read from the ldap (username, email, name).
I then added a SAML2 IdP in the “Identity Providers” section. The SAML2 IdP uses the same LDAP as user directory.
When I open /auth/realms/master/account/#/personal-info again and use the IdP as Login method this time, I get redirected to the IdP, login there, get redirected back.
KeyCloak then shows me my username and asks me to update my account information. I keep the username as is and select Submit. KeyCloak then tells me that the username already exists and asks me if I want to connect both accounts.
This is due to the default “first broker login” flow.
By this, I think, I verified that both, LDAP and SAML2 Login work on their own and both use the same username.

Now I want to achieve that users that log in via the SAML IdP get linked to their LDAP Account automatically (via same username).
The LDAP should be the user directory, but the external SAML IdP should be used for the Login.

How do I do this? I tried creating my own Login Flow as documented here: Server Administration Guide
I’ve added “Automatically Set Existing User” and “Create User If Unique” (I don’t want this, but I read it may help, but it doesn’t) to my custom flow, both as “Alternative”.
When I now log in via the IdP, I get “Invalid username or password.”.
KeyCloaks console logs this:

WARN  [] (default task-208) type=IDENTITY_PROVIDER_FIRST_LOGIN_ERROR, realmId=master, clientId=account-console, userId=null, ipAddress=, error=invalid_user_credentials, identity_provider=testidp, auth_method=openid-connect, redirect_uri=https://*/auth/realms/master/account/#/personal-info, identity_provider_identity=testuser, code_id=61*9a, authSessionParentId=61*9a, authSessionTabId=-i*I

There is really not much info, but I think Keycloak is not looking up/finding the user in the LDAP directory, which it successfully did before.

I don’t really know how to proceed, I cannot find much information about this.

P.S. Why am I doing this? The IdP I use only supports SAML. I want to use KeyCloak to also provide OIDC, but still want to keep the existing login.