Keycloak behavior after changing a filter in an ldap federation provider configuration

I have created an ldap federation provider configuration with an ldap filter for users and executed the import. Than I have changed the filter reducing the amount of users and executed the sync.
My expectation was that the users that are not in the new filter will be deleted but they are still there and able to login into the application behind the keycloak.
Why is it so and is there a way to delete the the users that are not in the new filter beside the “remove imported” that will delete all users (I have a lot and takes time).