Network response was not OK

All of the sudden, when trying to look for a user profile I get this error:
image

Upon taking a look at the logs I see this:

2024-09-12 09:00:09,289 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-24) Uncaught server error: java.lang.IllegalStateException: Duplicate key modifyTimestamp (attempted merging values org.keycloak.userprofile.AttributeMetadata@c933017c and org.keycloak.userprofile.AttributeMetadata@c933017c)
	at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.keycloak.userprofile.DefaultAttributes.getUserStorageProviderMetadata(DefaultAttributes.java:363)
	at org.keycloak.userprofile.DefaultAttributes.configureMetadata(DefaultAttributes.java:340)
	at org.keycloak.userprofile.DefaultAttributes.<init>(DefaultAttributes.java:90)
	at org.keycloak.userprofile.DeclarativeUserProfileProvider.createAttributes(DeclarativeUserProfileProvider.java:112)
	at org.keycloak.userprofile.DeclarativeUserProfileProvider.createUserProfile(DeclarativeUserProfileProvider.java:139)
	at org.keycloak.userprofile.DeclarativeUserProfileProvider.create(DeclarativeUserProfileProvider.java:117)
	at org.keycloak.admin.ui.rest.UserResource.getUnmanagedAttributes(UserResource.java:51)
	at org.keycloak.admin.ui.rest.UserResource$quarkusrestinvoker$getUnmanagedAttributes_0c18de18932405afb8d7dd47029ca6affeb63ce8.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

I have no clue how to solve this… I would greatly appreciate any help as this is a production instance

My Keycloak setup is as follows:
2 virtual machines running a keycloak container each that talk to each other via a shared cache infinispan. The database is a postgresql running in a virtual machine. The users in my realm are connected via ldap, and I have other realms where this ldap is connected and the user profile can be seen just fine.

Keycloak version 24.0.5