Background:
Our keycloak is setup using a standalone cluster running in kubernetes (currently with 2 nodes), with an AWS RDS instance of Postgres for the database storage.
The federated user storage is hosted in Oracle and we are using a user with the following permissions in Oracle:
GRANT SELECT ON sys.dba_pending_transactions TO keycloak_user;
GRANT SELECT ON sys.pending_trans$ TO keycloak_user;
GRANT SELECT ON sys.dba_2pc_pending TO keycloak_user;
GRANT EXECUTE ON sys.dbms_xa TO keycloak_user;
GRANT FORCE ANY TRANSACTION TO keycloak_user;
We have implemented a UserStorageProvider SPI class that implements the following interfaces: UserStorageProvider, UserLookupProvider, UserQueryProvider, CredentialInputUpdater, CredentialInputValidator, ImportedUserValidation, UserRegistrationProvider, OnUserCache
The Issue
We are conducting load testing on one of our other applications, that conducts logins as part of the test. During the load testing we are seeing the following error (our code is in org.mycompany package) :
ERROR: deadlock detected Detail: Process 15848 waits for ShareLock on transaction 62946269; blocked by process 31658. Process 31658 waits for ShareLock on transaction 62946272; blocked by process 15848. Hint: See server log for query details. Where: while deleting tuple (2,1) in relation “fed_user_attribute”
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:120)
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:100)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:59)
at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:109)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:453)
at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:378)
at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1550)
at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1663)
at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1645)
at org.keycloak.storage.jpa.JpaUserFederatedStorageProvider.deleteAttribute(JpaUserFederatedStorageProvider.java:121)
at org.keycloak.storage.jpa.JpaUserFederatedStorageProvider.setSingleAttribute(JpaUserFederatedStorageProvider.java:138)
at org.keycloak.storage.adapter.AbstractUserAdapterFederatedStorage.setSingleAttribute(AbstractUserAdapterFederatedStorage.java:323)
at org.mycompany.keycloak.federation.data.model.MyCompanyUserAdapter.setSingleAttribute(MyCompanyUserAdapter.java:152)
at org.keycloak.storage.adapter.AbstractUserAdapterFederatedStorage.setEnabled(AbstractUserAdapterFederatedStorage.java:243)
at org.mycompany.keycloak.federation.data.model.MyCompanyUserAdapter.(MyCompanyUserAdapter.java:56)
at org.mycompany.keycloak.federation.LegacyUserProvider.createAdapter(LegacyUserProvider.java:171)
at org.mycompany.keycloak.federation.LegacyUserProvider.validate(LegacyUserProvider.java:213)
at org.keycloak.storage.UserStorageManager.importValidation(UserStorageManager.java:115)
at org.keycloak.storage.UserStorageManager.getUserByEmail(UserStorageManager.java:263)
at org.keycloak.models.cache.infinispan.UserCacheSession.getUserByEmail(UserCacheSession.java:386)
at org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:208)
at org.keycloak.authentication.authenticators.browser.AbstractUsernameFormAuthenticator.getUser(AbstractUsernameFormAuthenticator.java:173)
at org.keycloak.authentication.authenticators.browser.AbstractUsernameFormAuthenticator.validateUserAndPassword(AbstractUsernameFormAuthenticator.java:146)
at org.keycloak.authentication.authenticators.browser.UsernamePasswordForm.validateForm(UsernamePasswordForm.java:55)
at org.keycloak.authentication.authenticators.browser.UsernamePasswordForm.action(UsernamePasswordForm.java:48)
at org.keycloak.authentication.DefaultAuthenticationFlow.processAction(DefaultAuthenticationFlow.java:161)
at org.keycloak.authentication.AuthenticationProcessor.authenticationAction(AuthenticationProcessor.java:937)
at org.keycloak.services.resources.LoginActionsService.processFlow(LoginActionsService.java:311)
at org.keycloak.services.resources.LoginActionsService.processAuthentication(LoginActionsService.java:282)
at org.keycloak.services.resources.LoginActionsService.authenticate(LoginActionsService.java:266)
at org.keycloak.services.resources.LoginActionsService.authenticateForm(LoginActionsService.java:339)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:543)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:432)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:393)
at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:395)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:364)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
Download CSV File of the logs for the deadlock transaction
How can I troubleshoot the issue ?
Thanks!