[RHEL 8.4] "Failed to generate keys" on startup

Hi all,

I have been tasked with deploying KC on-prem for a client. The OS is RHEL 8.4 and Docker/podman is not permitted (much to my chagrin), therefore I am doing an old-school deployment directly on the box.

When trying to boot up KC on the RHEL server, it crashes with the following stacktrace:

17:11:00,204 FATAL [org.keycloak.services] (ServerService Thread Pool -- 58) Error during startup: org.keycloak.component.ComponentValidationException: Failed to generate keys
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.keys.GeneratedRsaKeyProviderFactory.generateKeys(GeneratedRsaKeyProviderFactory.java:123)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.keys.GeneratedRsaKeyProviderFactory.validateConfiguration(GeneratedRsaKeyProviderFactory.java:103)
        at org.keycloak.keycloak-model-jpa@15.0.2//org.keycloak.models.jpa.RealmAdapter.importComponentModel(RealmAdapter.java:2020)
        at org.keycloak.keycloak-model-jpa@15.0.2//org.keycloak.models.jpa.RealmAdapter.addComponentModel(RealmAdapter.java:2000)
        at org.keycloak.keycloak-server-spi-private@15.0.2//org.keycloak.models.utils.DefaultKeyProviders.createRsaKeyProvider(DefaultKeyProviders.java:56)
        at org.keycloak.keycloak-server-spi-private@15.0.2//org.keycloak.models.utils.DefaultKeyProviders.createProviders(DefaultKeyProviders.java:36)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.managers.ApplianceBootstrap.createMasterRealm(ApplianceBootstrap.java:90)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.resources.KeycloakApplication$3.run(KeycloakApplication.java:201)
        at org.keycloak.keycloak-server-spi-private@15.0.2//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:250)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.resources.KeycloakApplication.bootstrap(KeycloakApplication.java:172)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:136)
        at org.keycloak.keycloak-server-spi-private@15.0.2//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:250)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.resources.KeycloakApplication.startup(KeycloakApplication.java:128)
        at org.keycloak.keycloak-wildfly-extensions@15.0.2//org.keycloak.provider.wildfly.WildflyPlatform.onStartup(WildflyPlatform.java:36)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:114)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:152)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2835)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:376)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.spi.ResteasyDeployment.startInternal(ResteasyDeployment.java:288)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:98)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:140)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:42)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:305)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:145)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:588)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:601)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:829)
  at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: org.keycloak.common.util.PemException: org.bouncycastle.util.io.pem.PemGenerationException: unknown object passed - can't encode.
        at org.keycloak.keycloak-common@15.0.2//org.keycloak.common.util.PemUtils.encode(PemUtils.java:141)
        at org.keycloak.keycloak-common@15.0.2//org.keycloak.common.util.PemUtils.encodeKey(PemUtils.java:114)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.keys.GeneratedRsaKeyProviderFactory.generateKeys(GeneratedRsaKeyProviderFactory.java:121)
        ... 50 more
Caused by: org.bouncycastle.util.io.pem.PemGenerationException: unknown object passed - can't encode.
        at org.bouncycastle.bcpkix@1.68.00.0//org.bouncycastle.openssl.MiscPEMGenerator.createPemObject(Unknown Source)
        at org.bouncycastle.bcpkix@1.68.00.0//org.bouncycastle.openssl.MiscPEMGenerator.generate(Unknown Source)
        at org.bouncycastle.bcprov@1.68.0//org.bouncycastle.util.io.pem.PemWriter.writeObject(Unknown Source)
        at org.bouncycastle.bcpkix@1.68.00.0//org.bouncycastle.openssl.jcajce.JcaPEMWriter.writeObject(Unknown Source)
        at org.bouncycastle.bcpkix@1.68.00.0//org.bouncycastle.openssl.jcajce.JcaPEMWriter.writeObject(Unknown Source)
        at org.keycloak.keycloak-common@15.0.2//org.keycloak.common.util.PemUtils.encode(PemUtils.java:135)
        ... 52 more

I have a hunch that it might be related to FIPS mode being enabled on this RHEL instance (also something that is a hard requirement of the client). I am going to try and replace the BouncyCastle jars with FIPS versions and see if that does anything but I also wanted to post here in case anyone can guide me in a better direction.

FYI, I have tried both Java 8 and 11 to the same effect. I’ve also tried it with a fresh, clean version of KC without any modifications.

Thanks!