X509 authentication with Keycloak-on-kubernetes via ingress

Hi Everyone,

Does anyone have an example config for x509 authentication w/ Keycloak on Kubernetes via an ingress endpoint? I have x509 working fine w/ a NodePort setup, but access via ingress fails and Keycloak cycles to the username/password form.

Here’s the logging:

18:37:54,474 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-2) AUTHENTICATE
18:37:54,474 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-2) AUTHENTICATE ONLY
18:37:54,474 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) processFlow: x509-browser
18:37:54,475 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) check execution: 'auth-cookie', requirement: 'ALTERNATIVE'
18:37:54,475 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) authenticator: auth-cookie
18:37:54,475 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Going through the flow 'x509-browser' for adding executions
18:37:54,475 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Going through the flow 'x509-browser forms' for adding executions
18:37:54,475 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Selections when trying execution 'auth-cookie' : [ authSelection - auth-cookie,  authSelection - auth-x509-client-username-form,  authSelection - auth-username-password-form]
18:37:54,475 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) invoke authenticator.authenticate: auth-cookie
18:37:54,475 DEBUG [org.keycloak.services.util.CookieHelper] (default task-2) Could not find cookie KEYCLOAK_IDENTITY, trying KEYCLOAK_IDENTITY_LEGACY
18:37:54,475 DEBUG [org.keycloak.services.managers.AuthenticationManager] (default task-2) Could not find cookie: KEYCLOAK_IDENTITY
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) authenticator ATTEMPTED: auth-cookie
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) check execution: 'auth-x509-client-username-form', requirement: 'ALTERNATIVE'
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) authenticator: auth-x509-client-username-form
18:37:54,476 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Going through the flow 'x509-browser' for adding executions
18:37:54,476 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Going through the flow 'x509-browser forms' for adding executions
18:37:54,476 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-2) Selections when trying execution 'auth-x509-client-username-form' : [ authSelection - auth-x509-client-username-form,  authSelection - auth-username-password-form]
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) invoke authenticator.authenticate: auth-x509-client-username-form
18:37:54,476 DEBUG [org.keycloak.services] (default task-2) [X509ClientCertificateAuthenticator:authenticate] x509 client certificate is not available for mutual SSL.
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) authenticator ATTEMPTED: auth-x509-client-username-form
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) check execution: 'x509-browser forms flow', requirement: 'ALTERNATIVE'
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) processFlow: x509-browser forms
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) check execution: 'auth-username-password-form', requirement: 'REQUIRED'
18:37:54,476 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-2) authenticator: auth-username-password-form

Any ideas?

Thanks

–John

1 Like

I have the same issue. Thanks for posting.

This seems like a very interesting challenge.

As you may know, mutual SSL authentication is not completely done by Keycloak. The application server or reverse proxy does the TLS part of it (connection establishment, path validation etc), puts the client certification inside the request (via request attribute, on application servers, or HTTP Request for reverse proxies) and then Keycloak does the rest of the validation and authentication logic.

In Kubernetes, by default, SSL termination is done at ingress level. You can do a passthrough, but it’s not a very common configuration.

So you’ll need to configure your ingress controller to do mutual SSL authentication.

If yours is Nginx Ingress (the open source, not the paid version), this guide may help you: Forwarding Client Certificates with NGINX Ingress | VMware Tanzu Developer Center. The client certificate will be put into ssl-client-cert HTTP Header.

Now, with the certificate available at that http header, you can now configure keycloak to extract it. NGINX certificate lookup provider.

This is how you can configure the x509cert-lookup SPI: Configuring providers - Keycloak

The certificate being extracted, the rest of x509 authentication flow is normal: Server Administration Guide

Let us know if that helped.

1 Like

@weltonrodrigo Thanks a bunch for the follow-up and excellent points! We are using Keycloak for x509 authentication for Jupyterhub on Kubernetes. Ingress w/ x509 is working fine for Jupyterhub as evidenced by the fact I am prompted for my certificate upon access to Jupyterhub and that Jupyterhub is able to extract the CN from the cert.

I’ve previously configured keycloak ingress per the instructions you linked from VMWare, so the answer is likely within the instructions for the x509cert-lookup provider you provided links to, so I’ll give all that a try.

Thanks again for your time and I’ll post an update later today.

–John

@weltonrodrigo thanks again for your time on this question → much, much appreciated.

I am attempting to adding the x509cert-lookup SPI as -D java args to no avail thus far. I’ve tried various syntax approaches and none of them is correct.

Please let me know if you see any issues w/ this args array:

-Dspi-x509-cert-lookup-nginx-enabled=true -Dspi-x509-cert-lookup-nginx-ssl-client-cert=ssl-client-cert -Dspi-x509-cert-lookup-nginx-ssl-cert-chain-prefix=USELESS -Dspi-x509-cert-lookup-nginx-certificate-chain-length=2

Thanks

–John

@hokiegeek2 Not sure if this SPI configuration mechanism exist in prior (non quarkus-based) versions. Which version are you running?

For the current version (18, and probably 17 will work too), you need to pass the parameters in the command line, as:

kc.sh start \
   --spi-x509-cert-lookup-nginx-enabled=true \
   --spi-x509-cert-lookup-nginx-ssl-client-cert=ssl-client-cert \
   --spi-x509-cert-lookup-nginx-ssl-cert-chain-prefix=USELESS \
   --spi-x509-cert-lookup-nginx-certificate-chain-length=2 \
   <your other parameters>

You should probably set those in the build phase of quarkus, ie: running kc.sh build --spi-x509-cert-lookup-nginx-enabled=true … as part of your Dockerfile. Or use the autobuild feature. See this: Running Keycloak in a container - Keycloak

@weltonrodrigo running 16.1.1 and I am deploying via the bitnami helm chart v keycloak-7.0.0

I think what is complicating things is that I don’t see a kc.sh script, so I am attempting to figure out where I can pass these params into.

–John

@weltonrodrigo cool, I am running docker 17.0.1 and your start command works:

Updating the configuration and installing your custom providers, if any. Please wait.
2022-06-21 16:31:45,324 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 10133ms
2022-06-21 16:31:49,184 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: false
2022-06-21 16:31:50,531 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-06-21 16:31:50,542 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-06-21 16:31:50,580 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-06-21 16:31:50,807 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.6.Final
2022-06-21 16:31:51,283 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: node_774039, Site name: null
2022-06-21 16:31:53,163 INFO  [org.keycloak.quarkus.runtime.storage.database.liquibase.QuarkusJpaUpdaterProvider] (main) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
2022-06-21 16:31:55,969 INFO  [org.keycloak.services] (main) KC-SERVICES0050: Initializing master realm
2022-06-21 16:31:58,157 INFO  [org.keycloak.services] (main) KC-SERVICES0009: Added user 'admin' to realm 'master'
2022-06-21 16:31:58,299 INFO  [io.quarkus] (main) Keycloak 17.0.1 on JVM (powered by Quarkus 2.7.5.Final) started in 12.776s. Listening on: http://0.0.0.0:8080
2022-06-21 16:31:58,300 INFO  [io.quarkus] (main) Profile dev activated. 
2022-06-21 16:31:58,301 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
2022-06-21 16:31:58,303 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.

kc.sh is only present in quarkus-based versions.

For wildfly-based (<17), you’ll need to use the standalone.xml based SPI configuration: Server Installation and Configuration Guide

For kubernetes, this can involve generating a custom image to run cli scripts as part of your docker build. Alternatively you can run those as part of the pod startup.

I suppose this is the way to run custom cli scripts in bitnami keycloak chart: Where to mount Keycloak CLI startup scripts · Issue #6963 · bitnami/charts · GitHub

Examples of scripts for SPI configuration: Server Installation and Configuration Guide

Something like (not tested):

initdbScripts:
  x509-auth.sh: |
    debug_execute jboss-cli.sh <<EOF
    embed-server --server-config=${KEYCLOAK_CONF_FILE} --std-out=discard
    batch
    /subsystem=keycloak-server/spi=x509cert-lookup/:write-attribute(name=default-provider,value=nginx)
    /subsystem=keycloak-server/spi=x509cert-lookup/provider=nginx/:add(properties={sslClientCert => "ssl-client-cert", sslCertChainPrefix => "USELESS", certificateChainLength => "2"}, enabled=true)
    run-batch
    stop-embedded-server
    EOF

thanks @weltonrodrigo! I was able to upgrade to 17.0.1 and I ran the build command to install the x509 spi:

h build --spi-x509-cert-lookup-nginx-enabled=true --spi-x509-cert-lookup-nginx-ssl-client-cert=ssl-client-cert --spi-x509-cert-lookup-nginx-ssl-cert-chain-prefix=USELESS --spi-x509-cert-lookup-nginx-certificate-chain-length=2

When I run kc.sh I don’t see that the config has been updated, so not sure if the spi is installed and configured (?):

As you may know, containers running inside kubernetes have a ephemeral file system. Changes won’t be available at next book.

I suppose you should run the kc.sh build as a initDbScript. Or use the autobuild feature in the command line.

To check if things are ok, I suppose you should check keycloak.conf

I noticed you didn’t set the --spi-x509-cert-lookup-provider=nginx

@weltonrodrigo yeah, just wanted to see the result of that command. Question → if I add the auto-build followed by the spi params, would that work:

@weltonrodrigo yeah, the auto-build param did not appear to do anything, 'cause the keycloak.conf file is unchanged:

So bottom line is that I have to build a docker image w/ the kc.sh build command w/ the params we discussed here, correct?

Not sure what is going on here.

Maybe spi configurations parameters are a runtime thing and not a configuration (build) phase thing? On the Quarkus distribution, some configurations (like database vendor) need the build phase, some not (like hostname).

In that case I’d try starting keycloak (kc.sh start) with those parameters and search the logs to see if it works. In theory, you can set those parameters as environment variables, but see SPI Configuration · Discussion #10311 · keycloak/keycloak · GitHub for examples of what works and what doesn’t.

Have you tried running it with KC_LOG_LEVEL=DEBUG?

@weltonrodrigo cool, good suggestions! I am currently running in DEBUG and when I grep for x509 I don’t see anything, so I don’t see evidence that passing in

--autobuild --spi-x509-cert-lookup-provider nginx --spi-x509-cert-lookup-nginx-enabled true --spi-x509-cert-lookup-nginx-ssl-client-cert ssl-client-cert --spi-x509-cert-lookup-nginx-ssl-cert-chain-prefix USELESS --spi-x509-cert-lookup-nginx-certificate-chain-length 2

configures the x509-cert-lookup spi

Tomorrow I am gonna kc.sh build to create the image and then test that

Thanks again for all your time and efforts today!

–John

Hmmm…I’m getting concerned the SPI is not being set. I am able to update the server config by enabling metrics and see that the config has updated:

keycloak@cc2ecc165bbf:/opt/bitnami/keycloak/conf$ kc.sh build --metrics-enabled=true
Updating the configuration and installing your custom providers, if any. Please wait.
2022-06-22 14:35:52,107 WARN  [org.keycloak.services] (build-17) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2022-06-22 14:35:52,609 WARN  [org.keycloak.services] (build-17) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
2022-06-22 14:35:55,851 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 5749ms
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

keycloak@cc2ecc165bbf:/opt/bitnami/keycloak/conf$ kc.sh show-config 
Current Mode: none
Runtime Configuration:
	kc.cache =  ispn (PersistedConfigSource)
	kc.config.args =  show-config (SysPropConfigSource)
	kc.db =  dev-file (PersistedConfigSource)
	kc.home.dir =  /opt/bitnami/keycloak/bin/../ (SysPropConfigSource)
	kc.http-enabled =  false (PropertiesConfigSource[source=jar:file:///opt/bitnami/keycloak/lib/lib/main/org.keycloak.keycloak-quarkus-server-17.0.1.jar!/META-INF/keycloak.conf])
	kc.http-relative-path =  / (PersistedConfigSource)
	kc.metrics-enabled =  true (PersistedConfigSource)
	kc.provider.file.keycloak-metrics-spi-2.5.3.jar.last-modified =  1652196241000 (PersistedConfigSource)
	kc.quarkus-properties-enabled =  false (PersistedConfigSource)
	kc.show.config =  none (SysPropConfigSource)
	kc.version =  17.0.1 (SysPropConfigSource)

But when I attempt to set the x509cert-lookup-provider, I don’t see that change:

keycloak@cc2ecc165bbf:/opt/bitnami/keycloak/conf$ kc.sh build --spi-x509cert-lookup-provider=nginx --metrics-enabled=true
Updating the configuration and installing your custom providers, if any. Please wait.
2022-06-22 14:40:15,392 WARN  [org.keycloak.services] (build-49) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2022-06-22 14:40:16,002 WARN  [org.keycloak.services] (build-49) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
2022-06-22 14:40:19,197 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 6114ms
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

keycloak@cc2ecc165bbf:/opt/bitnami/keycloak/conf$ kc.sh show-config 
Current Mode: none
Runtime Configuration:
	kc.cache =  ispn (PersistedConfigSource)
	kc.config.args =  show-config (SysPropConfigSource)
	kc.db =  dev-file (PersistedConfigSource)
	kc.home.dir =  /opt/bitnami/keycloak/bin/../ (SysPropConfigSource)
	kc.http-enabled =  false (PropertiesConfigSource[source=jar:file:///opt/bitnami/keycloak/lib/lib/main/org.keycloak.keycloak-quarkus-server-17.0.1.jar!/META-INF/keycloak.conf])
	kc.http-relative-path =  / (PersistedConfigSource)
	kc.metrics-enabled =  true (PersistedConfigSource)
	kc.provider.file.keycloak-metrics-spi-2.5.3.jar.last-modified =  1652196241000 (PersistedConfigSource)
	kc.quarkus-properties-enabled =  false (PersistedConfigSource)
	kc.show.config =  none (SysPropConfigSource)
	kc.version =  17.0.1 (SysPropConfigSource)

Is there another way to check that the spi has been set?

–John

I think you could try setting those parameters in the keycloak.conf and then starting the server to see if they are actually working as intendend.

Maybe we are missing some parameter to actually activate the x509cert-lookup SPI.

Try setting spi.x509cert-lookup.provider=nginx in the keycloak.conf along with the other settings.

I found this documentation, double check your parameters: Enabling client certificate lookup on keycloak 18

@weltonrodrigo okay, finally got keycloak 17.0.1 to get the spi lookup provider in the configuration via the env variable like this:

Double check the parameters, I had mistyped them x509-cert-lookup instead of x509cert-lookup.

cool, I have the config values set in the Docker container via a combo of env variables and passing in --autobuild:

1 Like