X509 authentication with Keycloak-on-kubernetes via ingress

Getting advice
61

@mikhust excellent and yeah, I have the ca.crt. One thing I don’t have in my config is this:

    - name: KC_HTTPS_CERTIFICATE_FILE
      value: "/etc/x509/https/tls.crt"
    - name: KC_HTTPS_CERTIFICATE_KEY_FILE
      value: "/etc/x509/https/tls.key"
62

@mikhust @weltonrodrigo I fixed the empty header error. I am now back to seeing this error: [32m12:52:16,321 DEBUG [org.keycloak.services] (default task-14) [X509ClientCertificateAuthenticator:authenticate] x509 client certificate is not available for mutual SSL.
Can either of you confirm what scenario(s) one would see this error?

63

I did not see such error before
may be this will help x509 login code flow not working in version 18 · Issue #12463 · keycloak/keycloak · GitHub

64

@mikhust @weltonrodrigo okay, making progress. It appears that, via ingress, the certificate being passed into keycloak is the tls cert. First, the error message I am getting now is as follows:

image

Logging shows the CN corresponds to the wildcard TLS cert, not the user making the request:

09:02:07,294 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-18) invoke authenticator.authenticate: auth-x509-client-username-form
09:02:07,294 DEBUG [org.keycloak.services] (default task-18) Extended Key Usage validation is not enabled.
09:02:07,294 DEBUG [org.keycloak.services] (default task-18) Certificate Policy validation is not enabled.
09:02:07,295 WARN  [org.keycloak.services] (default task-18) [X509ClientCertificateAuthenticator:authenticate] Unable to extract user identity from certificate.
09:02:07,295 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-18) new JtaTransactionWrapper
09:02:07,295 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-18) was existing? true
09

So…I think the final question is how to specify that the cert to load via an ingress workflow is the client cert as opposed to the TLS cert, because when I access keycloak via NodePort, the x509_cert_subject_distinguished_name is the user, whereas the x509_cert_subject_distinguished_name accessing keycloak via ingress is of the TLS cert

Also, I am using the bitnami keycloak helm chart with this values.yaml file

65

@weltonrodrigo @mikhust I upgraded our keycloak to 19.0.1 and am back at x509 authentication via NodePort working and x509 via ingress failing b/c the TLS cert is loaded instead of the client.

I added the keycloak command-line options:

extraEnvVars:
  - name: KEYCLOAK_LOG_LEVEL
    value: DEBUG
  - name: KEYCLOAK_EXTRA_ARGS
    value: "--https-client-auth=required --spi-x509-cert-lookup-nginx-enabled=true --spi-x509-cert-lookup-nginx-ssl-client-cert=ssl-client-cert --spi-x509-cert-lookup-nginx-ssl-cert-chain-prefix=USELESS --spi-x509-cert-lookup-nginx-certificate-chain-length=2"
proxyAddressForwarding: true

Do either of you know of the Java command line param needed to extract the correct, client cert when accessing keycloak via ingress?

66

I didn’t quite understand what “TLS cert” would be. Is that the server cert?

If it is, that’s pretty strange. There is a directive in nginx configuration where you choose what will be set on the headers. Maybe that is wrong?

A correct setting would be

proxy_set_header Ssl-Client-Cert $ssl_client_escaped_cert;

but you very probably wouldn’t need it to change those settings, as the default configuration for nginx ingress comes with that correctly set.

67

@weltonrodrigo yes, Keycloak logging shows that the CN is from the wildcard cert used for TLS when I access Keycloak via ingress. When I access Keycloak via NodePort, the CN corresponds to the client cert, again proved by logging in Keycloak.

68

@weltonrodrigo @mikhust thanks to both of you for all your help on this! I have x508 via ingress working with 19.0.1. Specifically, I am able to authenticate to the Keycloak admin behind an ingress via x509 authentication and I also have x509 authentication working with Jupyterhub behind ingress configured to access Keycloak behind ingress. So, mission accomplished, and, again, thanks to you both.

The remaining issue is that I am getting the following error in Keycloak:

2022-09-27 09:30:46,036 WARN  [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-22) Keycloak Truststore is null, but it is required !
2022-09-27 09:30:46,036 WARN  [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-22)   see https://www.keycloak.org/docs/latest/server_installation/index.html#_truststore
2022-09-27 09:30:46,036 INFO  [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-22) Impossible to rebuild end user cert chain : client certificate authentication will fail.

My hypothesis is that this is due to the fact that my jks file only has the root CA but not the intermediate CA. Do you think this is correct?

–John

69

It looks more like keycloak has no knowledge or correct path to the truststore

70

but wouldn’t that prevent Keycloak from being able to get the cert from the header? If the truststore was empty/missing seems like I’d get an error that there’s no cert in the header.

71

yeah, seems strange. Looking at the code, the keystore is used to verify the cert coming from the headers.

So I’m not sure why it is working :sweat_smile: