You print has the spi.x509.cert.lookup I believe it should be spi.x509cert.lookup (note the dot between x509 and cert).
@weltonrodrigo and the x509certlookup is configured and running. But…Keycloak is saying keystore is null or empty:
@weltonrodrigo
and it’s odd (I think) that the keystore is empty/null 'cause the startup logging I think looks good:
@weltonrodrigo okay, I reverted back to 16.1.1 and realized the startup logging I show here is the same irrespective of whether the keystore and truststore load.
Specifically, in 17.0.1, the keystore and truststore both fail to load, which results in this error when I attempt to access the keycloak admin panel via x509:
Interestingly, the very same keystore and truststore load fine in 16.1.1:
^[[0m^[[32m10:38:34,188 DEBUG [org.keycloak.truststore.FileTruststoreProviderFactory] (ServerService Thread Pool -- 65) Trusted root CA found in trustore : alias : caroot | Subject DN : CN=Testing Root CA-1
^[[0m^[[32m10:38:34,193 DEBUG [org.keycloak.truststore.FileTruststoreProviderFactory] (ServerService Thread Pool -- 65) File truststore provider initialized: /opt/bitnami/keycloak/certs/keycloak.truststore.jks
@weltonrodrigo is there a reason keystore/truststore would load in 16.1.1 and not 17.0.1? Also, since --autobuild does not work for 16.1.1, do you know of a way to load the nginx x509cert-lookup?
I basically need to replace the nginx x509cert-lookup with nginx:
@weltonrodrigo now I can get the nginx x509cert-lookup attempt to parse from the incoming HTTP header the cert, but message says header is empty:
I need to figure out the header name keycloak is expecting
The HTTP Header "" is empty is telling, because there is a parameter to set the header name:
sslClientCert
For wildfly-based that would be:
/subsystem=keycloak-server/spi=x509cert-lookup/provider=nginx/:add(properties={sslClientCert => "ssl-client-cert", sslCertChainPrefix => "USELESS", certificateChainLength => "2"}, enabled=true)
For Quarkus-based would be:
--spi-x509cert-lookup-nginx-sslclientcert=ssl-client-cert
Not sure about the camelCase to argument mapping in this case. That could be --spi-x509cert-lookup-nginx-ssl-client-cert=ssl-client-cert
Maybe you can enable request dumping to see if the header is really present:
Wildfly:
/subsystem=undertow/configuration=filter/custom-filter=request-logging-filter:add(class-name=io.undertow.server.handlers.RequestDumpingHandler, module=io.undertow.core)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=request-logging-filter:add
Quarkus (not sure):
quarkus.log.category."org.apache.http".level=DEBUG
Environment variable for that is (note the double underscore): QUARKUS_LOG_CATEGORY__ORG_APACHE_HTTP__LEVEL=DEBUG
This thread hints that nginx maybe setting the cert under another header name: auth-tls-pass-certificate-to-upstream does not work with https · Issue #3511 · kubernetes/ingress-nginx · GitHub I’d try the header name Ssl-client-certificate.
@weltonrodrigo looks like I have to stick with 16.1.1 for now. At this point I am stuck on this error:
11:56:31,737 WARN [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookup] (default task-2) HTTP header "" is empty
11:56:31,737 DEBUG [org.keycloak.services] (default task-2) [X509ClientCertificateAuthenticator:authenticate] x509 client certificate is not available for mutual SSL.
Here are the ingress-related env variables I have:
- name: KEYCLOAK_LOG_LEVEL
value: DEBUG
- name: KEYCLOAK_X509CERT_LOOKUP_PROVIDER
value: "nginx"
- name: KEYCLOAK_X509CERT_LOOKUP_SSL_CLIENT_CERT
value: "ssl-client-cert"
- name: KEYCLOAK_X509CERT_LOOKUP_SSL_CERT_CHAIN_PREFIX
value: "USELESS"
- name: KEYCLOAK_X509CERT_LOOKUP_SSL_CERTIFICATE_CHAIN_LENGTH
value: "2"
And here’s my nginx ingress setttings:
nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
nginx.ingress.kubernetes.io/auth-tls-secret: demo/ingress-test-tls-secret
nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Accept-Encoding "";
proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
proxy_set_header proxy-ssl-client-subject-dn $ssl_client_s_dn;
proxy_set_header proxy-ssl-client-issuer-dn $ssl_client_i_dn;
proxy_set_header proxy-ssl-client-verify $ssl_client_verify;
proxy_set_header ssl-client-cert $ssl_client_cert;
nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
Do you see any errors in my configuration?
–John
Seems fine.
Not sure tough if the KEYCLOAK_X509CERT_LOOKUP_* env variables work for wildfly-based SPI configuration. I believe you’ll need to set those using a jboss cli script.
Obviously if you see nginx-provider loading in the logs, it should be OK.
So I’d try to check if the cert is really in the header.
You could point the ingress to another pod running a HTTP dump container, something like GitHub - daime/http-dump: Dumps HTTP requests
Also, I believe new versions of the nginx ingress controller ignore nginx.ingress.kubernetes.io/configuration-snippet for security reasons.
@weltonrodrigo gotcha, cool, thanks! I do know that KEYCLOAK_LOG_LEVEL and KEYCLOAK_X509CERT_LOOKUP_PROVIDER work because the log level is set to debug and the
Nginx x509cert-lookup is registered correctly:
@hokiegeek2, made a test using daime/http-dump image.
A valid nginx.ingress.kubernetes.io/auth-tls-secret is necessary to turn mutual-ssl on.
Even if you make it mandatory by setting nginx.ingress.kubernetes.io/auth-tls-verify-client: on nginx-controller will ignore it if the nginx.ingress.kubernetes.io/auth-tls-secret annotation is missing and will happily ignore mutual-ssl requests.
See: Annotations - NGINX Ingress Controller
The header is indeed Ssl-Client-Cert.
A successful request dumped by http-dump:
GET / HTTP/1.1
Host: <redacted>
Accept: */*
Ssl-Client-Cert: -----BEGIN%!C(MISSING)ERTIFICATE-----%!A(MISSING)MIIF3DCCA8SgAwIBAgIUGPLIyWCLDOY%!B(MISSING)iil6%!F(MISSING)Sf4%!B(MISSING)OMS7jIwDQYJKoZIhvcNAQEL%!A(MISSING)BQAwNzELMAkGA1UEBhMCUEsxEDAOBgNVBAoTB0NvZGVnaWMxFjAUBgNVBAMTDUNv%!A(MISSING)ZGVnaWMgQ0EgRzIwHhcNMjIwNzAzMjA0NTUyWhcNMjIwOTAxMjA0NTUyWjBvMSww%!A(MISSING)KgYJKoZIhvcNAQkBFh1yb2RyaWdvQGZhbWlsaWFuYXNjaW1lbnRvLm9yZzEaMBgG%!A(MISSING)A1UEAxMRd2VsdG9uIG5hc2NpbWVudG8xFjAUBgNVBAoTDW1lIG15c2VsZiBJbmMx%!A(MISSING)CzAJBgNVBAYTAkJSMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAubZk%!A(MISSING)hcdxQAGZSeaRl2WLFGQ6OxRei%!B(MISSING)sBk89nLmkmt8V5k1EnkQeObvZC%!F(MISSING)amHPkhllmBA%!A(MISSING)%!F(MISSING)jL2iyd25%!F(MISSING)wIoZ5NohNaijBH8WrJaL2pxVg9pVq98795BQ2SB%!F(MISSING)ekYqVDpEaes0U%!F(MISSING)%!A(MISSING)Wrk3h6K5UfcyHmuvkzv7S%!B(MISSING)vyqPj6kOvzwWnxVabA3GibIcPE%!F(MISSING)a%!B(MISSING)UYXzMpDhuNDUT%!A(MISSING)n3uuh5aVwLdTPjjC%!B(MISSING)0TA2RfpoMkcbZKUDdW%!B(MISSING)1%!F(MISSING)zV%!B(MISSING)ywA3Fftxx5E94OTDrV%!B(MISSING)%!B(MISSING)9nM%!A(MISSING)U%!F(MISSING)r2DVBihLrOzMme5BdLMaPK7UVWq6lgk21DP3sdJfdtUOGFtKLq9m3Ks02d87Tk%!A(MISSING)Va0Iq033QePnK%!F(MISSING)ngbQIDAQABo4IBpjCCAaIwFgYDVR0lAQH%!F(MISSING)BAwwCgYIKwYBBQUH%!A(MISSING)AwIwDgYDVR0PAQH%!F(MISSING)BAQDAgTQMB8GA1UdIwQYMBaAFJXrz%!F(MISSING)R5EzTy3VAeQXtrcU5Q%!A(MISSING)6BAIMIHFBgNVHSAEgb0wgbowgbcGCSsGAQQBg6hkATCBqTCBpgYIKwYBBQUHAgIw%!A(MISSING)gZkMgZZBcyBwZXIgdGhpcyBwb2xpY3kgaWRlbnRpdHkgb2YgdGhlIHN1YmplY3Qg%!A(MISSING)aXMgbm90IGlkZW50aWZpZWQuIFlvdSBtdXN0IG5vdCB1c2UgdGhlc2UgY2VydGlm%!A(MISSING)aWNhdGVzIGZvciB2YWx1YWJsZSB0cmFuc2FjdGlvbnMuIE5PIExJQUJJTElUWSBJ%!A(MISSING)UyBBQ0NFUFRFRC4wCQYDVR0TBAIwADA7BgNVHR8ENDAyMDCgLqAshipodHRwczov%!A(MISSING)L3BraS5jb2RlZ2ljLmNvbS9jcmxzL0NvZGVnaWNDQS5jcmwwKAYDVR0RBCEwH4Ed%!A(MISSING)cm9kcmlnb0BmYW1pbGlhbmFzY2ltZW50by5vcmcwHQYDVR0OBBYEFC4UrvbFX0SY%!A(MISSING)JKniQnXp6jX7x63AMA0GCSqGSIb3DQEBCwUAA4ICAQBEB7imPOMzdRjoRroaAcvu%!A(MISSING)UTVML8oVih9OUUByUodcVHNFrgG3lQnxk%!B(MISSING)lxAwO7HIS2aZGLqFnPv9ZoPkdNXL1M%!A(MISSING)Kk4WvihegcFQKiRCuZbpDZ7lcMaihdm3ZFSxVBYww5x%!F(MISSING)A4eA%!F(MISSING)zDpKjlpFhXAVAeA%!A(MISSING)e1oANKpZls1gKJOfNGMCk%!B(MISSING)COEiNIEGglxiRLdpAkDe%!F(MISSING)NGBq63xmPzNpJ8I4HuH6s%!A(MISSING)urqqslCiLZX8YA8IabXNnj%!F(MISSING)9exniuTEMAC6spnnm%!B(MISSING)BvvVWY2sNfStBNBYyLiv9Ac%!A(MISSING)luhFLNfB8qCDn7jD7kMEZJ9ae3%!F(MISSING)eZjlp25VNJbsI8QL7cHOlRaebBG%!F(MISSING)d4WaWIzJu%!A(MISSING)S0QvvDxfEp0KSSvH1kQH%!B(MISSING)SAap4VKMbB9%!F(MISSING)67cQOHjm8gKWuOgratxgwui9LTmmsIg%!A(MISSING)GK2XI%!F(MISSING)%!F(MISSING)105VBofAxCD4w8lv6zO%!F(MISSING)mTmusg6zp%!F(MISSING)h%!F(MISSING)Px%!B(MISSING)OXvCa1%!F(MISSING)LkroHMIkJQDEI9N%!A(MISSING)g08yD8MsCUCYxcoJU2YyODK1hAIrfM7AXdgA%!B(MISSING)Xvq0C%!B(MISSING)oYqBKhnEEojadlpkmYk4P%!A(MISSING)LGgYJvE%!B(MISSING)%!B(MISSING)ajrRMlAqVNvaX20ynPAnJUdZwSZsHwZLh8Gj3aOKg5KM3BGQj5oESvZ%!A(MISSING)ToTlDk1Y8Zs%!B(MISSING)SzdNfGtJvbP21dyTbFVhEiheftkv5rENG9jmeFxKdNgay37xRQ17%!A(MISSING)BSSxnK%!B(MISSING)JwO%!F(MISSING)ZAQnCFPCO6g%!D(MISSING)%!D(MISSING)%!A(MISSING)-----END%!C(MISSING)ERTIFICATE-----%!A(MISSING)
Ssl-Client-Issuer-Dn: CN=Codegic CA G2,O=Codegic,C=PK
Ssl-Client-Subject-Dn: C=BR,O=me myself Inc,CN=welton nascimento,emailAddress=rodrigo@familianascimento.org
Ssl-Client-Verify: SUCCESS
User-Agent: curl/7.64.1
X-Forwarded-For: <redacted>
X-Forwarded-Host: <redacted>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Real-Ip: <redacted>
X-Request-Id: 62917d77f650bd6092ac96d5b0000255
X-Scheme: https
Something funny seems to be happening with Ssl-Client-Cert. It is supposed to be a URLEncoded version of the cert, but there are those MISSING texts. I suppose this is a http-dump thing, we can ignore it.
Create a JKS keystore file with the appropriate certificates from your users’ certificates issuer and mount it via a configmap at /truststore/truststore.jks. In this case I used changeit as the keystore password.
On the keycloak side, those are the relevant configuration (valid for quarkus-based versions):
- command:
- /opt/keycloak/bin/kc.sh
- start
- --auto-build ## REQUIRED to enable nginx provider. Or you can use a pre-built custom image with nginx provider pre-activated.
env:
- name: KC_SPI_TRUSTSTORE_FILE_FILE
value: /truststore/truststore.jks
- name: KC_SPI_TRUSTSTORE_FILE_PASSWORD
value: changeit
## Activate nginx provider
- name: KC_SPI_X509CERT_LOOKUP_PROVIDER
value: nginx
## Set nginx provider header name
- name: KC_SPI_X509CERT_LOOKUP_NGINX_SSL_CLIENT_CERT
value: ssl-client-cert
## Set loglevel for the nginx provider
- name: QUARKUS_LOG_CATEGORY__ORG_KEYCLOAK_SERVICES_X509__LEVEL
value: TRACE
This is what a correct configuration will show in the logs. AbstractClientCertificateFromHttpHeadersLookupFactory is initialized 3 times because there are 3 x509-lookup providers. We only need to configure the nginx one.
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,825 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) certificateChainLength: '1'
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,826 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslClientCert: ''
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,826 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslCertChainPrefix was not configured
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,827 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) certificateChainLength: '1'
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,827 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslClientCert: 'ssl-client-cert'
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,827 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslCertChainPrefix was not configured
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,827 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) certificateChainLength: '1'
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,828 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslClientCert: ''
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:51:23,828 TRACE [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookupFactory] (main) sslCertChainPrefix was not configured
This is what a successful authentication looks like:
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,246 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) Loading Keycloak truststore ...
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,247 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) Keycloak truststore loaded for NGINX x509cert-lookup provider.
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,252 DEBUG [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookup] (executor-thread-6) Found a valid x.509 certificate in "ssl-client-cert" HTTP header
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,254 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) End user certificate found : Subject DN=[E=rodrigo@familianascimento.org,CN=welton nascimento,O=me myself Inc,C=BR] SerialNumber=[142430049820086541029243563005103597852424334898]
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,280 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) Certification path building OK, and contains 2 X509 Certificates
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,280 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) Rebuilded user cert chain DN : E=rodrigo@familianascimento.org,CN=welton nascimento,O=me myself Inc,C=BR
idp-keycloak-5f66645b7d-8k2tz idp-keycloak 2022-07-04 00:53:08,280 DEBUG [org.keycloak.services.x509.NginxProxySslClientCertificateLookup] (executor-thread-6) Rebuilded user cert chain DN : CN=Codegic CA G2, O=Codegic, C=PK
1 Like
I suppose you should change KEYCLOAK_X509CERT_LOOKUP_SSL_CLIENT_CERT to KEYCLOAK_X509CERT_LOOKUP_NGINX_SSL_CLIENT_CERT and it will work.
hi @weltonrodrigo and @hokiegeek2 ,
i followed this thread and tried to follow, although i have config keycloak on centos8 also with nginx without image container.
Here is keycloak config:
Here is nginx config:
And get exception:
I’m new to mutual certificate validation and don’t know what exactly is going wrong.
Can you please tell me where is my problem? Or do I need more knowledge about this? Please let me know.
Thanks very much
KC.proxy should be edge.
Instead of SSL_CLIENT_CERT as the header name, it should be ssl-client-cert.
Not sure about the nginx configuration part. It should have config settings the certificate in the header and CA certificate.
There are some nginx tutorials about that. Make sure that part is ok before trying to put keycloak in the equation.
1 Like
@weltonrodrigo thanks a lot for your return,
yes , my nginx already notify me to choose cert:
thank you again <3
hi @weltonrodrigo, I have a problem with that.
After I choose certificate ,browser redirect to my Keycloak. But I want to use certificate for user login not adminKeycloak. So, do you have any solution for this problem?
Best regards to you
@sonlam17 sorry, been away from this for a few weeks. I can get x509 certificate authentication to work just fine unless Keycloak is behind a Kubernetes ingress. I am still prompted for a certificate, and it appears that the certificate makes it to Keycloak, but then it appears that Keycloak fails to parse the corresponding header. I’ve tried several configurations to no avail. I will re-engage on this in the next few days to hopefully figure it out.
hi @hokiegeek2, I think your problem might be because the trust store you gave keycloak doesn’t have the certificate of your choice. Or you miss the config truststore in Keycloak: Configuring a Truststore - Keycloak