Keycloak behind pfSense/haproxy not loading administration console

[gitwittidbit]

Hi,

I am running a few services in my home lab behind a pfsense firewall with haproxy enabled. That all works well. Now I am trying to add Keycloak to the mix.

When I access Keycloak at its local IP address, everything works fine. But when I access it via its FQDN (login.example.com), it displays the welcome page but does not load the administration console (I just get a blank screen).

Do I have to tell Keycloak somewhere that it should react to calls of its FQDN (for example, Nextcloud does not like to be called upon at FQDNs it doesn’t know about)? Or what might be the problem here?

Thanks for any hints!

[dasniko]

You have to tell Keycloak that it runs behind a proxy and your proxy must forward certain headers.
Depending on the Keycloak distro (Wildfly or Quarkus), this is done differently. See the proper docs for details.

[gitwittidbit]

Thanks for the quick reply!

So I found this in the docs:

"There is some extra configuration you have to do in this scenario so that the actual client IP address is forwarded to and processed by the Keycloak server instances. Specifically:

• Configure your reverse proxy or loadbalancer to properly set X-Forwarded-For and X-Forwarded-Proto HTTP headers.

• Configure your reverse proxy or loadbalancer to preserve the original ‘Host’ HTTP header.

• Configure the authentication server to read the client’s IP address from X-Forwarded-For header."

But I am not 100% how to implement this with my haproxy (as part of pfsense).

I have the option “X-Forwarded-For” on anyways in the (shared) frontend config.

I have added a custom header “X-Forwarded-Proto” in the Backend and tried setting it to “http” and “https”.

But I am not sure to "preserve the original ‘Host’ HTTP header. I interpret this as not to replace it by setting a new header but rather adding the X-Forwarded-Proto header.

The docs continue as follows:

" If your proxy is forwarding requests via the HTTP protocol, then you need to configure Keycloak to pull the client’s IP address from the X-Forwarded-For header rather than from the network packet. To do this, open up the profile configuration file ( standalone.xml , standalone-ha.xml , or domain.xml depending on your operating mode) and look for the urn:jboss:domain:undertow:12.0 XML block.

X-Forwarded-For HTTP Config

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
   <buffer-cache name="default"/>
   <server name="default-server">
      <ajp-listener name="ajp" socket-binding="ajp"/>
      <http-listener name="default" socket-binding="http" redirect-socket="https"
          proxy-address-forwarding="true"/>
      ...
   </server>
   ...
</subsystem>

Add the proxy-address-forwarding attribute to the http-listener element. Set the value to true ."

Here, I struggled a bit to find the standalone.xml, as I am running the docker version. But I think I found it although it looks a bit different (there are a few more elements in my version (e.g. a https-listener) and there already was a proxy-address-forwarding element there which I changed (rather than added) according to the docs.

I then restarted the host but, alas, it isn’t working.

Is there anyone with a setup like mine who could share their config? Or is anyone able to pinpoint where I made a mistake?

Thanks

[dasniko]

If you use the Docker image, you can just set the env var PROXY_ADDRESS_FORWARDING=true, this sets all you need in the standalone.xml, no need to modify it manually (which is always a bad idea).

Pro tip: The Wildfly version is deprecated, if you just start using Keycloak, you should start with the Quarkus-based version. Here’s the guide for using it behind a proxy: Keycloak - Server - Using a reverse proxy

[gitwittidbit]

Thank you for the pointer! Glad, I am not spending more time with a dead man walking…

So I have started setting up a new config for the Quarkus based version now.

I (believe) I have managed to set up a docker-compose file to start keycloak in production mode by providing the following commands at run time:

• hostname: I put keycloak’s FQDN (e.g. keycloak.example.com)
• proxy: my proxy is terminating TLS, so I set this to ‘edge’

And I also set the environment variable PROXY_ADDRESS_FORWARDING=true

(And I configured a postgres database, but I put aside setting up a cluster for the moment.)

I can now reach the welcome page at “login.example.com” and from there the management console (although I understand that this is not good practice - so I might need to play around a little more to block the admin console or give it a different FQDN).

So thank you very much for your help!

(The next issue is already waiting to be addressed. But for that I will open a new thread.)

[gitwittidbit]

Sorry, one more follow up question:

The guide for setting up Keycloak behind a proxy is what I used unsuccesfully before with my old installation.

And as I am struggling with configuring one of my services to actually use Keycloak behind the proxy, I am wondering whether this has to do with the settings I need to make in the proxy config.

Would you be able to help me with this?

What does “Configure your reverse proxy or loadbalancer to preserve the original ‘Host’ HTTP header.” mean? I interpreted this as do not replace the original header with a new one but add to it instead. But somewhere else I read that I should create a new header as otherwise bad actors might spoof the origin of the request. So if I am to create a new header, how do I preserve the original ‘Host’ HTTP header?

(Maybe the problem is my very limited knowledge of HTTP headers in general: Is there only one HTTP header that may contain different pieces of information and so creating a new one replaces the original one or are there more headers that can exist next to each other and so creating a new does not affect the others?)

Thanks again for your help!

[gitwittidbit]

Okay, did some reading.

So it is my understanding now that these HTTP headers stand next to each other and I can delete, set or change each one independently.

I also found out that haproxy lets the X-Forwarded-Host header pass untouched. So in order to preserve it, as prescribed in the keycloak docs, I shouldn’t have to do anything.

This leaves me with X-Forwarded-For and X-Forwarded-Proto. I set both of them (in the haproxy backend settings) but it doesn’t help. While Nextcloud redirects to Keycloak for the login, after the login the redirection back to Nextcloud gets stuck somewhere. (For good measure, I also tried setting X-Forwarded-Host, but it doesn’t make a difference.)

Is there anyone who has a setup similar to mine? Nextcloud and Keycloak behind haproxy (on pfSense).

Any ideas what else I could try would be greatly appreciated.

[peterbecich]

Could this be related to ssl - Using Keycloak behind a reverse proxy: Could not open Admin loginpage because mixed Content - Stack Overflow

Specifically ssl - Using Keycloak behind a reverse proxy: Could not open Admin loginpage because mixed Content - Stack Overflow

[regilero]

There’s a bunch of issues on github related to finding the right configuration of various settings for keycloak 18 and 19 to get everything right. Some of the settings are not so well documented.

Search for ‘proxy’ or ‘edge’ keywords.

On this one for example I added a working configuration for version 18.02, in case it can help you

keycloak(behind nginx) .well-known/openid-configuration path not return correct token or jwt url（custom port loss）

On the proxy side you should ensure that the X-Forwarded headers are set, but it’s usually the case.

[peterbecich]

Thanks. Because I am attempting to use the Keycloak operator (keycloak/operator at main · keycloak/keycloak · GitHub), I believe the environment variables you have specified here keycloak(behind nginx) .well-known/openid-configuration path not return correct token or jwt url（custom port loss） · Issue #13148 · keycloak/keycloak · GitHub will need to be put in a custom Keycloak image Using custom Keycloak images - Keycloak. I will try this.

---

When using the operator, here are some of the default KC_* environment variables:

kubectl describe pod example-kc-0 --namespace=keycloak
...
Controlled By:    StatefulSet/example-kc
Containers:
  keycloak:
    Container ID:  
    Image:         docker.io/peterbecich/keycloak:latest
    Image ID:      
    Ports:         8443/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
...
    Environment:
      KC_CACHE:                       ispn
      KC_HEALTH_ENABLED:              true
      KC_CACHE_STACK:                 kubernetes
      KC_DB:                          postgres
      KC_DB_URL_HOST:                 postgres-db
      KC_DB_USERNAME:                 <set to the key 'username' in secret 'keycloak-db-secret'>        Optional: false
      KC_DB_PASSWORD:                 <set to the key 'password' in secret 'keycloak-db-secret'>        Optional: false
      KEYCLOAK_ADMIN:                 <set to the key 'username' in secret 'example-kc-initial-admin'>  Optional: false
      KEYCLOAK_ADMIN_PASSWORD:        <set to the key 'password' in secret 'example-kc-initial-admin'>  Optional: false
      jgroups.dns.query:              example-kc-discovery.keycloak
      KC_HOSTNAME:                    test.keycloak.org
      KC_HTTPS_CERTIFICATE_FILE:      /mnt/certificates/tls.crt
      KC_HTTPS_CERTIFICATE_KEY_FILE:  /mnt/certificates/tls.key
      KC_PROXY:                       passthrough
...

I know now these are not set in the Dockerfile. I have set KC_* in a custom Keycloak image, and these environment variables are ignored by the operator. So Using custom Keycloak images - Keycloak does not solve this.

The KC_* environment variables you have specified here keycloak(behind nginx) .well-known/openid-configuration path not return correct token or jwt url（custom port loss） · Issue #13148 · keycloak/keycloak · GitHub will need to be set somewhere else to take effect in the operator. Possibly here: keycloak/KeycloakDeployment.java at ca0dea913258115f8b87c87e5078709941af8194 · keycloak/keycloak · GitHub

[peterbecich]

Turns out only one small change was necessary for Keycloak with the Quarkus operator: Blank page after clicking Admin Console - #7 by dcrouchelli

Including the port number in the hostname in example-kc.yaml:

hostname: localhost:8443

---

I still have not learned how to set these KC_* environment variables in the Keycloak container using the Quarkus operator. I have asked StackOverflow: Quarkus Kubernetes operator: how to set environment variables in application container (not operator container)? - Stack Overflow

---

Thank you!