Various issues with Keycloak Operator

Configuring the server
1

In trying to get Keycloak installed on k8s via the operator, I encountered a number of issue that need attention:

  • There is no way to configure the storageClassName for PersistentVolumeClaims, so clusters need to have a default StorageClass, and be fine with deploying the DB backing store on that class.
  • There is no way to configure resource requests/limits, and no defaults are provided.
  • There is no way to configure the frontendUrl, affecting both the server config, and the Ingress (if enabled).
  • Per above, the default Ingress created by the operator has a host property of *, which is unlikely to be desired or useful. It is also unsecured (HTTP 80 only). Suggest cert-manager compatible handling of Ingress certificates (both auto-generated via issuer annotations, and tls/secretRef for pre-existing certs).
  • The Service created for the frontend has a service.alpha.openshift.io/serving-cert-secret-name annotation, which I’m guessing on openshift does some sort of TLS tunnelling, but on vanilla k8s does nothing. The symptom that leads me to presume this, is that attempting to connect to port 8443 via the service results in a failure to negotiate TLS, and a dropped connection, resulting in no access to the server. Perhaps this is actually due to a lack of TLS certs in the container or something though, rather than something openshift-specific, either way, the service is inaccessible.
  • Having worked around this by creating a custom service pointing to port 8080 in the container, and disabling the operator-provisioned Ingress, I’m able to create a functional external Ingress, and gain access to the welcome screen at keycloak.example.com/auth/, however clicking on the link to the admin console (keycloak.example.com/auth/admin/), I’m redirected to the OIDC login page, and get an Invalid parameter: redirect_uri error, as described here. Deleting the operator and adding the env var suggested there did indeed allow successful login.

So, with a bunch of hacking:

  • create custom service pointing to insecure server container port 8080
  • create custom ingress
  • delete operator
  • modify StatefulSet to add PROXY_ADDRESS_FORWARDING="true" env var

It is possible to get a Keycloak instance running on vanilla k8s via the operator, but it’s far from optimal, and at this stage I’ve not thoroughly tested everything to make sure that it is fully functional (if anything internal is using the 8443 port it will not work), since this is clearly not a hack I’m willing to put into production. And since the operator must be deleted/disabled, realms/etc clearly can’t be managed via CRDs in this state anyway.

2

I am “glad” to see I am not the only one facing those issue. It seems we went more or less through the same path. I’ll open a proper git hub issue for the operator as soon as I can.
Unfortunately, I don’t feel good enough in go to make a contribution.

1 Like
3

I believe they do all their tracking via the RedHat JIRA instance they have set up for the parent Keycloak project, which is why I didn’t just lodge issues immediately. Most of this should be relatively easy to solve, but there are also no contribution guidelines for the project, so not sure what the process is, if there are CLAs to sign or whatever.

I did manage to work out how to filter out the operator-specific issues in JIRA and some of these have been reported already:

https://issues.redhat.com/browse/KEYCLOAK-12397
https://issues.redhat.com/browse/KEYCLOAK-12396
KEYCLOAK-12307
KEYCLOAK-12058
(Discourse won’t let me put more than two links in a post, you’ll need to add the URL prefix to those last two).

But not all issues are covered, and some of these refer to openshift-specific use-cases that will not fix vanilla k8s. The major problem of non-functional port 8443 looks like it may be fixed by a change in the keycloak-containers project, but it doesn’t look to me like the change has been released yet (looks like container fixes only happen when there’s a release of the upstream Keycloak project??).

4

Thank you for giving the Keycloak Operator a go! Please bear in mind, that 7.0.1 was an initial release and we definitely need some time to improve it.

Let me try to answer all the questions:

Yes, that’s true - there no way to override this at the moment. Parametrizing this one option and creating a new StorageClass (and expect user to modify it to his needs) would be a second option to consider. I created KEYCLOAK-12680 to track this.

Yes, that’s correct. I created KEYCLOAK-12681 to track this.

We already have a KEYCLOAK-12397 to track this.

This one is slightly more complicated. For OpenShift Routes (similar concept as Ingress) we use TLS Passthrough configuration with OpenShift-generated key and certificate. This is the most secured configuration as we can not assume you trust your router (you usually do, but as a security-related project we can not assume it). We will probably introduce an option use other types of TLS termination (like Reencrypt), see KEYCLOAK-12307. As for Kubernetes specific settings, we would like to rely more on community contributions. For very advanced usecases, we suggest to disable default external access and configure Ingress manually. Does this answer your question?

Yes, this is OpenShift specific annotation that generates a new TLS key and injects it into Keycloak Pods. For vanilla K8s it does nothing. As for the TLS issue - I believe you hit a very nasty error I fixed a while ago, see this Pull Request. In short - Keycloak HTTPs endpoint didn’t work correctly with vanilla K8s. Also, your workaround seems reasonable.

Yes, it’s already on our radar, see KEYCLOAK-12679.

Thanks a lot for evaluating our Operator! Unfortunately there are many projects under Keycloak umbrella and managing Github issues for all of them is really hard. For the time being we disabled them and moved all issue management to our Keycloak JIRA. We are looking at some options to sync those two ticket systems together (and enable Github issues back) but is something that won’t happen anytime soon. Thanks for understanding!

Yes, the vanilla K8s HTTPS issue should be fixed in the next Keycloak release. If you’d like to try it out (without waiting for the next Keycloak release), you may clone our Operator repo and use the master tag here.

Again, I would like to thank you both, @pdf and @fabricepipart for evaluating our Operator. Your comments are all very valuable for us and we’ll do our best to improve the general experience of the Keycloak Operator.

5

I just found this discussion and am curious if the keycloak operator version 21.0.1 and later can automatically annotate the keycloak service with service.alpha.openshift.io/serving-cert-secret-name. We want to use an automatically generated certificate for our instance in OpenShift. Is this possible with keycloak operator version 21.0.1? I couldn’t find any documentation about it.

6

Is it possible to get client real address?