Docker Swarm Example for Clustered Docker

I have spent quite a while trying to configure Keycloak Cluster managed through Docker Swarm. Keycloak offers several peer discovery protocols, including JDBC_PING and TCPPING. At first, I opted for JDBC_PING. This one uses database itself to store data about peers in a cluster, so it looked very promising in the beginning.
And in fact, you can easily find several examples of setting up a Docker Swarm based Keycloak cluster using JDBC_PING online, for example, here.
What I found out later is that in case of swarm node’s failure with subsequent restart, Keycloak instances seem to have a difficult time re-grouping and re-discovering their peers. This happens in the event of cluster leader’s failure because the followers do not seem to be willing to take over that role. (This was the case with v10.0.2).
Given that failure-resistance was the main reason for even attempting to have cluster of the application we are working on, this was a deal-breaker for us. So, another solution had to be found.
The other option was TCPPING protocol. This one would require passing peer IPs on container startup. In this case, we had two ways (mind you, we are working with docker swarm here):

  1. Two (or more) separate services are defined, i. e. keycloak1, keycloak2, etc. Their only difference is their names, and we specify an env. variable corresponding to initial hosts as “keycloak1[7600],keycloak2[7600]”. The problems with this approach are obvious:
  • It is a total docker swarm/compose anti-pattern which would prevent one from easily scaling the service
  • A problem of several endpoints for another part of application which has to be solved by setting up a proxy or through programmatic means.
  1. The second solution is to teach Keycloak containers to discover their peers dynamically on startup. This is the way that I chose. For this to work, I needed to do some tweaking, but it surely did pay off. I described the way I implemented it in this repo.
    By choosing this approach, we make use of docker built-in load-balancing between different tasks of the same service. We also have a single port to access the service from the outside if needed. Moreover, scaling docker becomes as easy as using docker service scale