Keycloak 18 - Distributed cache on Azure cloud

Configuring the server
1

Hi everyone,
I was wondering if someone tried implementing distributed cache with infinispan Cloud discovery protocols.
I am really having trouble understanding what to configure since a lot of topics and websites talks about Wildfly and standalone.xml files ( Since Quarkus this kind of file no longer exists )
If someone experienced implementing clustering with ditributed cache on cloud i would really appreciate a feed back.
Thank you !
Sites visited :
Infinispan 11.2.7. Cloud discovery protocols
Jgroups-Azure

2

Since version 17, keycloak migrated from wildfly to quarkus, so any documentation based on wildfly (standalone.xml) will not help you.

The good news is that quarkus comes with distributed cache pretty much solved. Take a look at Configuring distributed caches - Keycloak

Just set the appropriate --cache-stack command line argument, or KC_CACHE_STACK environment variable (All configuration - Keycloak).

Just notice that for virtual machines running in the same subnet, cluster setup is automatic, you only need another discovery protocol with different subnets or involved container setups (as in kubernetes, fargate, AKS, ECS etc).

1 Like
3

Hi! Apologies for replying to an old thread but it is relevant…

We’re hosting Keycloak 18 as a custom Docker build in Azure Web App… with 3 nodes.

I am only partially able to follow the instructions provided…

:white_check_mark: Add KC_CACHE_STACK=azure to the build stage in the dockerfile

From the Keycloak Caching Documentation (linked above)… the instruction is…

Cloud vendor specific stacks have additional dependencies for Keycloak. For more information and links to repositories with these dependencies, see the Infinispan documentation.

To provide the dependencies to Keycloak, put the respective JAR in the providers directory and build…

Reading the Infinispan documentation:

Providing dependencies for cloud discovery protocols

To use aws.S3_PING, GOOGLE_PING2, or azure.AZURE_PING cloud discovery protocols, you need to provide dependent libraries to Infinispan.

Procedure

  • Add the artifact dependencies to your project pom.xml.

You can then configure the cloud discovery protocol as part of a JGroups stack file or with system properties.

So where does the “respective JAR” for Keycloak come from?
Where does the “dependent libraries” for Infinispan come from?

Do I need to download and build an Azure flavour of Infinispan source?
Do I need to download and build an Azure JGroups source?

Both infer I add them as dependencies to my POM.xml… but I do not have a POM.xml as I am using the Docker 18.0.2 reference image to build from with minimal provider injection (for Email OTP) and theming.

Also, how do I inject the JGroup required configuration at runtime? JGroups Azure Configuration. As mentioned, that documentation is based on Wildfly. Ideally it would be via Azure Web App Configuration (e.g. environment variables)…

Sorry if the questions are basic… I’ve tried looking for videos, but everything seems to be around “dev” and PoC environments… we didn’t hit the caching problem until our multi-node prod environment…

The good news is that quarkus comes with distributed cache pretty much solved.

Is it?

4

Hi. You are right. The documentation is somewhat sparse in that regard and I also didn’t understood it properly.

Besides adding KC_STACK=azure, you’ll need the libraries that implement azure ping protocol for jgroups.

For azure, that is org.jgroups.azure:jgroups-azure and com.microsoft.azure:azure-storage.

Just put those two jars in the /opt/keycloak/providers/ folder of the keycloak container. You can either mount the folder or copy them in a dockerfile and create your own keycloak container image (preferred method).

In your dockerfile, copy the two jars and RUN KC.sh build.

You’ll also need to configure the jgroups azure using properties as in GitHub - jgroups-extras/jgroups-azure: Implementation of Azure ping protocol using storage blobs using official Azure SDK.

Yeah, the keycloak docs right now about that are subpar.

UPDATE:

Link to jars:

Try those two and see if it works. It’s possible that you’ll need transitive dependencies of those two.

2 Likes
5

Hi weltonrodrigo. I was trying your proposal with 2 seperate keycloak instances deployed as Azure web apps. Set the stack, mounted the jar files. I am seeing ISPN files created in the storage account(below screenshot), but clusterization still doesn’t seem to work for me. In the web apps logs I see after 10 join attempts that each keycloak is starting as a singleton.

image
image1542Ă—80 5.67 KB

Do you have some experience with clusterizing keycloak deployed on azure as web apps?

6

Hi. Pretty late here, but answering for Google purposes.

The azure_ping protocol is a way for instances to find each other, but they still need to comunicate via normal network requests, directly.

So, in an App Services setup is quite improbably that the keycloak instances will be able to reach each other without the use o private endpoints.

Search the web about how to make an app services deployment reach another and maybe this will work.

7

Hi again
I am already using private endpoints, no success, keycloak instances are not seeing each other over azure ping protocol.
BTW, private endpoint communication is established from keycloak azure web app with other services from the Azure vnet(postgresql flexible server communication over Private Endpoint, Frontend and Backend apps too) and that one works, only keycloak clusterization doesn’t seem to work with this setup…

8

Keycloak instances deployed as Azure App Service scaled out instances are not seen its other, not due to Azure Ping protocol but due to TCP protocol.
TCP should be configured appropriately in the stack. But is a real challenge to do that as the private IPs of the instances are not known in advance.

I have tried the obvious, but with with no luck.

<TCP bind_addr="LINK_LOCAL" />

LINK_LOCAL
Picks a link-local IP address, from 169.254.1.0 through 169.254.254.255.

Other options is to set somehow the external_addr (the idea is from this great repo GitHub - ivangfr/keycloak-clustered: Keycloak-Clustered extends quay.io/keycloak/keycloak official Keycloak Docker image by adding JDBC_PING discovery protocol.)
The official way to do that is by setting JGROUPS_DISCOVERY_EXTERNAL_IP with its docker name,
but again this is not possible as the docker name is selected by Azure when starting the App Service.

docker run --rm --name keycloak-clustered-1 -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD=admin \
  -e KC_DB=mysql \
  -e KC_DB_URL_HOST=mysql \
  -e KC_DB_URL_DATABASE=keycloak \
  -e KC_DB_USERNAME=keycloak \
  -e KC_DB_PASSWORD=password \
  -e KC_LOG_LEVEL=INFO,org.infinispan:DEBUG,org.jgroups:DEBUG \
  -e JGROUPS_DISCOVERY_EXTERNAL_IP=keycloak-clustered-1 \
  --network keycloak-net \
  ivanfranchin/keycloak-clustered:latest start-dev

Other idea is to start docker somehow with --network=host, but I don’t know if this is feasible in Azure App Service.

I think it would be a great option to scale out Keycloak running as App Service.
But I am out of ideas at the moment of how to accomplish that, to help me and the community in this direction.

9

New lead, after reading how Red Hat clustered JBoss EAP 7 in Azure App Services which is using JGroups like what Keycloak does.

It used App Service VNET Integration.But how it gets the host IP ( i.e 10.0.x.x, instead of the App Service 169.254.xx ?)
The answer was hidden in the overview-vnet-integration documentation
When virtual network integration is enabled, your app makes outbound calls through your virtual network. The outbound addresses that are listed in the app properties portal are the addresses still used by your app. However, if your outbound call is to a virtual machine or private endpoint in the integration virtual network or peered virtual network, the outbound address is an address from the integration subnet. The private IP assigned to an instance is exposed via the environment variable, WEBSITE_PRIVATE_IP.

So I tried the following, focusing on the TCP protocol of my stack:

<jgroups>
          <stack name="azure-ping-tcp" extends="tcp">
		    <TCP external_addr="${env.WEBSITE_PRIVATE_IP}" bind_addr="${env.KC_BIND_ADDR:SITE_LOCAL}" bind_port="${env.KC_BIND_PORT:7800}/>
            <azure.AZURE_PING
			    storage_account_name="${AZURE_STORAGE_ACCOUNT_NAME}"
				storage_access_key="${AZURE_STORAGE_ACCESS_KEY}"
				container="${AZURE_CONTAINER}"/>
		   <MERGE3 min_interval="20000" max_interval="100000"/>
           <FD_SOCK/>
           <FD_ALL timeout="10000" max_tries="5"/>
           <VERIFY_SUSPECT timeout="2000"/>
           <pbcast.NAKACK2 use_mcast_xmit="false" use_mcast_xmit_req="false"/>
           <UNICAST3/>
           <pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M"/>
           <pbcast.GMS max_join_attempts="10" print_local_addr="true" join_timeout="3000" view_bundling="true" />
           <UFC max_credits="2M" min_threshold="0.1"/>
		   <MFC max_credits="2M" min_threshold="0.4" />
           <FRAG3 frag_size="60K"/>
		   <pbcast.STATE_TRANSFER />
        </stack>
    </jgroups>

It seems it is bound to VNET IPs, but still the scaled out instances cannot communicate over the VNET.

I have started thinking of that there some special VNET configuration and i tried the JBoss EAP 7 ARM template which it works as expected.
(GitHub - SLionB/clustered-jboss-demo: Demo app for clustered Red Hat JBoss EAP on App Service)

Where is the hidden sauce in the receipt to allow intercommunication between Azure App Service instances running Keycloak/JGroups in the same Azure Service Plan?