How can isolate Infinispan between namespace in one Kubernetes cluster?

Currently, I have one Kubernetes with 2 namespaces: NS1 and NS2. I’m using jboss/keycloak Docker image.

I am operating 2 Keycloak instances in those 2 namespaces and I expect that will run independently. But it is not true for Infinispan caching inside Keycloak. I got a problem that all sessions of KC in NS1 will be invalidated many times when the KC pod in NS2 is being stated “Crash Loopback”.

The logs said as following whenever the “Crash Loopback” KC pod in NS2 tries to restart:

15:14:46,784 INFO [org.infinispan.CLUSTER] (remote-thread--p10-t412) [Context=clientSessions] ISPN100002: Starting rebalance with members [keycloak-abcdef, keycloak-qwerty], phase READ_OLD_WRITE_ALL, topology id 498

keycloak-abcdef is the KC pod in NS1 and keycloak-qwerty is the KC pod in NS2. So, the KC pod in NS1 can see and be affected by KC pod from NS2.

After researching, I see that Keycloak uses Infinispan cache to manage session data and Infinispan uses JGroups to discover nodes with the default method PING. I am assuming that this mechanism is the root cause of the problem “invalidated session” because it will try to contact other KC pods in the same cluster (even different namespaces) to do something like synchronization.

Is there any way that can isolate the working of Infinispan in Keycloak between namespaces?

Thank you!

The problem you are seeing comes from the discovery mechanism used by jgroups. When two keycloak pods end up in the same node, they will find each other, as, normally all pods in a node will have the same broadcast address.

What you can do is change the discovery mechanism to KUBE_PING or DNS_PING. If all your instances use those methods, they will find only instances in the same namespace. DNS_PING is the simpler of the two, but it needs a service of type headless.

KUBE_PING will need a service account, role and roleBinding. If you are not the cluster admin, you’ll probably won’t have permission to create those.

Example of DNS_PING configuration:

env:
  - name: JGROUPS_DISCOVERY_PROTOCOL
    value: dns.DNS_PING
  - name: JGROUPS_DISCOVERY_PROPERTIES
    value: dns_query=keycloak-headless
  - name: CACHE_OWNERS_COUNT
    value: "2"
  - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
    value: "2"

Where keycloak-headless is the service with Type headless.

1 Like

Hi @weltonrodrigo,
Thank you for your suggestion.
It sounds great to me but in the meantime, I’m also got a suggestion that I can use JDBC_PING in my situation to limit the discovery mechanism only doing its job for instances/nodes which are using the same database. What do you think about this solution?

I would say it’s great recommendation wildfly - How can isolate Keycloak Infinispan between namespaces in one Kubernetes cluster to prevent KC pod from discovering and synchronizing from one other - Stack Overflow

You don’t have any additional dependency with jdbc ping. Kube ping requires Kuberneres API dependency, dns ping requires DNS in the cluster. Anyway, you are the admin, so you can evaluate and accept risk of each additional dependency and increased complexity, where things can go wrong.

1 Like

I agree with this assessment. JDBC_PING only needs a preexistent table on the database.

This can become a problem to you if you are orchestrating several keycloak, as the migration scripts for keycloak will only create the tables (or relations) needed by keycloak, not by JDBC_PING.

So, your options:

  • create a headless service for DNS_PING
  • create role and rolebinding for KUBE_PING
  • create table (or relation) for JBDC_PING

choose your fighter :robot:

1 Like

This is wrong. JDBC_PING creates itself its own table, which is necessary to manage the node registries.

Do you have any doc on this? In my experience, activating JDBC_PING resulted in relation "jgroupsping" does not exist in Postgres.

I‘m using this in all my trainings/workshops, and also at various customers in production. It‘s being created on startup. There‘s also the attribute „initialize_sql“ (or similar) to adjust the create-table statement, …

Just the first hit on a google search:

A default table will be created at first connection, …

http://www.jgroups.org/javadoc/org/jgroups/protocols/JDBC_PING.html

2 Likes

Hi @dasniko,

I’m running jboss/keycloak:15.0.2 but it will not create the jgroupsping relation on startup and the error JGRP000145: Error updating JDBC_PING table: org.postgresql.util.PSQLException: ERROR: relation "jgroupsping" does not exist will be printed out.

But if I change the environment value from

ENV JGROUPS_DISCOVERY_PROTOCOL JDBC_PING

to

ENV JGROUPS_DISCOVERY_PROTOCOL JDBC_PING
ENV JGROUPS_DISCOVERY_PROPERTIES datasource_jndi_name=java:jboss/datasources/KeycloakDS,info_writer_sleep_time=500,initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING ( own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, created timestamp default current_timestamp, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))"

the jgroupsping relation will be created as expected.

Do you think is there any reason that prevents jgroupsping from automatically creating? Or initialize_sql is mandatory in the configuration? Thank you for your support!

Also thanks a lot to @weltonrodrigo and @jangaraj for your sharing solution.

As you might see in your properties env var, the SQL is DB specific. The default SQL statements in JDBC_PING are MySQL dialect. If you use something different, it might fail. Most likely because of the byte-array data type in the initialize sql, which is different from DB to DB. You are using the “bytea” datatype, so I assume you are using Postgresql, and this doesn’t understand the MySQL datatype. There should be an error message in the logs of your database…

1 Like

Oh yes, I’m using Postgresql for Keycloak DB. It’s clear to me now.
Thank you for your explanation!