Keycloak - Infinispan Redis cache store

Question

Currently setting up a keycloak cluster in standalone-ha mode, to be able to run on docker swarm. In keycloak, the user sessions are cached in an embedded infinispan store and infinispan can be configured to be a distributed cache across the cluster.

I have also set the owner to be 2, but the problem is that.. during scale-down, there is a possibility for the user-sessions to be lost, If both the owners containing the cache are killed during scale-down.

I have also read about Infinispan Redis cache store, but I am not sure how to configure this.

Question 1: Is it possible to configure Keycloak Infinispan to use a Redis Store ?

Question 2: If this is not possible, is there a way that one could overcome this problem ?

Any suggestions would be helpful.

Downbeat

Can you configure your scale down policy to only terminate one server at a time? Infinispan should detect the missing server and rebalance the sessions that were cached on that server over remaining nodes. – Omophagia

As you use Swarm, this may be of interest: wildfly-swarm.gitbooks.io/wildfly-swarm-users-guide/content/… – Verbify

Answer 1

Due to this PR https://github.com/keycloak/keycloak/commit/056ba75a72b1595ca9fa471f5693201fd5b2c7ae by default (Keycloak latest release 6.0.1) the Infinispan Connection SPI which uses InfinispanChangelogBasedTransaction.java has a very particular use of a CacheDecorator.java which will skipCacheStore. This means that no matter if you configure a store with persistence the store will be ignored.

In order to achieve what you want, besides configuring the store, you would have to customize most of the SPIs here https://github.com/keycloak/keycloak/tree/master/model/infinispan/src/main/resources/META-INF/services in order to make sure that Keycloak will be using the cache store.

This will also not be easy since there are a lot of perks involved into the process, for example, since Keycloak is using the Marshaller of Jboss, if you customize this SPIs you would have to bring most of the org.keycloak.models.sessions.infinispan package and register your module to make sure that Wildfly will be able to see the entities to marshall.

Another thing is that you should, with Redis, configure most of the caches pointing to one common database, except the authenticationSessions which cannot be in the same database as sessions, otherwise, there will be conflicts like the RootAuthenticationSesssionEntity being found but expected to be SessionEntityWrapper.

To resume, the process will be painfull, but if you want to dare and do it, this is how I achieved it:

• Introduced a custom InfinispanConnectionProviderFactory in order to have full capability to use infinispan configuration and then configure my containers like:

private Configuration getRedisConfiguration(int database) {
    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.persistence()
      .passivation(false)
      .addStore(RedisCacheStoreConfigurationBuilder.class)
      .ignoreModifications(false)
      .fetchPersistentState(false)
      .purgeOnStartup(false)
      .preload(false)
      .shared(true)
      .addProperty("host", System.getenv("REDIS_HOST"))
      .addProperty("port", System.getenv("REDIS_PORT"))
      .addProperty("database", String.valueOf(database));

    return cb.build();
  }

The RedisCacheStoreConfigurationBuilder that you see there is basically a stripped-down version of the original store but I don't need Sentinel or Server mode I just want to connect to a host, port, and database.

Then I basically copied the org.keycloak.models.sessions.infinispan removing everything related to remove cache, and instead of using normally the cache without the decorator to skipCacheStore.

Let me know if I can help with something, I will most prepare a post that instructs more detailed how to do this, involving also a repository that will contain the codes that I am talking about. Please let me know more if someone is still trying this.

Snooker

Do you have post, with detailed steps. I am curious, how to use it. It's because I would like to know, when the user session is expired. – Shilohshim

there is a background job that runs to remove the expired the sessions.. one of the interfaces of the models you have the removeExpiredSessions method to implement, which will be called by this background thread, you can configure at which freequency it runs.. – Snooker

What if we provide session authentication classes to store in redis and do not touch cache classes? – Jungly

Answer 2

When expecting to scale large systems dynamically, it is expected to avoid such a constraint to register a list of available nodes in configurations. So Redis nodes discovery is a benefit here.

Infinispan itself supports Redis Store as an extension implementing both SPI and adding XML entities to ease configuration:

• http://infinispan.org/docs/cachestores/redis/

• https://github.com/infinispan/infinispan-cachestore-redis

• http://infinispan.org/docs/stable/user_guide/user_guide.html#custom_cache_stores

But this extension is not supported (yet) in WildFly infinispan subsystem - as Keycloak relies on WildFly.

So I expect the following tasks to get Infinispan Redis Store available for WildFly, and so Keycloak:

• Create jboss module for infinispam redis store jar - see modules/system/layers/base/org/infinispan/

• Create a "Custom Cache Store" factory able to instanciate Redis Store objects (store, servers and connection pool) from WildFly configuration key/value properties. This has to be added to WildFly as a jboss module too

• Use WildFly infinispan subsystem "local-cache" "custom" to configure this factory with attributes class and properties: https://wildscribe.github.io/WildFly/11.0/subsystem/infinispan/cache-container/local-cache/store/custom/index.html

Work is in progress and it is possible code and configuration will be published.

Verbify

By the way, standalone-ha where Infinispan is configured with JGroups "IP multicast" address is expected to implement dynamic cluster replication without need for Redis... – Verbify

I don't know if there are any updates from wildfly on usage of these stores. Another option of configuring keycloak to use a remote infinispan instance over HotRod and then having the infinispan cluster basically proxy to redis as the "cache store", I haven't done this myself. – Veinule

@YvesMartin thanks for the explanation. Please did you eventually publish the blog post and sample code/configuration somewhere? – Catching

Answer 3

Any particular reason for using a Redis store behind Infinispan?

A simpler solution might be to configure persistence to file or shared DB. For a cache use case like this, file based persistence might be enough. See here for example on configuring Infinispan with file based persistence. Alternatively, you can store to shared DB, e.g. Postgresql, but that requires more set up (see ref card for example).

Package

True but in case you want to scale up dynamically, having to configure/register a (maybe long) list of server nodes (either for infinispan or postgresql) is a big constraint. Having a Redis Store as infinispan backend behind a dynamic load balancer avoids any maintenance when scaling up. (we deploy in Openshift / Kubernetes) – Verbify

If using remote Infinispan clients, these take initial Infinispan server list and then discover nodes dynamically. The binary protocol linking Infinispan clients and servers handles topology information. – Derive