Apache NiFi Distribute Map Cache problems in a cluster

Question

I moved a flow that was working on a single node NiFi to a 3 nodes NiFi cluster, and suddenly a wait processor that makes use of the DistributedMapCache doesn't works as expected, this means that wasn't able to recover a value from the cache. To be sure that the value exists in the cache I configured the cache to use a persistent directory, so I was able to check the content of the cache, after some testing a realize that:

• The cache was written in the node that have the flow that writes to the cache (DistributedMapCacheClientService points to localhost)
• The client is able to recover only values that are in the same node.

Another test was to force the execution of the flows in the Primary node, in this case everything works as expected.

Another test was to write the cache from all the nodes at the same time, only worked for the primary node.

So my questions are:

• Is the DistributedMapCache really distributed across a cluster?
• May I be missing some point or misconfiguring something?

Maladminister

Answer 1

You shouldn't be using localhost in DistributedMapCacheClientService, just use the hostname of one of the nodes, then all your clients will be pointing to the same server.

Diphthongize

Hi Bryan, thanks for your response, I'll try it asap, but with this approach, what happens if the node is down? – Veinstone

The DistributedMapCacheServer does not provide high availability so you will experience failures when it is down and it would be up to you to to handle the failures in the data flow by routing them appropriately, alternatively you can use an external cache via the RedisDistributedMapCacheClient, HBaseDistributedMapCacheClient, or CouchbaseDistributedMapCacheClient – Diphthongize

The idea is to use a Redis cache instead. – Veinstone

Thanks Bryan, now is working, I don't know/understand which is the difference, anyway I am going to move towards a real fault HA cache. – Veinstone

So this isn't really a distributed cache? – Scutt

I think "distributed" was probably an inaccurate name, its more like "remote". The idea is for a NiFi node to be able to access a cache that does not live on it's own node. Whether that cache supports HA, is dependent on the cache implementation. The provided DMC Server that can run in NiFi does not support HA, while using Redis/HBase/etc does. – Diphthongize