Clustering and Shared Data in Vert.x

Question

I'm developing in Vert.x (based on Netty and Hazelcast), and I'm trying to share data between two server instances (eache of those instances in different machines, on the same lan).

My problem is that I don't know how to configure the vert.x servers to allow them to share their concurrent memory maps (the theory says that's possible).

I've read many documents off Vert.x and Hazelcast but I haven't had results yet. (I don't know how to force vert.x to load hazelcast xml configuration files).

Thanks in advance!

Lunn

Answer 1

There are options for sharing data among vertx instances on different machines

Option 1.

You could use the Vert.x ClusterManager and it's maps:

ClusterManager clusterManager = ((VertxInternal)vertx).clusterManager();
Map map = clusterManager.getSyncMap("mapName"); // shared distributed map

That map is backed by a Hazelcast IMap and is distributed. This assumes you're running vertx with the -cluster parameter and have configured clustering.

However note that this is internal API and is not generally recommended for production. If you are doing a one time experiment then it could be useful.

Option 2.

You can get access to Hazelcast once vertx is started in clustered mode:

Set<HazelcastInstance> instances = Hazelcast.getAllHazelcastInstances();
HazelcastInstance hz = instances.stream().findFirst().get();
Map map = hz.getMap("mapName"); // shared distributed map

Joinder

Went with option 1 and wrapped it in a class. If we want to update vertx in the future where this breaks, we've got one-stop shopping. FYI the aim is to register for notifications to monitor for downed nodes. – Olnton

Vert.x 3 (and above) have a public async API to give you access to the plugable cluster manager - which may be implemented by Hazelcast IMDG, Apache Zookeeper, Apache Ignite, Infinispan (vertx.io/docs/#clustering) or others - the API is pretty simple, if you want, for example, to implement an ETCd (github.com/guoyu511/vertx-etcd) or Consul (github.com/reactiverse/consul-cluster-manager) cluster managers. See other answers for details. – Recognize

Answer 2

With Vert.x 3 - if you configure your Vert.x instances into "clustered mode" (which can be as simple as adding -cluster to the command line of the Vert.x launcher, see here for details), then you can use the SharedData interface to get access to "distributed maps" which allows cluster members to read and write data across the cluster transparently.

Example:

if (vertx.isClustered()) {
    log.info("Using clustered data store");
    vertx.sharedData().<String, MyEntity>getClusterWideMap("entities", 
            res -> {
                AsyncMap<String, MyEntity> dataMap = res.result();
                setDataStore(dataMap);
            });
}

Recognize

Answer 3

Afaik you can't share data between different instances of vert.x -- from the documentation

"[...] Such a use case is better solved by providing a shared map structure that can be accessed directly by different verticle instances in the same vert.x instance."

Since "vert.x instance" means "jvm instance" you can't use sharedmap/set between different jvm. You can use the event bus for this.

Edit before others people downvotes: my answer is from 2012, 6 years ago, when this was not possible. Now it is possible as others people already said

Inculcate

I'm not completely familiar with vert.x & not trying to disagree with your statement - this should be possible using hazelcast (maps are replicated). – Maisiemaison

@Lunn The documentation also says: "In later versions of vert.x we aim to extend this to allow data to be shared by all vert.x instances in the cluster." So keep an eye on that. – Detrital

@CarloBertuccini is there any way to use Hazelcast data structures in Vert.x? – Reine

Answer 4

Vert.x 2 does not support cluster-wide shared data. However, Vert.x 3 does expose an asynchronous API that wraps the underlying Hazelcast cluster manager.

For Vert.x 2, though, you can use the Hazelcast instance directly in your worker verticles. Just use Hazelcast's static methods to get the Vert.x Hazelcast instance:

HazelcastInstance hazelcast = Hazelcast.getAllHazelcastInstances().iterator().next();

Note that you should only access the Hazelcast API directly in this way from within a worker verticle. The Hazelcast API is blocking, so it will block the event loop if used in a normal verticle.

Seventeenth

Answer 5

As has been pointed out, the data sharing objects bundled in Vert.x don't support sharing data across multiple Vert.x instances. In order to do that you would have to do either:

1. Use a "normal" shared database.
2. Set up a Verticle which manages the SharedMap instances and publishes and listens for updates to and from the rest of the cluster.
3. Design your application so you don't need to share data in this way.

Flagg