Hazelcast vs. Ignite benchmark

Question

I am using data grids as my primary "database". I noticed a drastic difference between Hazelcast and Ignite query performance. I optimized my data grid usage by the proper custom serialization and indexes, but the difference is still noticeable IMO.

Since no one asked it here, I am going to answer my own question for all future references. This is not an abstract (learning) exercise, but a real-world benchmark, that models my data grid usage in large SaaS systems - primarily to display sorted and filtered paginated lists. I primarily wanted to know how much overhead my universal JDBC-ish data grid access layer adds compared to raw no-frameworks Hazelcast and Ignite usage. But since I am comparing apples to apples, here comes the benchmark.

Angulate

Answer 1

I have reviewed the provided code on GitHub and have many comments:

Indexing and Joins

1. Probably the most important point is that Apache Ignite indexing is a lot more sophisticated than Hazelcast. Unlike Hazelcast, Ignite supports ANSI 99 SQL, so you can write your queries at will.
2. Most importantly, unlike Hazelcast, Ignite supports group-indexes and SQL JOINs across different caches or data types. Imagine that you have Person and Organization tables, and you need to select all Persons working for the same Organization. This would be impossible to do in 1 step in Hazelcast (correct me if I am wrong), but in Ignite it is a simple SQL JOIN query.

Given the above, Ignite indexes will take a bit longer to create, especially in your test, where you have 7 of them.

Fixes in TestEntity Class

In your code, the entity you store in cache, TestEntity, recalculates the value for idSort, createdAtSort, and modifiedAtSort every time the getter is called. Ignite calls these getters several times while the entity is being stored in the index tree. A simple fix to the TestEntity class provides 4x performance improvement: https://gist.github.com/dsetrakyan/6bfe089d53f888448503

Heap Measurement is Not Accurate

The way you measure heap is incorrect. You should at least call System.gc() before taking the heap measurement, and even that would not be accurate. For example, in the results below, I get negative heap size using your method.

Warmup

Every benchmark requires a warm-up. For example, when I apply the TestEntity fix, as suggested above, and do the cache population and queries 2 times, I get better results.

MySQL Comparison

I don't think comparing a single-node Data Grid test to MySQL is fair, neither for Ignite, nor for Hazelcast. Databases have their own caching and whenever working with such small memory sizes, you are usually testing database in-memory cache vs. Data Grid in-memory cache.

The performance benefit usually comes in whenever doing a distributed test over a partitioned cache. This way a Data Grid will execute the query on each cluster node in parallel and the results should come back a lot faster.

Results

Here are the results I got for Apache Ignite. They look a lot better after I made the aforementioned fixes.

Note that the 2nd time we execute the cache population and cache queries, we get better results because the HotSpot JVM is warmed up.

It is worth mentioning that Ignite does not cache query results. Every time you run the query, you are executing it from scratch.

[00:45:15] Ignite node started OK (id=0960e091, grid=Benchmark)
[00:45:15] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=8.0GB]
Starting - used heap: 225847216 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1001824120 bytes
Cache: 100000 entries, heap size: 775976904 bytes, inserts took 14819 ms
------------------------------------
Starting - used heap: 1139467848 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 978473664 bytes
Cache: 100000 entries, heap size: **-160994184** bytes, inserts took 11082 ms
------------------------------------
Query 1 count: 100, time: 110 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 285 ms, heap size: 1037116472 bytes
Query 3 count: 100, time: 19 ms, heap size: 1037116472 bytes
Query 4 count: 100, time: 123 ms, heap size: 1037116472 bytes
------------------------------------
Query 1 count: 100, time: 10 ms, heap size: 1037116472 bytes
Query 2 count: 100, time: 116 ms, heap size: 1056692952 bytes
Query 3 count: 100, time: 6 ms, heap size: 1056692952 bytes
Query 4 count: 100, time: 119 ms, heap size: 1056692952 bytes
------------------------------------
[00:45:52] Ignite node stopped OK [uptime=00:00:36:515]

I will create another GitHub repo with the corrected code and post it here when I am more awake (coffee is not helping anymore).

Mercurialism

Thanks, Dmitriy. Just the brief history of my project. I'll write you, guys, a separate letter privately. I have big plans for Ignite. – Angulate

Thanks, Dmitriy. Just the brief history of my project. I started Px100 with Hazelcast (it was using standard SQL/JPA in the previous version if you are wondering). I wasn't naive thinking it'd outperform MySQL by simply being "in memory", but I expected at least comparable performance. Like everyone else, I used Serializable and did't care about indexes. I optimized it per Hazelcast's official recommendation. It improved things a little, but the problems reappeared on large data sets. I'll write you, guys, a separate letter about Px100 privately. I have big plans for Ignite. – Angulate

@AlexRogachevsky Alex, sounds good. Looking forward to hearing the plans you have for Apache Ignite. – Mercurialism

Dmitry, I followed your suggestion regarding calculated getters, however you should also make sure you call them once and then pass that value along. Those getters are quite common for anything, but dumb DTO-style entities. The second question I have is about compound indexes. I'd like to have IgniteBiTuple explained in the javadoc + detailed examples in the documentation. And what exactly the boolean means: probably ascending. Things like that are not always "a horse would understand kind of obvious". Lastly you have a serious performance bug - when specifying log4j logging w/o log4j config. – Angulate

@AlexRogachevsky Here is the documentation of the Ignite SQL functionality, including compound indexes: apacheignite.gridgain.org/v1.3/docs/sql-queries – Mercurialism

@AlexRogachevsky As far as Getters, I am not sure I agree, but we will consider it. What you were doing is recalculating the same value over and over again every time the getter was called, which is not common. – Mercurialism

@AlexRogachevsky Thanks for noticing the log4j with null-appender bug (for some reason it was turning on debug mode) - already fixed in master and will be part of the next release. – Mercurialism

Dmitry, I configure indexes programmatically. BTW it doesn't seem to work. I cannot have vendor-specific annotations in a universal framework/API. Please revise the javadoc for CacheTypeMetadata class, particularly setGroups() method. It currently reads "Sets group-indexed fields." and the complicated parameter: a map of maps of IgniteBiTuple tells me very little. You could streamline that signature too. You probably didn't count on anyone using it instead of annotations. I do. As far, as getters, I refactored the ones you saw, but I still have many calculating values from changed fields. – Angulate

Getters and setters exist for a reason. There can be logic in them, and completely transient calculated fields. E.g. an entity, InsuranceApplication has a List<Person>, which in turn has a List<String> of phones. I want to find all applications, where the 5th oldest person's 3rd phone starts w/ 310. You could write a complex (and slow) query to do it at runtime, or you can have a getter calculating it and cache/store that value with the "query fields". Essentially those getters are custom SQL functions. Don't think in terms of SQL. In NoSQL you don't need to have a "declared" physical field. – Angulate

Does the documentation explain this? It has no effect on performance as well. CacheTypeMetadata type = new CacheTypeMetadata(); LinkedHashMap<String, IgniteBiTuple<Class<?>,Boolean>> compoundIndex = new LinkedHashMap<>(); compoundIndex.put("textField", new IgniteBiTuple<>(String.class, true)); compoundIndex.put("id", new IgniteBiTuple<>(Long.class, false)); type.getGroups().put("compIdx1", compoundIndex); – Angulate

Answer 2

Here is the benchmark source code: https://github.com/a-rog/px100data/tree/master/examples/HazelcastVsIgnite

It is part of the JDBC-ish NoSQL framework I mentioned earlier: Px100 Data

Building and running it:

cd <project-dir>
mvn clean package
cd target
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.HazelcastTest 100000
java -cp "grid-benchmark.jar:lib/*" -Xms512m -Xmx3000m -Xss4m com.px100systems.platform.benchmark.IgniteTest 100000

As you can see, I set the memory limits high to avoid garbage collection. You can also run my own framework test (see Px100DataTest.java) and compare to the two above, but let's concentrate on pure performance. Neither test uses Spring or anything else except for Hazelcast 3.5.1 and Ignite 1.3.3 - the latest at the moment.

The benchmark transactionally inserts the specified number of appr. 1K-size records (100000 of them - you can increase it, but beware of memory) in batches (transactions) of 1000. Then it executes two queries with ascending and descending sort: four total. All query fields and ORDER BY are indexed.

I am not going to post the entire class (download it from GitHub). The Hazelcast query looks like this:

PagingPredicate predicate = new PagingPredicate(
        new Predicates.AndPredicate(new Predicates.LikePredicate("textField", "%Jane%"),
            new Predicates.GreaterLessPredicate("id", first.getId(), false, false)),
        (o1, o2) -> ((TestEntity)o1.getValue()).getId().compareTo(((TestEntity)o2.getValue()).getId()),
        100);

The matching Ignite query:

SqlQuery<Object, TestEntity> query = new SqlQuery<>(TestEntity.class,
        "FROM TestEntity WHERE textField LIKE '%Jane%' AND id > '" + first.getId() + "' ORDER BY id LIMIT 100");
    query.setPageSize(100);

Here are the results executed on my 2012 8-core MBP with 8G of memory:

Hazelcast

Starting - used heap: 49791048 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 580885264 bytes
Map: 100000 entries, used heap: 531094216 bytes, inserts took 5458 ms
Query 1 count: 100, time: 344 ms, heap size: 298844824 bytes
Query 2 count: 100, time: 115 ms, heap size: 454902648 bytes
Query 3 count: 100, time: 165 ms, heap size: 657153784 bytes
Query 4 count: 100, time: 106 ms, heap size: 811155544 bytes

Ignite

Starting - used heap: 100261632 bytes
Inserting 100000 records: ....................................................................................................
Inserted all records - used heap: 1241999968 bytes
Cache: 100000 entries, heap size: 1141738336 bytes, inserts took 14387 ms
Query 1 count: 100, time: 222 ms, heap size: 917907456 bytes
Query 2 count: 100, time: 128 ms, heap size: 926325264 bytes
Query 3 count: 100, time: 7 ms, heap size: 926325264 bytes
Query 4 count: 100, time: 103 ms, heap size: 934743064 bytes 

One obvious difference is the insert performance - noticeable in real life. However very rarely one inserts a 1000 records. Typically it is one insert or update (saving entered user's data, etc.), so it doesn't bother me. However the query performance does. Most data-centric business software is read-heavy.

Note the memory consumption. Ignite is much more RAM-hungry than Hazelcast. Which can explain better query performance. Well, if I decided to use an in-memory grid, should I worry about the memory?

You can clearly tell when data grids hit indexes and when they don't, how they cache compiled queries (the 7ms one), etc. I don't want to speculate and will let you play with it, as well, as Hazelcast and Ignite developers provide some insight.

As far, as general performance, it is comparable, if not below MySQL. IMO in-memory technology should do better. I am sure both companies will take notes.

The results above are pretty close. However when used within Px100 Data and the higher level Px100 (which heavily relies on indexed "sort fields" for pagination) Ignite pulls ahead and is better suited for my framework. I care primarily about the query performance.

Angulate

"if I decided to use an in-memory grid, should I worry about the memory?", well, yeah, you should, unless you want a query to bring down your cluster with an OOME. Of course, performance is really really important, but that doesn't mean you shouldn't worry about memory usage. – Trichinopoly