HashMap performance Java 9 25% less than Java 8?

Question

Note: This is not about a performance issue. I only observe a difference in performance I cannot explain / understand.

While benchmarking some newly developed code, targeted for Java 9 I discovered something weird. A (very) simple benchmark of HashMap with 5 keys reveals that Java 9 is much slower than Java 8. Can this be explained or is my (benchmark) code just plain wrong?

Code:

@Fork(
    jvmArgsAppend = {"-Xmx512M", "-disablesystemassertions"}
)
public class JsonBenchmark {

    @State(Scope.Thread)
    public static class Data {

        final static Locale RUSSIAN = new Locale("ru");
        final static Locale DUTCH = new Locale("nl");

        final Map<Locale, String> hashmap = new HashMap<>();

        public Data() {
            hashmap.put(Locale.ENGLISH, "Flat flashing adjustable for flat angled roof with swivel");
            hashmap.put(Locale.FRENCH, "Solin pour toit plat inclinée");
            hashmap.put(Locale.GERMAN, "Flachdachkragen Flach Schrägdach");
            hashmap.put(DUTCH, "Plakplaat vlak/hellend dak inclusief glijschaal");
            hashmap.put(RUSSIAN, "Проход через плоскую кровлю регулир. для накл. кровли");
        }

    }

    @Benchmark
    public int bmHashMap(JsonBenchmark.Data data) {
        final Map<Locale, String> m = data.hashmap;
        int sum = 0;
        sum += m.get(Data.RUSSIAN).length();
        sum += m.get(Locale.FRENCH).length();
        sum += m.get(Data.DUTCH).length();
        sum += m.get(Locale.ENGLISH).length();
        sum += m.get(Locale.GERMAN).length();
        return sum;
    }    

}

Results:

• Java 8_151: JsonBenchmark.bmHashMap thrpt 40 47948546.439 ± 560763.711 ops/s
• Java 9_181: JsonBenchmark.bmHashMap thrpt 40 34962904.479 ± 276045.691 ops/s (-/- 27%!)

UPDATE

Thanks for the answers and great comments.

1. Suggestion by @Holger. My first reaction was: That must be the explanation. However, if I only benchmark the String#length() function, there is no significant difference in performance. And, when I only benchmark the HashMap#get() methods (as suggested by @Eugene) there is still a difference of about 10 - 12 %.

2. Suggestion by @Eugene. I changed the parameters (more warmup iterations, more memory), but I am not able to reproduce your outcome. I increased the heap to 4G however. But this cannot explain the difference, is not it?

3. Suggestion by @Alan Bateman. Yes, this improves the performance! However, still a difference of around 20%.

Highpressure

Answer 1

You are testing more than just HashMap. You are not only calling HashMap.get, you are implicitly calling Locale.hashCode and Locale.equals. Further, you are calling String.length.

Now, all four could have changed their performance characteristics, so you would need far more tests to reason about which method(s) exhibit(s) different performance.

But the hottest candidate is String.length. In Java 9, the String class does not use a char[] array anymore, but a byte[] array, to encode Latin 1 strings using only one byte per character, dramatically reducing the memory footprint of typical applications. This, however, implies that the length is not always identical to the array length anymore. So the complexity of this operation has changed.

But keep in mind that your result is about 77 ns difference in a microbenchmark. This is not enough to estimate the impact on a real application…

Lycopodium

Though this could be a little much of an ask and I wouldn't mind you denying this altogether. But (being a novice in jmh) if possible, could you state a complexity comparison of the .length in both the versions. Would be great to see that in the answer and statistical while understanding. Looked up here to see if I could find out something, but most of it went over my head. – Leatheroid

Good answer. Another thing to mention is -XX:-CompactStrings to see the performance different when not using compact strings. – Sphene

@nullpointer: The old implementation was like array.length, the new implementation is like arrays.length>>coder wheres coder is either zero or one, depending on the string. The impact one a real application depends on how often length() is actually invoked and the ratio between latin1 strings and other strings. Some other string operations might be even faster for latin1 string (less bytes to shovel around), which may more than compensate. – Lycopodium

@Lycopodium I like how Aleksey (the same person working on this and jol) said that adding coder field would not make String class any bigger, the space for it was already there because of 8 bytes alignment. – Sempstress

@Eugene: and it’s still less fields than it had back then, when it had an offset and length field (before Java7, update6). – Lycopodium

@Sempstress Do you have a link/reference to this statement? – Highpressure

@Highpressure I'll try to dig it out... it was from a JUG if i recall correctly. – Sempstress

Answer 2

I had a hint this was about jmh set-up, more then it was about HashMap. As already noted you are measuring a lot more than simply HashMap::get here. But even so, I had doubts that java-9 would be that much slower, so I measured myself (latest jmh build from sources, java-8 and 9).

I haven't changed your code - just added way more heap (10GB) and way more warm-ups, thus reducing the "error" you see after ±

Using java-8:

Benchmark            Mode  Cnt   Score   Error  Units
SOExample.bmHashMap  avgt   25  22.059 ± 0.276  ns/op

Using java-9:

Benchmark            Mode  Cnt   Score   Error  Units
SOExample.bmHashMap  avgt   25  23.954 ± 0.383  ns/op

The results are on-par pretty much without a noticeable difference (these are nano-seconds after all) as you see it. Also if you really want to test just HashMap::get than your methods could simply return the invocation of that, like this:

@Benchmark
@Fork(5)
public int bmHashMap(SOExample.Data data) {
    return data.hashmap.get(data.key); // where key is a random generated possible key
}

Sempstress