LinkedHashMap memory consumption

Question

The user uploads a huge file consisting of 1 million words. I parse the file and put the each line of the file into a LinkedHashMap<Integer, String>.

I need O(1) access and removal by key. Also, I need to preserve the access order, iterate from any position and sort.

The memory consumption is huge. I enabled Strings deduplication feature which appears in Java 8, but it turns out that the LinkedHashMap consumes most of the memory.

I found that LinkedHashMap.Entry consumes 40 bytes, but there are only 2 pointers - one for the next entry and one for the previous entry. I thought 1 pointer should be 64 bits or 32 bits. Buy if I divide 409,405,320(bytes) by 6,823,422(entries count) I have 60 bytes per entry.

I think I don't need the previous pointer, the next pointer should be enough to keep order. Why does LinkedHashMap consume so much memory? How can I reduce memory consumption?

Festoon

is it possible the Integer wrapper is using that much extra memory? Maybe an int-based implementation could help – Effulgence

@1blustone If you look at the image, you can see that Integers are taking up 16% of the heap. LinkedHashMap.Entrys are taking up more than 3x more. I believe the OP wants to know why this should be the case. – Zip

You are too quick when browsing sources. that entry inherits from HashMap.Node which has 4 more fields, and there are additional object headers whose size is just implementation detail. – Fagen

I've made a small test and my entry takes 40 bytes java.util.LinkedHashMap$Entry 240,034,920 (32.3%) 6,000,873 (24.5%). Maybe you have some memory leaks: (https://hoangx281283.wordpress.com/2012/11/18/wrong-use-of-linkedhashmap-causes-memory-leak/ – Kingmaker

Answer 1

How to reduce memory consumption?

1) Add -XX:+UseCompressedOops flag to your JVM startup.

2) Implement your own version of LinkedHashMap, optimized for your needs. I. e. use primitive int as a key instead of Integer, remove "previous" pointer if you don't need it, etc. Note that copying OpenJDK source might be impossible unless you wish to release your modified hash map implementation under GPLv2 license, because OpenJDK is GPLv2. However you can copy and modify LinkedHashMap implementation from Android Open Source Project, because it is Apache licensed.

Nolanolan