maximum limit on Java array

Question

I am trying to create 2D array in Java as follows:

int[][] adjecancy = new int[96295][96295];

but it is failing with the following error:

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/04/07 11:58:55 - please wait.
JVMDUMP032I JVM requested System dump using 'C:\eclipse\workspaces\TryJavaProj\core.20170407.115855.7840.0001.dmp' in response to an event
JVMDUMP010I System dump written to C:\eclipse\workspaces\TryJavaProj\core.20170407.115855.7840.0001.dmp
JVMDUMP032I JVM requested Heap dump using 'C:\eclipse\workspaces\TryJavaProj\heapdump.20170407.115855.7840.0002.phd' in response to an event
JVMDUMP010I Heap dump written to C:\eclipse\workspaces\TryJavaProj\heapdump.20170407.115855.7840.0002.phd

A way to solve this is by increasing the JVM memory but I am trying to submit the code for an online coding challenge. There it is also failing and I will not be able to change the settings there.

Is there any standard limit or guidance for creating large arrays which one should not exceed?

Remediable

Does it have to be a 2D array? – Procora

You are trying to allocate 37GB of memory. This is quite a lot and even with increasing JVM memory would require a big machine. You need to find a smarter algorithm (that's why it's called a coding challenge) – Stater

You're asking if it's possible to allocate 40gb of memory without using 40gb of memory? No, it's not. If you tell us why you're trying to do that, maybe we can help improve your solution. – Maddeu

You should think of a solution that does not require storing almost 10 billion ints in main memory. – Cacuminal

The name of the variable suggests you're creating an adjacency matrix. You're attempting to do it in the naive way, which will only work for small matrices. It wouldn't be much of a challenge if you could solve it in the most obvious way. – Shipper

Answer 1

int[][] adjecancy = new int[96295][96295];

When you do that you are trying to allocate 96525*96525*32 bits which is nearly 37091 MB which is nearly 37 gigs. That is highly impossible to get the memory from a PC for Java alone.

I don't think you need that much data in your hand on initialization of your program. Probably you have to look at ArrayList which gives you dynamic allocation of size and then keep on freeing up at runtime is a key to consider.

There is no limit or restriction to create an array. As long as you have memory, you can use it. But keep in mind that you should not hold a block of memory which makes JVM life hectic.

Mcalpine

Answer 2

Array must obviously fit into memory. If it does not, the typical solutions are:

• Do you really need int (max value 2,147,483,647)? Maybe byte (max value 127) or short is good enough? byte is 8 times smaller than int.
• Do you have really many identical values in array (like zeros)? Try to use sparse arrays.

for instance:

Map<Integer, Map<Integer, Integer>> map = new HashMap<>();
map.put(27, new HashMap<Integer, Integer>()); // row 27 exists
map.get(27).put(54, 1); // row 27, column 54 has value 1.

They need more memory per value stored, but have basically no limits on the array space (you can use Long rather than Integer as index to make them really huge).

• Maybe you just do not know how long the array should be? Try ArrayList, it self-resizes. Use ArrayList of ArrayLists for 2D array.

• If nothing else is helpful, use RandomAccessFile to store your overgrown data into the filesystem. 100 Gb or about are not a problem in these times on a good workstation, you just need to compute the required offset in the file. The filesystem is obviously much slower than RAM but with good SSD drive may be bearable.

Cadell

I like this answer alot, because it not only states the obvious, but also outlines multiple possible solutions without spoiling the challenge. There is probably a trick to this challenge, like a more memory-efficient representation for a sparse matrix. – Procora

Answer 3

It is recommended to allocate Maximum Heap Size that can be allocated is 1/4th of the Machine RAM Size.

1 int in Java takes 4 bytes and your array allocation needs approximately 37.09GB of Memory.

In that case even if I assume you are allocating Full Heap to just an Array your machine should be around 148GB RAM. That is huge.

Have a look at below.

Ref: http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gc-ergonomics.html

Hope this helps.

Golf

Answer 4

It depends on maximum memory available to your JVM and the content type of the array. For int we have 4 bytes of memory. Now if 1 MB of memory is available on your machine , it can hold maximum of 1024 * 256 integers(1 MB = 1024 * 1024 bytes). Keeping that in mind you can create your 2D array accordingly.

Hemophilia

Answer 5

Array that you can create depends upon JVM heap size.

96295*96295*4(bytes per number) = 37,090,908,100 bytes = ~34.54 GBytes. Most JVMs in competitive code judges don't have that much memory. Hence the error.

To get a good idea of what array size you can use for given heap size - Run this code snippet with different -Xmx settings:

    Scanner scanner = new Scanner(System.in);
    while(true){
        System.out.println("Enter 2-D array of size: ");
        size = scanner.nextInt();

        int [][]numbers = new int[size][size];
        numbers = null;
    }

e.g. with -Xmx 512M -> 2-D array of ~10k+ elements.

Generally most of online judges have ~1.5-2GB heap while evaluating submissions.

Stralsund