Why I can't create an array with large size?

Question

Why it is impossible to create an array with max int size?

int i = 2147483647;
int[] array = new int[i];

I found this explanation:

Java arrays are accessed via 32-bit ints, resulting in a maximum theoretical array size of 2147483647 elements.

But as you can see my code doesn't work. It is also impossible to create an array with size

new int[Integer.MAX_VALUE - 5];

Technical details

• 64-Bit HotSpot JVM
• OSX 10.10.4

PS

And why -5 actually?

Descent

Do you have the required 8 GB of memory for your array? – Kendy

As your extract shows, it's a maximum theoretical limit. Also, you say "doesn't work"; what's the error? I'd wager for an OOM – Furculum

@Kendy Yes, I use VM flag -Xmx12g. – Descent

@Furculum Yes, I got OOM. – Descent

possible duplicate of Do Java arrays have a maximum size? – Watchful

Answer 1

Theory

There are two possible exceptions:

• OutOfMemoryError: Java heap space means your array does not fit into java heap space. In order to solve you can increase the maximum heap size by using JVM option -Xmx. Also take into account that the maximum size of object cannot be larger than the largest heap generation.
• OutOfMemoryError: Requested array size exceeds VM limit means platform-specific size was exceeded:
  • the upper bound limit is set by the restrictions of the size type used to describe an index in the array, so theoretical array size is limited by 2^31-1=2147483647 elements.
  • the other limit is JVM/platform specific. According to chapter 10: Arrays of The Java Language Specification, Java SE 7 Edition there is no strict limit on array length, thus array size may be reduced without violating JLS.

Practice

In HotSpot JVM array size is limited by internal representation. In the GC code JVM passes around the size of an array in heap words as an int then converts back from heap words to jint this may cause an overflow. So in order to avoid crashes and unexpected behavior the maximum array length is limited by (max size - header size). Where header size depends on C/C++ compiler which was used to build the JVM you are running(gcc for linux, clang for macos), and runtime settings(like UseCompressedClassPointers). For example on my linux:

• Java HotSpot(TM) 64-Bit Server VM 1.6.0_45 limit Integer.MAX_VALUE
• Java HotSpot(TM) 64-Bit Server VM 1.7.0_72 limit Integer.MAX_VALUE-1
• Java HotSpot(TM) 64-Bit Server VM 1.8.0_40 limit Integer.MAX_VALUE-2

Useful Links

• https://bugs.openjdk.java.net/browse/JDK-8059914
• https://bugs.openjdk.java.net/browse/JDK-8029587

Seineetmarne

But I can use only Integer.MAX_VALUE-5 in array size on 64-Bit Server VM 1.7.0_72. So the limit is not Integer.MAX_VALUE-1, right? – Descent

@Descent this is because you are running JVM on another OS. This limit is platform and JVM version specific – Seineetmarne

The question is why should anyone add the length (in array elements) to the header size (in bytes). That mixing of units doesn’t make the slightest sense. E.g, as you wrote, 64-Bit Server VM 1.8.0_40 limits int[] to size to Integer.MAX_VALUE-2. So what does that gain, when we know that its object header has the size of at least four ints (64 bit mark word, 32 bit or 64 bit class word (depending on whether compressed), 32 bit array length)? The object size in 32 bit words still is larger than jint’s maximum, the object size in bytes is already far beyond that. – Forestaysail

Is there any kind of exposed API to query this limit? Something like System.getMaxArraySize()? – Wild

Answer 2

Some VMs reserve some header words in an array.

The maximum "safe" number would be 2 147 483 639 (Integer.MAX_VALUE - 8)

Source-http://www.docjar.com/html/api/java/util/ArrayList.java.html

**
  191        * The maximum size of array to allocate.
  192        * Some VMs reserve some header words in an array.
  193        * Attempts to allocate larger arrays may result in
  194        * OutOfMemoryError: Requested array size exceeds VM limit
  195        */
  196       private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

So It depends on the maximum memory available to your JVM on your SYSTEM NOW

Edit : Why It's Showing OOM.

Number of Elements = 2 147 483 639

number of bytes required for one element = 4

Total Memory for just Element 8589934556 KB == 8.589934555999999 GB

Now If the total memory usage of the array is not a multiple of 8 bytes, then the size is rounded up to the next mutlitple of 8 .

So You need more than what you are allocating due to Overheads too and that should be continuous memory

Untuck

I still get OOM error even if with new int[Integer.MAX_VALUE - 8];. – Descent

What is Your Ram Size ? – Untuck

I can confirm that MAX_VALUE - 8 works for me: OpenJDK Runtime Environment (build 1.8.0_45-b14) / OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode). With -Xmx12g. (Though my poor little machine with 8G RAM took minutes to recover from the abuse!) – Silurian

Maybe because you need object header .. though i'm not sure why the engineers designed it that way – Untuck

Answer 3

It's not enough to just have enough heap for that allocation; you need to have a single heap region of sufficient size. As you know, heap is divided into generations.

For a single allocation of 8 GB you must ensure that much for a single heap region (plus some overhead). With 12 GB of -Xmx you may still be short. Use additional options to control the size of the Old Generation.

Olsen

Good one but why single heap region is needed? Java heap memory is not contagious, so why in this case JVM is trying to do a contagious allocation? – Snooty

Arrays are always allocated in contiguous blocks by HotSpot. – Olsen

Hmmm... I set my heap size to -Xms12g -Xmx12g (min and max values with the reserve) and now it works. – Descent

@MarkoTopolnik Oh I see. Thanks. Is it only case with HotSpot, or is it a JVM that all JVM implementations must do contagious allocation for arrays? Is there any other case where contagious allocation is mandatory? Could you help with some reference where I can read this information? – Snooty

The specification does not mandate it, but probably all OpenJdk derivatives do it. Without that you'd get a significant performance hit for each access. – Olsen

Any reference where I can do more reading about it? Is there any other case where contagious allocation is mandatory? – Snooty

Sorry, I'm on phone currently. – Olsen

Ok, not a problem. Thank for the note. I will wait, meanwhile I will try to find it. – Snooty

Answer 4

find your max heap size by going to cmd and enter this line

javaw -XX:+PrintFlagsFinal | find "MaxHeapSize"

and then divide it by 1.5, you will get the approximate maximum size of the array for your computer

Indianapolis

Answer 5

Well, Ivan has already correctly pointed out that the array length does have a well defined upper limit and that it is again JVM/Platform dependent. In fact, more importantly, he also stated that how much length of array you can actually create in your code will mainly be controlled by how much max heap space you have allocated to your program while execution.

I would just like to add small code snippet to support his explanation. For example, theoretically, an array [] should accept length <= INTEGER.MAX_VALUE - x (here x is header size that is again JVM/Platform specific) but suppose you run following Java program with VM option -Xmx32m then you will see that none of the created array reach the length close to MAX_ARRAY_SIZE (i.e. 2147483639)

byte[] array : 1 byte
0 l=1048576 s=1mb
1 l=2097152 s=2mb
2 l=4194304 s=4mb
3 l=8388608 s=8mb
java.lang.OutOfMemoryError: Java heap space l=16777216 s=16mb

char[] array : 2 byte
0 l=1048576 s=2mb
1 l=2097152 s=4mb
2 l=4194304 s=8mb
java.lang.OutOfMemoryError: Java heap space l=8388608 s=16mb

int[] array : 4 byte
0 l=1048576 s=4mb
1 l=2097152 s=8mb
java.lang.OutOfMemoryError: Java heap space l=4194304 s=16mb

double[] array : 8 byte
0 l=1048576 s=8mb
java.lang.OutOfMemoryError: Java heap space l=2097152 s=16mb

Below is the code:

    byte[] barray = null;
    System.out.println("\nbyte[] array : 1 byte");
    try {
        for (ii=0; ii < 32; ii++) {
            barray = new byte[(int)Math.pow(2, ii)*1024*1024];
            System.out.println(ii + " l=" + barray.length + " s=" + barray.length / (1024 * 1024) + "mb");
        }
    }
    catch (Throwable e) {
        barray = null;
        System.out.println(e + " l=" + (int)Math.pow(2, ii)*1024*1024 + " s=" + (int)Math.pow(2, ii)*1024*1024 / (1024 * 1024) + "mb");
    }

    char[] carray = null;
    System.out.println("\nchar[] array : 2 byte");
    try {
        for (ii=0; ii < 32; ii++) {
            carray = new char[(int)Math.pow(2, ii)*1024*1024];
            System.out.println(ii + " l=" + carray.length + " s=" + 2*carray.length / (1024 * 1024) + "mb");
        }
    }
    catch (Throwable e) {
        carray = null;
        System.out.println(e + " l=" + (int)Math.pow(2, ii)*1024*1024 + " s=" + 2*(int)Math.pow(2, ii)*1024*1024 / (1024 * 1024) + "mb");
    }

    int[] iarray = null;
    System.out.println("\nint[] array : 4 byte");
    try {
        for (ii=0; ii < 32; ii++) {
            iarray = new int[(int)Math.pow(2, ii)*1024*1024];
            System.out.println(ii + " l=" + iarray.length + " s=" + 4*iarray.length / (1024 * 1024) + "mb");
        }
    }
    catch (Throwable e) {
        iarray = null;
        System.out.println(e + " l=" + (int)Math.pow(2, ii)*1024*1024 + " s=" + 4*(int)Math.pow(2, ii)*1024*1024 / (1024 * 1024) + "mb");
    }

    double[] darray = null;
    System.out.println("\ndouble[] array : 8 byte");
    try {
        for (ii=0; ii < 32; ii++) {
            darray = new double[(int)Math.pow(2, ii)*1024*1024];
            System.out.println(ii + " l=" + darray.length + " s=" + 8*darray.length / (1024 * 1024) + "mb");
        }
    }
    catch (Throwable e) {
        darray = null;
        System.out.println(e + " l=" + (int)Math.pow(2, ii)*1024*1024 + " s=" + 8*(int)Math.pow(2, ii)*1024*1024 / (1024 * 1024) + "mb");
    }

Coma