Why is Max size of int array lower than Int32.MaxValue? [duplicate]

Question

Even though this post says it should work, if you create an int array of size Int32.MaxValue, it throws an OutOfMemoryException: Array dimensions exceeded supported range.

From my testing, it seems like the maximum size that an array can be initialized to is Int32.MaxValue - 1048576 (2,146,435,071). 1048576 is 2^20. So only this works:

var maxSizeOfIntArray = Int32.MaxValue - 1048576;
var array = new int[maxSizeOfIntArray];

Does any one know why? Is there a way to create a larger integer array?

PS: I need to use arrays instead of lists because of a Math.Net library that only returns arrays for sets of random numbers that are cryptographically secure pseudo random number generator

Yes I have looked at the other questions linked but they are not correct as those questions say the largest size is Int32.MaxValue which is not the same as what my computer lets me do

Yes, I do know the size of the array will be 8GB, I need to generate a data set of billions of rows in order to test the randomness with the die harder suite of tests

I also tried the option of creating a BigArray<T> but that doesn't seem to be supported in C# anymore. I found one implementation of it, but that throws an IndexOutOfRangeException at index 524287, even though I set the array size to 3 million.

Stafani

Times 4 bytes per int is quite a chunk of memory. Are you really sure you need that much? – Tomtit

a library that only returns arrays then why would you need to create the array? – Glaive

https://mcmap.net/q/113019/-can-39-t-create-huge-arrays – Film

As @BrianWilson explains in his answer, that array would consume 8GB of memory. That's a lot. What is your use case for an array of such an immense dimension. You may be better off with a sparse array implementation (the easiest of which is implemented over top of a Dictionary<int, T> for a sparse array of type T). I suspect you cannot allocate more than the GC's ephemeral segment size in a single allocation: learn.microsoft.com/en-us/dotnet/standard/garbage-collection/… – Dairy

@HansKesting I need it to create a super big file of >10 billion numbers, in order to test the bias in a random number generator using dieharder – Stafani

When you need to create a file, then you could just write to the file without containing everything in memory first – Tomtit

Answer 1

An Int32 is 32 bits, or 4 bytes. The max value of an Int32 is 2,147,483,647. So, if you could create an array of 2,147,483,647 elements, where each element is 4 bytes, you would need a contiguous piece of memory that is 8GB in size. That is ridiculously huge, and even if your machine had 128GB of RAM (and you were running in a 64-bit process), that would be outside of realistic proportions. If you really need to use that much memory (and your system has it), I would recommend going to native code (i.e., C++).

Liverish

How is using "native" code supposed to help memory consumption for the storage of integers? – Dearly

How does this answer the OP's question? And how would using Native code change anything? – Bledsoe

Mark Benningfield, I'm sorry, if you want an explanation of how/why native code gives you more power and flexibility on memory usage and handling, please ask a question. I'll say this, though---.NET (and .NET core) are virtual machines that are written themselves in native code. You can do anything .NET can do in native code, and more. You are much closer to the hardware. – Liverish

Johan Donne: same thing: I'm not going to use a StackOverflow comment to explain how native code works. Feel free to ask a question. – Liverish