Allocating more than 1,000 MB of memory in 32-bit .NET process

Question

I am wondering why I'm not able to allocate more that 1,000 MB of memory in my 32-bit .NET process. The following mini application throws an OutOfMemoryException after having allocated 1,000 MB. Why 1,000 MB, and not say 1.8 GB? Is there a process-wide setting I could change?

static void Main(string[] args)
{
    ArrayList list = new ArrayList();
    int i = 0;
    while (true)
    {
        list.Add(new byte[1024 * 1024 * 10]); // 10 MB
        i += 10;
        Console.WriteLine(i);
    }
}

PS: Garbage collecting does not help.

Edit, to clarify what I want: I have written a server application which deals with very large amounts of data before writing to database/disk. Instead of creating temporary files for everything, I have written an in-memory cache, which makes the whole thing super-fast. But memory is limited, and so I tried to find out what the limits are. And wondered why my small test program threw the OutOfMemoryException after exactly 1,000 MB.

Deguzman

Answer 1

The virtual address space limit of a Win32 process is 1.5GB (not entirely true). Additionally in the .NET frameworks there is a limiter to the % of memory a .NET process can consume. The machine.config has a processModel element with an attribute memoryLimit which is the % of available memory a process can consume. The default value is 60%.

If the machine you're running on has 2GB of memory or you haven't enabled the /3GB switch in your BOOT.INI then you're going to get ~1.3GB of memory per process.

I can't find the KB article but if I remember correctly .NET 1.x cannot address beyond the 1.5GB (1.8GB?) limit regardless of your settings.

http://blogs.msdn.com/tmarq/archive/2007/06/25/some-history-on-the-asp-net-cache-memory-limits.aspx http://social.msdn.microsoft.com/Forums/en-US/clr/thread/c50ea343-b41b-467d-a457-c5a735e4dfff http://www.guidanceshare.com/wiki/ASP.NET_1.1_Performance_Guidelines_-_Caching#Configure_the_Memory_Limit

Erlond

Thank you, that gives me an idea. However, the links are not really helpful. – Deguzman

Only the server CLR uses the memoryLimit configuration. The workstation CLR always uses as much memory as it can get. – Certainly

@Certainly - I was never aware there was a difference. – Erlond

Answer 2

Having enormous blocks of memory is never a good idea, even in 64bit. You get big problems with contiguous memory and fragmentation.

The problem here is finding a contiguous block. You could try enabling 3gb mode (which might help it find a few more bytes) but I really advise against it. The answers here are:

• use less memory
• use a database/file system
• use x64

You might also want to read Eric Lippert's blog (he seems to have a blog entry for every common .NET question...)

Anathematize

Johannes, why would having more address space NOT make it easier to find more contiguous address space??? I am completely confused by your comment. Can you explain? – Nildanile

Answer 3

I have recently been doing extensive profiling around memory limits in .NET on a 32bit process. We all get bombarded by the idea that we can allocate up to 2.4GB (2^31) in a .NET application but unfortunately this is not true :(. The application process has that much space to use and the operating system does a great job managing it for us, however, .NET itself seems to have its own overhead which accounts for approximately 600-800MB for typical real world applications that push the memory limit. This means that as soon as you allocate an array of integers that takes about 1.4GB, you should expect to see an OutOfMemoryException().

Obviously in 64bit, this limit occurs way later (let's chat in 5 years :)), but the general size of everything in memory also grows (I am finding it's ~1.7 to ~2 times) because of the increased word size.

What I know for sure is that the Virtual Memory idea from the operating system definitely does NOT give you virtually endless allocation space within one process. It is only there so that the full 2.4GB is addressable to all the (many) applications running at one time.

I hope this insight helps somewhat.

I originally answered something related here (I am still a newby so am not sure how I am supposed to do these links):

Is there a memory limit for a single .NET process

Forth

Answer 4

You can allocate MUCH MORE memory than ~2 GB by building your application to a 64-bit architecture, which requires that you create a new build configuration in Visual Studio, and that build of the application will only run on 64-bit versions of Windows. In .NET, using the default "Any CPU" build option for your application, I find that I am only able to allocate about 1.5 GB of memory from the heap (even on 64-bit Windows machine), which is because the application actually only runs in 32-bit mode when it is built in "Any CPU" mode. But by compiling to x64 architecture, you can allocate much, much more memory from the heap during the execution of your application, and I will explain how to create a x64 build for your application below:

Again, using the normal (default) "Any CPU" build option in your .NET project, your application will ALWAYS run under 32-bit mode, even on a 64-bit Windows OS. Therefore you won't be able to allocate more than about 1.5 to 2 GB of RAM memory during application execution. To run your .NET application in true 64-bit mode, you will need to go into the build configuration manager and create a build type for the x64 architecture, and then recompile your program for x64 explicitly using that build type. The x64 build mode option can be created for your .NET solution using the following steps:

1. In the Visual Studio "Solution Explorer" pane, right click on the Solution icon and choose the "Configuration Manager" option from the pop-up menu. This will open the build "Configuration Manager" dialog window for the .NET Solution file.
2. On the right, top side of the build "Configuration Manager" dialog, click on the down arrow and select the "<new>" option. This will open the "New Solution Platform" dialog.
3. In the "New Solution Platform" dialog, for the "Platform" option, choose "x64" from the drop-down menu. Then click the "OK" button and the new, x64 build option will now be available in the Configuration Manager dialog.
4. Then, on the "Configuration Manager" dialog, select "x64" in the "Active Solution Platform" drop-down menu. The click the "Close" button.
5. In the Visual Studio "Solution Explorer" pane, right click on the CS Project icon and choose the "Properties" option from the pop-up menu (the last option at the bottom of this menu). This will open the CS Project properties window.
6. On left side of the CS Project properties window, click on the "Build" tab to show the build properties for your code project. At the top of this window, notice that the "Platform" should now say "x64" (as opposed to the default "Any CPU" option). If the "Platform" drop-down doesn't show "x64", you should select it now.
7. Then just build your code and in the "bin" folder, you should now have a x64 folder with the new 64-bit build of your application within it.

Using a 64-bit build of your application on a 64-bit Windows OS will allow your program to allocate much more than ~2GB of memory, presumably up to 2^64 address spaces (if you have the RAM and disk space available, which are the real limiting factors as of the time of writing this response).

If you're STILL running out of memory in your application, you can also increase the size of the Windows memory page file. On Windows, the page file allows the operating system to shift memory from RAM to the disk, if it runs out of RAM memory space. But there is a big time cost in shifting sections of RAM memory to and from the disk, so it may be a real hit on the performance of your application. Regardless of performance, by increasing the page size, you could (in theory) make the page file as large as there is free space available on the C: drive of your windows machine. In that case, your application would be able to allocate, for example, up to 4 TB of memory (or whatever amount of memory that your page file size is set to) during the execution of your program. To change the page file settings for your Windows machine, do the following:

1. Open the "System Properties" dialog by right clicking on "This PC" and choosing the "Properties" option on the pop-up menu. This can also be accomplished in later versions of Windows (Windows 10, Win 2012 Server, etc...) by going to "Start" > "Control Panel" > "System and Security" > "System".
2. On the left side of the "System" dialog, click on the "Advanced System Properties" option. This will show the "Advanced" tab of the legacy "System Properties" dialog for Windows.
3. On the "Advanced" tab of the "System Properties" dialog, click the "Settings" button in the "Performance" box. This will open the "Performance Options" dialog.
4. On the "Performance Options" dialog, click on the "Advanced" tab to see the current size setting for the Windows memory page file.
5. To increase the page file size, click on the "Change" button and the "Virtual Memory" dialog will be opened.
6. On the "Virtual Memory" dialog, select the "C:" drive, then under "Custom Size", set the "Initial" and "Maximum" sizes. You can use any size up to the maximum amount of free space on the C: drive, but making this change will reserve that space for the page file on the hard drive.
7. Then click "Ok" on all dialogs to commit the new settings. Then reboot your computer to ensure all changes have been completed properly and that the new page file settings are in operation.

Anyway, I hope this helps people understand why they can run into this 1.5 - 2 GB memory limitation issue in a .NET application, even when running on a 64-bit Windows machine. This can be a very confusing issue for people and I hope my explanation makes sense. Please feel free to message me with questions about this answer if needed.

Juryrig

Answer 5

Compile program as Any CPU and you will have unlimited amount of memory to use, and the program can still use x86 (32-bit) DLL Imports!!!

Micaelamicah

Answer 6

I think the issue here is that this application will be adding 10MB with every loop it makes, and the loop is: "while(true)" which means it will adding these 10MBs till the application is stopped. So if it were to run for 100 loop it would have added close to 1GBs to RAM, and I'm assuming that it would have done this in less than 30 seconds. My point is you are trying to 10 megabyte's of memory per loop, in a never ending loop

Bestiality

Answer 7

I am truly sorry if I didn't get your point but:

static void Main(string[] args)
{
    ArrayList list = new ArrayList();
    int i = 0;
    while (true)
    {
        using(byte newBt = new byte[1024 * 1024 * 10])
        {
            list.Add(newBt); // 10 MB
            i += 10;
            Console.WriteLine(i);
        }
    }
}

Have you tried the using method? And this might be a stupid question but why did you create an eternal loop? O if you try the code strip the symbols >.> xD.

Source: http://msdn.microsoft.com/en-us/library/yh598w02(v=vs.80).aspx

Undue

I've edited your answer to fix the code formatting. Next time you need to post code paste it, select it and press CTRL+K or the {} icon, it will be automatically formatted – Philbert

This totally misses the point of the question. The infinite loop allocates memory until it fails, thereby determining the maximum amount of memory the process can consume. Your code doesn't compile because it tries to assign a byte array to a byte. Also, neither of those implements IDisposable. And if it did free the memory on each iteration, it would break the test (it would eventually report about (2^31)/10, but that number would have nothing to do with how much memory it can allocate). – Terrilyn