C# chunked array

Question

I need to allocate very large arrays of simple structs (1 GB RAM). After a few allocations/deallocations the memory becomes fragmented and an OutOfMemory exception is thrown.

This is under 32 bit. I'd rather not use 64 bit due to the performance penalty I get - the same application runs 30% slower in 64 bit mode.

Are you aware of some implementations of IList compatible arrays which allocate memory in chunks and not all at once? That would avoid my memory fragmentation problem.

Malia

Odd, this is really the wrong question. It should have asked "why is by 64-bit program 30% slower?" Your 32-bit process is going to be a lot slower too after you chunk the arrays. – Letha

Originally the app was 64 bit, but after reading some articles that suggest that the 64 bit JIT is not so good at optimizing code as the 32 bit one (https://mcmap.net/q/672266/-64bit-net-performance-tuning), I've converted it to 32 bit and found this to be true. Even if I lose speed here, it will still start faster than the 64 bit version - I'm using IronPython + C# + C++/CLI, and the 64 bit version start takes 10 seconds longer (due to IronPython) – Malia

Answer 1

Josh Williams presented a BigArray<T> class on his blog using a chunked array:

BigArray<T>, getting around the 2GB array size limit

You will find more useful information in this related question:

C# huge size 2-dim arrays

A simple ad-hoc fix might be to enable the 3GB switch for your application. Doing so allows your application to use more than the 2GB per-process limit of 32-bit Windows. However, be aware that the maximum object size that the CLR allows is still 2GB. The switch can be enabled using a post-built action for your main executable:

call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin.exe /LARGEADDRESSAWARE "$(TargetPath)"

Realism

I've already done the /LARGEADDRESSAWARE trick and it helped a lot – Malia

Answer 2

When instantiating an array, .Net tries to find a contiguous part of memory for your array. Since total memory limit for a 32-bit app is 2Gb, you can see that it is going to be tough to find such block after several allocations.

1. You can try using something like a LinkedList<T>, to avoid the need for contiguous allocation, or restructure your code to make these chunks smaller (although you will not be completely safe this won't happen with a 500Mb array also).

2. On the other hand, one solution would be to instantiate this large buffer only once, at the start of your app, and then implement an algorithm which would reuse this same space during you app's lifetime.

3. If you can use IEnumerable instead of IList to pass your data to rest of your program, you would be able to collapse this list using the SelectMany LINQ method.

4. And at the end, you can simply implement the IList interface in a custom class, and use several smaller arrays under the hood.

Adiabatic

Answer 3

Would LinkedList work for you? http://msdn.microsoft.com/en-us/library/he2s3bh7.aspx

Genesisgenet

It's too wasteful to store each item (8 bytes) in a separate node – Malia