Why would OutOfMemoryException be thrown while using PLINQ Take()?

Question

After I upgraded to DotNet 4.5, a query started giving me OutOfMemoryExceptions.

The (distilled) query is:

var tests = new int[]{}
    .AsParallel()
    .GroupBy(_ => _)
    .Take(int.MaxValue)
    .ToArray();

I'm posting this for anyone with the same problem. I'll answer below.

Eddings

Are you running in 32bit or 64bits? – Paphian

I was using this kind of code inside a function that received a parameter with the number of elements to be returned. Passing int.MaxValue (there was an overload which did just that) essentially meant 'take everything'. – Eddings

Answer 1

It appears to be a change in the framework.

The Take() operator is implemented in the TakeOrSkipQueryOperator internal class. There is one branch in the code that goes through a WrapHelper() function that creates a FixedMapHeap instance which in turn creates an array of Key elements of the size passed originally to Take() (which would be an array of 8Gb in the given example).

Eddings

@Steven Set a Take value for which you have enough memory to store that many results. (Or get more memory.) – Tientiena

Sounds like a reasonable optimisation much of the time - should allow for better parallelisation in a lot of cases since different threads can write to different parts of that heap without risk of interfering with each other. The down-side in this case is clear though. – Hircine

you should just remove the Take from the query when you want all elements. – Ditmore