PLINQ query giving overflow exception

Question

I'm running a PLINQ query as follows:

ParallelQuery<string> winningCombos = from n in nextComboMaker.GetNextCombo()
                                              .AsParallel().WithCancellation(_cancelSource.Token)
                                              where ComboWasAWinner(n) 
                                              select n;

ConcurrentBag<string> wins = new ConcurrentBag<string>();

foreach (var winningCombo in winningCombos)
{
        wins.Add(winningCombo);
        if (wins.Count == _maxWinsAllowed)
            break;
}

The GetNextCombo method is just returning the next combination of letters and numbers, with possibilities up to the billions/trillions.

Now this is throwing the exception when I choose a range of combinations greater than the allowed size of an Int32, it always throws when the counter of combos it's run is 2147483584.

I've made sure it's nothing in the GetNextCombo by creating a fake combo to return every time (doing a yield return "234gf24fa23..." etc.)

The exception is being thrown by LINQ:

System.AggregateException was unhandled by user code
  Message=One or more errors occurred.
  Source=System.Core
  StackTrace:
       at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
       at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
       at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
       at System.Linq.Parallel.ForAllOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings)
       at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream)
       at System.Linq.Parallel.WhereQueryOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings)
       at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream)
       at System.Linq.Parallel.ScanQueryOperator`1.ScanEnumerableQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
       at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
       at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
       at System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings)
       at System.Linq.Parallel.ForAllOperator`1.RunSynchronously()
       at StockWiz.Library.PLINQArrayProcessor.DoProcessing() in C:\Users\dad\Documents\BitBucket\stockwiz_clone\stockwiz\StockWiz.Library\PLINQArrayProcessor.cs:line 50
       at System.Threading.Tasks.Task.Execute()
  InnerException: System.OverflowException
       Message=Arithmetic operation resulted in an overflow.
       Source=System.Core
       StackTrace:
            at System.Linq.Parallel.PartitionedDataSource`1.ContiguousChunkLazyEnumerator.MoveNext(T& currentElement, Int32& currentKey)
            at System.Linq.Parallel.WhereQueryOperator`1.WhereQueryOperatorEnumerator`1.MoveNext(TInputOutput& currentElement, TKey& currentKey)
            at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
            at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
            at System.Linq.Parallel.SpoolingTaskBase.Work()
            at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
            at System.Threading.Tasks.Task.Execute()
       InnerException: 

I'm wondering if there's anything I can do to change this query to not overflow, any order in which I do things, etc. Maybe not have the linq query do a where and select, although I've tried this:

 var query = nextComboMaker.GetNextCombo().AsParallel();

 query.ForAll(x => if(ComboWasAWinner(x) wins.Add(x) );

still the same overflow.

Carbine

Answer 1

You should be able to get around this by using a custom partitioner. The default partitioner chosen by PLINQ in this case only supports int-range number of items.

See http://msdn.microsoft.com/en-us/library/dd997416.aspx for how to do this.

Intosh

won't seem to work for me, you have to do .ToArray or ToList() on the IEnumerable I'm parallel'ing over and that will create a stack overflow once the size is greater than an int. What specifically can I override to make it count with a long? – Carbine