I have some code to process several million data rows in my own R-like C# DataFrame class. There's a number of Parallel.ForEach calls for iterating over the data rows in parallel. This code has been running for over a year using VS2013 and .NET 4.5 without issues.
I have two dev machines (A and B) and recently upgraded machine A to VS2015. I started noticing a strange intermittent freeze in my code about half the time. Letting it run for a long time, it turns out that the code does eventually finish. It just takes 15-120 minutes instead of 1-2 minutes.
Attempts to Break All using the VS2015 debugger keep failing for some reason. So I inserted a bunch of log statements. It turns out that this freeze occurs when there is a Gen2 collection during a Parallel.ForEach loop (comparing the collection count before and after each Parallel.ForEach loop). The entire extra 13-118 minutes is spent inside whichever Parallel.ForEach loop call happens to overlap with a Gen2 collection (if any). If there are no Gen2 collections during any Parallel.ForEach loops (about 50% of the time when I run it), then everything finishes fine in 1-2 minutes.
When I run the same code in VS2013 on Machine A, I get the same freezes. When I run the code in VS2013 on Machine B (which was never upgraded), it works perfectly. It ran dozens of time overnight with no freezing.
Some things I've noticed / tried:
- The freezes happen with or without the debugger attached on Machine A (I figured it was something with the VS2015 debugger at first)
- The freezes happen whether I build in Debug or Release mode
- The freezes happen if I target .NET 4.5 or .NET 4.6
- I tried disabling RyuJIT but that did not affect the freezes
I'm not changing the default GC settings at all. According to GCSettings, all runs are happening with LatencyMode Interactive and IsServerGC as false.
I could just switch to LowLatency before every call to Parallel.ForEach, but I'd really prefer to understand what's going on.
Has anyone else seen strange freezes in Parallel.ForEach after the VS2015 upgrade? Any ideas on what a good next step would be?
UPDATE 1: Adding some sample code to the nebulous explanation above...
Here is some sample code that I hope will demonstrate this issue. This code runs in 10-12 seconds on B machine, consistently. It encounters a number of Gen2 collections, but they take almost no time at all. If I uncomment the two GC settings lines, I can force it to have no Gen2 collections. It's somewhat slower then at 30-50 seconds.
Now on my A machine, the code takes a random amount of time. Seems to be between 5 and 30 minutes. And it seems to get worse, the more Gen2 collections it encounters. If I uncomment the two GC settings lines, it takes 30-50 seconds on Machine A also (same as Machine B).
It might take some tweaking in terms of the number of rows and array size for this to show up on another machine.
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using System.Linq;
using System.Runtime;
public class MyDataRow
{
public int Id { get; set; }
public double Value { get; set; }
public double DerivedValuesSum { get; set; }
public double[] DerivedValues { get; set; }
}
class Program
{
static void Example()
{
const int numRows = 2000000;
const int tempArraySize = 250;
var r = new Random();
var dataFrame = new List<MyDataRow>(numRows);
for (int i = 0; i < numRows; i++) dataFrame.Add(new MyDataRow { Id = i, Value = r.NextDouble() });
Stopwatch stw = Stopwatch.StartNew();
int gcs0Initial = GC.CollectionCount(0);
int gcs1Initial = GC.CollectionCount(1);
int gcs2Initial = GC.CollectionCount(2);
//GCSettings.LatencyMode = GCLatencyMode.LowLatency;
Parallel.ForEach(dataFrame, dr =>
{
double[] tempArray = new double[tempArraySize];
for (int j = 0; j < tempArraySize; j++) tempArray[j] = Math.Pow(dr.Value, j);
dr.DerivedValuesSum = tempArray.Sum();
dr.DerivedValues = tempArray.ToArray();
});
int gcs0Final = GC.CollectionCount(0);
int gcs1Final = GC.CollectionCount(1);
int gcs2Final = GC.CollectionCount(2);
stw.Stop();
//GCSettings.LatencyMode = GCLatencyMode.Interactive;
Console.Out.WriteLine("ElapsedTime = {0} Seconds ({1} Minutes)", stw.Elapsed.TotalSeconds, stw.Elapsed.TotalMinutes);
Console.Out.WriteLine("Gcs0 = {0} = {1} - {2}", gcs0Final - gcs0Initial, gcs0Final, gcs0Initial);
Console.Out.WriteLine("Gcs1 = {0} = {1} - {2}", gcs1Final - gcs1Initial, gcs1Final, gcs1Initial);
Console.Out.WriteLine("Gcs2 = {0} = {1} - {2}", gcs2Final - gcs2Initial, gcs2Final, gcs2Initial);
Console.Out.WriteLine("Press Any Key To Exit...");
Console.In.ReadLine();
}
static void Main(string[] args)
{
Example();
}
}
UPDATE 2: Just to move things out of the comments for future readers...
This hotfix: https://support.microsoft.com/en-us/kb/3088957 totally fixes the issue. I'm not seeing any slowness issues at all after applying.
It turned out not to have anything to do with Parallel.ForEach I believe based on this: http://blogs.msdn.com/b/maoni/archive/2015/08/12/gen2-free-list-changes-in-clr-4-6-gc.aspx though the hotfix does mention Parallel.ForEach for some reason.
GC.Collect()? – Chromolithography