When to use Parallel.For?

Question

I have recently moved to C#.net 4.

I love Parallel.For, but not sure when to use and when not to. I know that when order is not important for me - I'll use it.

But are there any tests regarding the overhead of working with Parallels? Meaning, if my loop runs only 10 times (and performs very little logic) - should I avoid Parallels? Are there any thumb rules?

Lozada

Here is a really good free ebook from Microsoft that covers this topic "Patterns for Parallel Programming: Understanding and Applying Parallel Patterns with the .NET Framework 4" – Stripling

Answer 1

I would avoid using Parallel.For unless performance is an issue.

Writing code that runs concurrently is in general harder than writing single threaded code. Furthermore if you make an error due to a concurrency issue it can be difficult to debug it. For example the bug might occur only sometimes and not be easily reproducible. Unless you have a specific need for increased performance I would suggest that you keep it simple and use an ordinary loop on a single thread.

Catena

I agree - don't just use it because it's there, instead, try to solve an actual problem with it. Also, profiling is your friend ;) – Euphrasy

I agree with Mark. Use least possible resources for a job. – Bondmaid

Answer 2

The Parallel.For loop uses ThreadPool to execute work in a loop by invoking a delegate once per each iteration of a loop.

The general idea of how Parallel.For works can be presented as follows:

public static void MyParallelFor(int inclusiveLowerBound, int exclusiveUpperBound, Action<int> body)
{
    // Get the number of processors, initialize the number of remaining
    // threads, and set the starting point for the iteration.
    int numProcs = Environment.ProcessorCount;
    int remainingWorkItems = numProcs;
    int nextIteration = inclusiveLowerBound;
    using (ManualResetEvent mre = new ManualResetEvent(false))
    {
        // Create each of the work items.
        for (int p = 0; p < numProcs; p++)
        {
            ThreadPool.QueueUserWorkItem(delegate
            {
                int index;
                while ((index = Interlocked.Increment(ref nextIteration) - 1) < exclusiveUpperBound)
                    body(index);

                if (Interlocked.Decrement(ref remainingWorkItems) == 0)
                    mre.Set();
            });
        }
        // Wait for all threads to complete.
        mre.WaitOne();
    }
}

Parallel.For returns ParallelLoopResult value type, which contains details on the completed loop. One of its overloads is as follows:

public static ParallelLoopResult For(int fromInclusive, int toExclusive, Action<int> body);

It's important to realize that parallel execution is not always faster than serial execution. To decide whether to use parallel or not you have to estimate the workload that will do per iteration of a loop. If the actual work being performed by the loop is small relatively to thread synchronization cost, it's better to use ordinary loop.

This is one of example when serial for loop performance is faster that parallel:

static void Main(string[] args)
{
    Action<int> action = new Action<int>(SimpleMethod);

    // ordinary For loop performance estimation
    var sw = Stopwatch.StartNew();

    for(int i = 0; i < 1000; i++)
        action(i);

    Console.WriteLine("{0} sec.", sw.Elapsed.TotalSeconds);

    // parallel For loop performance estimation
    sw = Stopwatch.StartNew();

    Parallel.For(0, 1000, action);

    Console.WriteLine("{0} sec.", sw.Elapsed.TotalSeconds);
}

static void SimpleMethod(int index)
{
    int d = 1;
    int result = index / d;
}

Output:

0.0001963 sec.
0.0346729 sec.

Servitude

Answer 3

Quoting SQLite FAQ: 'Threads are evil. Avoid them'

Parallelization is useful for performance. Application performance optimization is one of the most counter-intuitive things in software design, and should be done with extreme care, using the right measurement tools, or it will just look funny.

Some would optimize UI code to respond in a microsecond instead of milliseconds, clearly having no value and causing lots of damage.

Fernandafernande

Some truth in this, but altogether too simple. See for example #3416019 – Gualterio

Answer 4

Composability must be part of the answer. It's not same using parallelism in a library, where composability must be a priority (since you don't know how your clients will end using your code) than in an application, where you have some idea of how to combine your parts. The composability of parallel code is not that bad with .NET, but when you decide your matrix multiplication must be the piece of code that run in all the available core, you are making harder for the user of the library to run two different algorithms using matrix multiplication in parallel. In fact, I prefer to use SIMD in libraries whenever possible: it creates less clashes... except when you mix SSE and AVX in older CPUs.

I'll put an example, because the bot is complaining: let's say we want to write a matrix multiplication method. If you force Parallel.For or any other similar mechanism (explicit Tasks, for instance) in the common execution mode, the clients of the library will be using several cores for a matrix multiplication. Is that really what they want? Maybe not: their matrix multiplication may be part of a larger algorithm that needs parallelization in other, most effective, part of it.

On the other hand, you can't fail when parallelizing code with AVX or any other SIMD flavor. You are not taking away options from you library's consumers.

Note: the bot asks me additional citations or documentations. Sorry, Mr. Bot, but this is something that caught me up sometime ago. Original research. Sometimes, it happens.

Corabelle

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Plumage