increment a count value outside parallel.foreach scope

Question

How can I increment an integer value outside the scope of a parallel.foreach loop? What's is the lightest way to synchronize access to objects outside parallel loops?

var count = 0;
Parallel.ForEach(collection, item =>
{
  action(item);
  // increment count??
}

Ambassadress

Answer 1

I like to beat dead horses! :)

The "lightest" way to increment the count from multiple threads is:

Interlocked.Increment(ref count);

But as others have pointed out: if you're doing it inside Parallel.ForEach then you're probably doing something wrong.

I'm suspecting that for some reason you're using ForEach but you need an index to the item you're processing (it will never work with Parallel.ForEach). Am I close? Why do you need the count? What sort of VooDoo magic are you trying to do?

UPDATE:

You seem to be safe with the ConcurrentDictionary if your key is the URI and the value is TAnswer. I don't see a problem as long as you don't try to use the count to reference elements in your collection.

Finally...

If you need a counter, then use the Parallel.For loop... it safely increments the counter for you.

Benil

Fetching thousands of web pages using multiple threads given a list of starting uris. I'm interested in seeing how many of them have been processed so far. Using a concurrent dictionaries seems to cover my needs actually: <code>ConcurrentDictionary<Uri, TAnswer></code>. Then I can use the <code>Count</code> property of the dictionary. Still there are some cases where processing a sequence of items does not produce any results like fetched html documents. In such simple cases I thought I could use a just a variable with proper synchronization and wanted to know how to do it. – Ambassadress

Another use for a Parallel.ForEach with a count variable: Calculating pixel-by-pixel similarity between images. – Vehicular

When I try this, I get the following error, any idea?Property or indexer may not be passed as an out or ref parameter. I had an int I bind as the content for a label to keep track of how many items have been processed. I'd love to threadsafe increment it but no joy so far. – Thorny

Answer 2

Use Interlocked.Increment method in this way.

int count = 0;
Parallel.ForEach(users, (u) =>
{
    var currentCount = Interlocked.Increment(ref count);
    Log(String.Format("Step {0} of {1}", currentCount, users.Count));
});

Triolet

Thank you, all I want the count for is to show % progress. – Jackboot

Answer 3

Parallel.Foreach comes along with an extra overload that perfectly fits for that kind of scenario when you just want to count things inside the loop.

    int totalCount = 0;
    Parallel.ForEach(files, () => 0, (file, loopState, localCount) =>
    {
        if(ProcessFile(file))
           localCount++;
    
        return localCount;
    
    }, localCount => Interlocked.Add(ref totalCount, localCount));

    Console.WriteLine($"Processed {totalCount}/{files.Length} files.");

Into the loop body you can inject a thread local count varialble and increment/decrement it as usual. This way you do not have an InterLocked call in every loop cycle. Instead there is just one Interlocked.Add call at the end of each parallel Task summing up totalCount.

Note that with this method you should not use totalCount from within the loop body. If your problem requires to access totalCount from within the loop body you have to use the InterLocked.Incerment method as describe in other answers.

Adorno

Answer 4

Use Interlocked.Increment.

I wouldn't increment things inside a parallel foreach (unless, of course, you're using Interlocked.Increment or some other locking mechanism). That's not what it's for. The Parallel foreach should be run only with actions that cause no side effects in shared state. To increment a value in a parallel foreach will cause the very problem you're more than likely trying to avoid.

Tinhorn

agree. My question is why he needs that count. Check if all actions are finished? There should be better way to sync that. – Bosnia

What about tracking the progress of something fairly long running (minutes)? I need to do this to feedback to the client so they can gauge wait length and see progress. – Lankester

I too need to report a % complete. So if I have 10,000 iterations, I want to track % complete and send a message for every 1% increase. So when iteration 100 completed I need to detect that. – Cassiecassil