.NET multithreading, volatile and memory model

Question

Assume that we have the following code:

class Program
 {
    static volatile bool flag1;
    static volatile bool flag2;
    static volatile int val;
    static void Main(string[] args)
    {
      for (int i = 0; i < 10000 * 10000; i++)
      {
        if (i % 500000 == 0)
        {
          Console.WriteLine("{0:#,0}",i);
        }

        flag1 = false;
        flag2 = false;
        val = 0;

        Parallel.Invoke(A1, A2);

        if (val == 0)
          throw new Exception(string.Format("{0:#,0}: {1}, {2}", i, flag1, flag2));
      }
    }

    static void A1()
    {
      flag2 = true;
      if (flag1)
        val = 1;
    }
    static void A2()
    {
      flag1 = true;
      if (flag2)
        val = 2;
    }
  }
}

It's fault! The main quastion is Why... I suppose that CPU reorder operations with flag1 = true; and if(flag2) statement, but variables flag1 and flag2 marked as volatile fields...

Colous

Answer 1

In the .NET memory model, the runtime (CLI) will ensure that changes to volatile fields are not cached in registers, so a change on any thread is immediately seen on other threads (NB this is not true in other memory models, including Java's).

But this says nothing about the relative ordering of operations across multiple, volatile or not, fields.

To provide a consistent ordering across multiple fields you need to use a lock (or a memory barrier, either explicitly or implicitly with one of the methods that include a memory barrier).

For more details see "Concurrent Programming on Windows", Joe Duffy, AW, 2008

Judaize

Thank you wery match! But in the article about .net memory model ("Understand the Impact of Low-Lock Techniques in Multithreaded Apps" msdn.microsoft.com/en-us/magazine/cc163715.aspx) in the section "A Relaxed Model: ECMA" we can see, that 1. Reads and writes cannot move before a volatile read. 2. Reads and writes cannot move after a volatile write. Is it wrong? I just can understand... May be in the article means somethig else?... – Colous

+1 for mentioning the book - currently reading it myself (already halfway through ;-) – Hoy

@fedor-serdukov: I might be miss-remembering -- but I try and be significantly safer than required to avoid unexpected behaviour (especially after a few maintenance changes). Also, I don't think your code needs out of order for the exception to be throw. A1 and A2 could interleave so the two flags are set before either condition is checked, in fact once the thread pool is spun up, I would expect this to happen on a multi-core system from time to time. – Judaize

The important point here is the one about a memory barrier. Without that, there is nothing preventing a multi-core cpu from having stale data in their respective caches. Volatile fields only prevents the compiler from reusing a fetched value, and it also has a bearing on the reordering process, but one cpu/core might see writes retiring in a different order than you expect, until you add explicit memory barriers. See the wikipedia article on memory barriers: en.wikipedia.org/wiki/Memory_barrier – Gasworks

@Lasse: This is true in general, but the .NET memory model provides more guarantees than the general case (e.g. the double checked pattern works in .NET, but not on other platforms). – Judaize

Ok, good to know. I'm a bit out on thin ice when I deal with multithreading, so I tend to program overly defensive. – Gasworks

@Lasse: See my comment above... I might now the platform will give me more, but I generally don't rely on it except in very narrow cases. No shared data is a better approach anyway. – Judaize

Answer 2

ECMA-335 specification says:

A volatile read has “acquire semantics” meaning that the read is guaranteed to occur prior to any references to memory that occur after the read instruction in the CIL instruction sequence. A volatile write has “release semantics” meaning that the write is guaranteed to happen after any memory references prior to the write instruction in the CIL instruction sequence. A conforming implementation of the CLI shall guarantee this semantics of volatile operations. This ensures that all threads will observe volatile writes performed by any other thread in the order they were performed. But a conforming implementation is not required to provide a single total ordering of volatile writes as seen from all threads of execution.

Let's draw how it looks:

So, we have two half-fences: one for volatile write and one for volatile read. And they are not protecting us from reordering of instructions between them.
Moreover, even on such strict architecture like AMD64 (x86-64) it is allowed stores to be reordered after loads.
And for other architectures with weaker hardware memory model you can observe even more funny stuff. On ARM you can get partially constructed object observed if reference was assigned in non-volatile way.

To fix your example you should just put Thread.MemoryBarrier() calls between assignment and if-clause:

static void A1()
{
  flag2 = true;
  Thread.MemoryBarrier();
  if (flag1)
    val = 1;
}
static void A2()
{
  flag1 = true;
  Thread.MemoryBarrier();
  if (flag2)
    val = 2;
}

This will protect us from reordering of these instructions by adding full-fence.

Schacker