Why is calling a Python lambda expression from C# not thread-safe?

Question

I define a side-effect-free (pure) lambda expression in IronPython and assign it to a C# delegate. When invoking the delegate simultaneously from multiple threads i get exceptions of type AccessViolationException, NullReferenceException and FatalEngineExecutionError.

The occurance of the error is non-deterministic and it mostly takes several million iterations to provoke it, which says "race condition" to me. How can i avoid it?

The exceptions are only raised when running the process with x64 (x86 does not crash) and outside of the debugger. The test system is a Core I7 (8 threads) on Windows 7, .NET Framework 4.0 and IronPython 2.7.1.

Here's the minimal code to produce the error:

var engine = Python.CreateEngine();

double a = 1.0;
double b = 2.0;

while (true)
{
    Func<double, double, double> calculate = engine.Execute("lambda a,b : a+b");

    System.Threading.Tasks.Parallel.For(0, 1000, _ =>
    {
         for (int i = 0; i < 1000; i++) { calculate(a,b); }
    });

    Console.Write(".");   
}

Error message:

FatalExecutionEngineError was detected

Message: The runtime has encountered a fatal error. The address of the error was at 0xf807829e, on thread 0x3da0. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.

Update: Even if the engine is declared as thread-local, it crashes after some time:

var calculate = new ThreadLocal<Func<double, double, double>>(() => Python.CreateEngine().Execute("lambda a,b : a+b"));

Steverson

several million iterations of which loop? inner, parallel or while? – Charger

Answer 1

This is likely due to a race condition within the ScriptEngine. Note that ScriptEngine.Execute returns a reference to a PythonFunction rather than a Func (it's due to C#'s dynamic behavior, that you can treat the result as a Func. I'm no expert on IronPython but looking at the source of PythonFunction, there is no indication whatsoever that it is threadsafe.

Crista

Answer 2

Have you tried replacing the 64 bit data type (double) with a 32 bit data type (float)? I do know that "A simple read or write on a field of 32 bits or less is always atomic" whereas a 64 bit field can have problems depending on the processor and the code. It seems like kind of a long shot but it is worth a try.

Senseless

Changing double to float doesn't help. – Steverson

Answer 3

Running similar code (see below) on IronScheme, poses no such issues.

The only thing I can think of, is that engine.Execute("lambda a,b : a+b") creates an interpreted function instead of a compiled one. Combine that with a possibly thread unsafe interpreter, and kaboom.

double a = 1.0;
double b = 2.0;

while (true)
{
  var calculate = "(lambda (a b) (+ a b))".Eval<Callable>();

  System.Threading.Tasks.Parallel.For(0, 1000, _ =>
  {
    for (int i = 0; i < 1000; i++) { calculate.Call(a, b); }
  });

  Console.Write(".");
}

Charger