Matlab engines within parallel loop

Question

I am having some trouble utilising multiple matlab engines from within a piece of parallelised code. I can successfully spawn multiple engines using engOpenSingleUse but am unable to communicate with more than one engine (ie. calls to engPutVariable fail).

As always, a minimal (VS) example:

#include "stdafx.h"
#include <engine.h>
#include <omp.h>

int _tmain(int argc, _TCHAR* argv[])
{
//First spawn the matlab engine sessions
Engine *m_Engines[2];
for (int i = 0; i < 2; i++)
{
    m_Engines[i] = engOpenSingleUse(NULL, NULL, NULL);
}

//Then spawn the worker threads...
#pragma omp parallel num_threads(2)
{   
    // Allocate an engine to each thread
    int thread_num = omp_get_thread_num();
    Engine *thisEngine = m_Engines[thread_num];

    #pragma omp for
    for (int i = 0; i < 10000; i++)
    {
        // Create an mxArray and stick some data in it
        mxArray* mM = NULL;
        mM = mxCreateDoubleMatrix(1, 1, mxREAL);
        double data[1] = { 1.0 };
        memcpy((void *)mxGetPr(mM), (void *)data, sizeof(data));

        // Send it across to matlab
        engPutVariable(thisEngine, "A", mM);
        // Run some algorithm
        engEvalString(thisEngine, "A=A+1;");
        // Retrieve result
        mM = engGetVariable(thisEngine, "A");

        // Get it out of the mxarray
        double A = *mxGetPr(mM);
    }
}

return 0;
}

Any ideas? Am using Matlab R2012b on Win x64.

Thriftless

Try moving the engOpenSingleUse call inside the parallel region. Also note that the MATLAB engine functions are not thread-safe and care (read: synchronisation) is necessary when calling them. – Napper

Possible duplicate of Thread safety of Matlab engine API – Bader

Answer 1

As commented below your question, if you deliberatly call functions of a non-thread safe closed source library from multiple thread with no synchronization, then the behaviour is not predictable and you should avoid doing it. A possible idea is to span calls to MATLAB engine across multiple processes, but please consider that it may increase complexity (inter-process synchronization is required) and worsen performances.

By the way, C++11 and later users should program by using the new MATLAB Engine API for C++. Please note that engOpenSingleUse has not an equivalent function, at least not explicitly, in the new API (the closest replacement is matlab::engine::startMATLAB).

Polyhedron

"if you deliberatly call functions of a non-thread safe closed source library from multiple thread with no synchronization, then the behaviour is not predictable" You can remove "closed source" from that sentence and still be correct. I'm not sure why that needs to be specified there. – Could

@Chris You are right, of course. I wanted to emphasized that without even access to the source code, you can really predict nothing about what is going to happen (i.e. you are literally playing with a black box). – Polyhedron