boost::python - how to invoke a python function in its own thread from C++?

Question

I have a module written in python. this module is sort of an interface to many different functionalities I implemented in Python:

EmbeddingInterface.py simply imports this module and creates an instance:

import CPPController

cppControllerInstance = CPPController()

I would like to use cppControllerInstance in c++. this is what I have done so far:

#include <Python.h>
#include <boost\python.hpp>

using namespace boost;

python::object createController()
{
    try
    {
        Py_Initialize();

        python::object mainModule = python::import("__main__");
        python::object mainNamespace = mainModule.attr("__dict__");

        python::dict locals;

        python::exec(
            "print \"loading python implementetion:\"\n"
            "import sys\n"
            "sys.path.insert(0, \"C:\\Projects\\Python\\ProjectName\\Panda\")\n"
            "import EmbeddingInterface\n"
            "controller = EmbeddingInterface.cppControllerInstance\n",
            mainNamespace, locals);

            python::object controller = locals["controller"];
            return controller;
    }
    catch(...) {}
}

The Problem:

This 'controller' has some functions which must be called asynchronously. its work is continuous and in addition it can throw exceptions. which is why std::async sounded great.

But it doesn't work:

int main()
{
    python::object controller = createController();
    python::object loadScene = controller.attr("loadScene");
    //loadScene(); // works OK but blocking!
    std::async(loadScene); // non blocking but nothing happens!
    while(true); // do some stuff
}

I tried to invoke the python function 'loadScene' with its own thread but the function seemed to be blocking. It never returns.

What is the proper way of doing that?

Denominationalism

Answer 1

Seems you misunderstood the behavior of std::async

a snippet of test code:

#include <iostream>
#include <chrono>
#include <thread>
#include <future>

int doSomething(){
  std::cout << "do something"<<std::endl;
  return 1;
}

int main(){
   auto f = std::async(doSomething);

   std::this_thread::sleep_for(std::chrono::seconds(3));
   std::cout <<"wait a while"<<std::endl;
   f.get();
   return 0;
}

Output:

wait a while
do something

change the line

auto f = std::async(doSomething);

to

auto f = std::async(std::launch::async,doSomething);

Then output:

do something
wait a while

As your example, to run it immediately in another thread, you can try :

std::async(std::launch::async,loadScene);

Columnar

Indeed, at the time I did not understand the behavior of std::async and the fact the std::future destructor blocks. But regarding calling a python function on a different c++ thread, it is not as simple as you illustrate and requires manipulating with the GIL (python global interpreter lock). – Denominationalism

Answer 2

Your current codes are blocking because of while(true); There is no way to escape that.

Here is a way to do it

int taskDone; //Need to be accessible from loadScene

int main()
{
    python::object controller = createController();
    python::object loadScene = controller.attr("loadScene");
    //loadScene(); // works OK but blocking!
    taskDone=0; //Mean not done    
    std::async(loadScene); // In loadScene function, you need to do taskDone=1 when completes;
    while(!taskDone); // when the loadScene completes, this variable will exit the while loop
    //Do something
}

In reality, there should be event mechanism to inform finishing from this thread to other threads. And also data synchronization mechanism (mutex) to prevent 2 thread to change a variable at the same time (data racing).

Policy