Embedded Python: Multiple Sub-Interpreters not working

Question

I'm trying to understand sub-interpreters and GIL. But my experiment is failing often(The same code rarely works).

Gathering info from SO questions and few sites, I have the following code which spawns 2 non-python threads. Each of these threads are given a python sub-interpreter. I want to release GIL inside these threads and call a DLL function in C++(though this example does not detail that. Here I just write to stdout). Basically I want to see concurrency in the execution(Non-Python DLL invocation).

#include <iostream>
#include <thread>
#include <Python.h>

void worker(PyInterpreterState* interp, int n) 
{
    PyThreadState* ts;
    ts = PyThreadState_New(interp);

    PyThreadState_Swap(ts);

    PyThreadState* _save;
    _save = PyEval_SaveThread();

    std::cout << n << std::endl;  // Non-Python execution. My Focus.

    PyEval_RestoreThread(_save);

    PyThreadState_Swap(ts);

    PyThreadState_Clear(ts);
    PyThreadState_DeleteCurrent();

    return;
}


int main()
{
    try 
    {
        Py_Initialize();

        PyEval_InitThreads();

        PyThreadState* _main = PyThreadState_Get();

        PyThreadState* i1 = Py_NewInterpreter();
        PyThreadState* i2 = Py_NewInterpreter();

        std::thread t1(worker, i1->interp, 1);
        std::thread t2(worker, i2->interp, 2);

        t1.join();
        t2.join();

        PyThreadState_Swap(i1);
        PyThreadState_Clear(i1);
        Py_EndInterpreter(i1);
        PyThreadState_Swap(i2);
        PyThreadState_Clear(i2);
        Py_EndInterpreter(i2);

        PyThreadState_Swap(_main);

        Py_Finalize();
        return 0;
    }
    catch(std::exception& e)
    {
        std::cout << "Exception:" << std::endl << e.what() << std::endl;
    }
}

Running a single thread works all the time. When I run 2 threads as shown, I get any of the following outputs.

1. In PyEval_SaveThread(),

2
Fatal Python error: drop_gil: GIL is not locked
Python runtime state: initialized

Current thread 0x00002d08 (most recent call first):
<no Python frame>

2. In PyEval_SaveThread(),

1
Fatal Python error: PyEval_SaveThread: NULL tstate
Python runtime state: initialized

Current thread 0x00003eb8 (most recent call first):
<no Python frame>

Either of the thread succeeds, the other one fails.

3. Rarely works. The same code.

1
2

Can someone shed some light on this? Need help. Thanks.

Refutation

Does this answer your question? Python multi-thread multi-interpreter C API – Puggree

I realise the suggested duplicate is what you've based this question of. Note that they're getting the PyInterpreterState* with ts->interp and you're getting it by mis-casting the PyThreadState* – Puggree

Even ts->interp didn't help. Same issues. You are right, my question is based on the post you pointed, but that does not solve my problem. – Refutation

OK well what you're doing with casting a PyThreadState* to a PyInterpreterState* is definitely 100% wrong. I suggest you edit your post to fix that, and then hopefully someone else knows the answer to the rest of it. – Puggree

Edited the post. Thanks – Refutation

Answer 1

1. In your worker() function, you call PyThreadState_Swap(). The Python docs say that:

The global interpreter lock must be held and is not released.

You must acquire the GIL before calling PyThreadState_Swap() and release it before exiting worker().

2. In your main thread, you wait for the threads to terminate while holding the GIL. This is a deadlock as the threads will need to acquire the GIL to do any useful work in Python.

Please see my answer at https://mcmap.net/q/491765/-python-multi-thread-multi-interpreter-c-api for detailed working instructions on how to do that, and even a link to sample code.

Once you have the GIL, and the thread state for for the sub interpreter, so that it is now safe to call the regular Python API, you can just add a PyEval_SaveThread()/PyEval_RestoreThread() pair.

Quindecennial

Your GitHub code gave a lot of clarity. Thanks!. So, PyThreadState_Swap would return the previous state right? – Refutation