Deadlock when QThread tries to acquire Python GIL via PyGILState_Ensure()

Question

I have a C++/Qt application in which I want to embed the Python interpreter. I want to call Python from a QThread, but I'm getting a deadlock at the line where I call PyGILState_Ensure() in order to try to acquire the global interpreter lock (GIL).

I'll provide a minimal and straight-forward example below, which follows the recommendations given here:

//main.cpp:
#include <QCoreApplication>
#include <QThread>
#include "Worker.h"

void startThread()
{
    QThread* thread = new QThread;
    Worker* worker = new Worker();
    worker->moveToThread(thread);
    QObject::connect(thread, SIGNAL(started()), worker, SLOT(process()));
    QObject::connect(worker, SIGNAL(finished()), thread, SLOT(quit()));
    QObject::connect(worker, SIGNAL(finished()), worker, SLOT(deleteLater()));
    QObject::connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    thread->start();
}

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    Py_Initialize();
    startThread();
    Py_FinalizeEx();
    return a.exec();
}


//Worker.h:
#ifndef WORKER_H
#define WORKER_H

#include <QObject>
#include "Python.h"

class Worker : public QObject
{
    Q_OBJECT
public:
    explicit Worker(QObject *parent = nullptr) : QObject(parent) {}

Q_SIGNALS:
    void finished();

public Q_SLOTS:
    void process()
    {
        qDebug("Calling Python");
        PyGILState_STATE gstate = PyGILState_Ensure();
        PyRun_SimpleString("print(\"hello\")");
        PyGILState_Release(gstate);
        qDebug("Done calling Python");
        Q_EMIT finished();
    }
};

#endif // WORKER_H

Some additional comments:

• Note: The .pro file contains the line CONFIG += no_keywords, in order to avoid name conflicts with the Python header.
• It should be noted that while the thread halts execution at the call to PyGILState_Ensure(), the main thread continues uninhibited. If I change return a.exec(); to return 0;, the program will exit. (So perhaps deadlock is the wrong term to use.)
• Please note that I'm not interested in creating threads from within Python. I just want to straight-forwardly call a given Python script from a QThread.
• I've read other similar questions, but felt that the cases considered there are subtly different, and I've not been able to solve my problem from the answers given there. Also, I'm confused by the recommendations to call PyEval_InitThreads(), which if I understand the Python/C API documentation correctly, shouldn't be needed.

Judejudea

How do you know it is a dead lock? When application is stuck use debugger to inspect all threads call stacks. – Austronesian

If I stop the debugger at the line where PyGILState_Ensure() is called, and from there try to step one line forward, the program locks up. I.e. the thread gets stuck somewhere within PyGILState_Ensure(). – Judejudea

Answer 1

After browsing around SO, I've found a solution. Example 1 in this answer was especially helpful.

What I need to do is to call PyEval_InitThreads() from the main thread (not at all clear from the very cryptic documentation). Then, to enable PyGILState_Ensure() to acquire the GIL at all from other threads (or else get stuck in an infinite loop within the Python source code, continually trying and failing to acquire the GIL), I need to release the GIL in the main thread via a call to PyEval_SaveThread(). Finally, I should retrieve the GIL in the main thread again via a call to PyEval_RestoreThread() (making sure that all threads that want to call PyGILState_Ensure() are definitely finished before that, or else risking a lock again for the same reason as before).

Here is an updated version of main.cpp which solves the problem:

#include <QCoreApplication>
#include <QThread>
#include "Worker.h"

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    Py_Initialize();

    //Initialize threads and release GIL:
    PyEval_InitThreads();
    PyThreadState *threadState;
    threadState = PyEval_SaveThread();

    QThread* thread = new QThread;
    Worker* worker = new Worker();
    worker->moveToThread(thread);
    QObject::connect(thread, SIGNAL(started()), worker, SLOT(process()));
    QObject::connect(worker, SIGNAL(finished()), thread, SLOT(quit()));
    QObject::connect(worker, SIGNAL(finished()), worker, SLOT(deleteLater()));
    QObject::connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    thread->start();

    //wait until thread is finished calling Python code:
    thread->wait(1000); //(ugly, but in a proper Qt application, this would be handled differently..)

    //Retrieve GIL again and clean up:
    PyEval_RestoreThread(threadState);
    Py_FinalizeEx();

    return a.exec();
}

Judejudea