Allowing Ctrl-C to interrupt a python C-extension

Question

I'm running some computationally heavy simulation in (home-made) C-based python extensions. Occasionally I get stuff wrong and would like to terminate a simulation. However, Ctrl-C doesn't seem to have any effect (other than printing ^C to the screen) so I have to kill the process using kill or the system monitor.

As far as I can see python just waits for the C extension to finish and doesn't really communicate with it during this time.

Is there a way to make this work?

Update: The main answers (for my specific problem) turned out to be: 1. rewrite the code to regularly pass control back to the caller (answer Allowing Ctrl-C to interrupt a python C-extension below), or 2. Use PyErr_CheckSignals() (answer https://mcmap.net/q/530598/-allowing-ctrl-c-to-interrupt-a-python-c-extension below)

Watkin

See #1112843 – Entrenchment

related: CTRL+C doesn't interrupt call to shared-library using CTYPES in Python – Joselynjoseph

Read signal-safety(7) it is extremely relevant – Administrator

Answer 1

I would redesign the C extensions so that they don't run for a long period.

So, split them into more elementary steps (each running for a short period of time, e.g. 10 to 50 milliseconds), and have these more elementary steps called by Python code.

continuation passing style might be relevant to understand, as a programming style...

Administrator

Sorry, not an option at all in this case :) It's a simulation with a huge number of steps & speed is essential. Interfacing with python at each step (or even at regular intervals) would ruin the efficiency. – Watkin

Try to bunch the steps into something lasting a few milliseconds. Then the overhead of going to Python is negligible... – Administrator

It's certainly worth thinking about, but it does raise a lot of issues with memory management etc. Thanks! – Watkin

Haven't implemented it yet, but this is definitely on my todo list. Thanks again! – Watkin

Answer 2

However, Ctrl-C doesn't seem to have any effect

Ctrl-C in the shell sends SIGINT to the foreground process group. python on receiving the signal sets a flag in C code. If your C extension runs in the main thread then no Python signal handler will be run (and therefore you won't see KeyboardInterrupt exception on Ctrl-C) unless you call PyErr_CheckSignals() that checks the flag (it means: it shouldn't slow you down) and runs Python signal handlers if necessary or if your simulation allows Python code to execute (e.g., if the simulation uses Python callbacks). Here's a code example of an extension module for CPython created using pybind11 suggested by @Matt:

PYBIND11_MODULE(example, m)
{
    m.def("long running_func", []()
    {
        for (;;) {
            if (PyErr_CheckSignals() != 0)
                throw py::error_already_set();
            // Long running iteration
        }
    });
}

If the extension runs in a background thread then it is enough to release GIL (to allow Python code to run in the main thread that enables the signal handlers to run). PyErr_CheckSignals() always returns 0 in a background thread.

Related: Cython, Python and KeybordInterrupt ingored

Joselynjoseph

Is there a pure-python way to detect SIGINT in a callback, main.py -> long-running scipy optimizer which calls a shared lib -> callback.py ? (Shall I ask a new question ?) – Eyeleteer

@denis: " pure-python way to detect SIGINT" -> KeyboardInterrupt is raised in the main thread (with the default signal handler). – Joselynjoseph

This should be the accepted answer. There are legit reasons you may have to wait on a condition variable or something if you are doing blocking I/O that are unavoidable. You cannot always rearchitect around it. – Ifni

I found this helpful: pybind11.readthedocs.io/en/stable/… – Bertabertasi

Answer 3

Python has a signal handler installed on SIGINT which simply sets a flag that is checked by the main interpreter loop. For this handler to work properly, the Python interpreter has to be running Python code.

You have a couple of options available to you:

1. Use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release the GIL around your C extension code. You cannot use any Python functions when not holding the GIL, but Python code (and other C code) may run concurrently with your C thread (true multithreading). A separate Python thread can execute alongside the C extension and catch Ctrl+C signals.
2. Set up your own SIGINT handler and call the original (Python) signal handler. Your SIGINT handler can then do whatever it needs to do to cancel the C extension code and return control to the Python interpreter.

Limpid

(1) releasing GIL would make any difference only if the C extension runs in a background thread. (2) call Py_Err_CheckSignals() instead of manuall calling Python signal handler – Joselynjoseph

Answer 4

There is an alternative way to solve this problem if you do not want your C Extension (or ctypes DLL) to be tied to Python, such as a case where you want to create a C library with bindings in multiple languages, you must allow your C Extension to run for long periods, and you can modify the C Extension:

Include the signal header in the C Extension.

#include <signal.h>

Create a signal handler typedef in the C Extension.

typedef void (*sighandler_t)(int);

Add signal handlers in the C extension that will perform the actions necessary to interrupt any long running code (set a stop flag, etc.), and save the existing Python signal handlers.

sighandler_t old_sig_int_handler = signal(SIGINT, your_sig_handler);
sighandler_t old_sig_term_handler = signal(SIGTERM, your_sig_handler);

Restore the existing signal handlers whenever the C extension returns. This step ensures that the Python signal handlers are re-applied.

signal(SIGINT, old_sig_int_handler);
signal(SIGTERM, old_sig_term_handler);

If the long-running code is interrupted (flag, etc.), return control to Python with a return code indicating the signal number.

return SIGINT;

In Python, send the signal received in the C extension.

import os
import signal

status = c_extension.run()

if status in [signal.SIGINT, signal.SIGTERM]:
    os.kill(os.getpid(), status)

Python will perform the action you are expecting, such as raising a KeyboardInterrupt for SIGINT.

Throughout

Read signal-safety(7) – Administrator

Answer 5

Not elegant, but the only approach I found that also interrupts external library calls in C++ and kills any running child processes.

#include <csignal>
#include <pybind11/pybind11.h>

void catch_signals() {
  auto handler = [](int code) { throw std::runtime_error("SIGNAL " + std::to_string(code)); };
  signal(SIGINT, handler);
  signal(SIGTERM, handler);
  signal(SIGKILL, handler);
}

PYBIND11_MODULE(example, m)
{
    m.def("some_func", []()
    {
        catch_signals();
        // ...
    });
}

import sys

from example import some_func

try:
    some_func()
except RuntimeError as e:
    if "SIGNAL" in str(e):
        code = int(str(e).rsplit(" ", 1)[1])
        sys.exit(128 + code)
    raise

Homogeneous