Tcl_DoOneEvent is blocked if tkwait / vwait is called

Question

There is an external C++ function that is called from Tcl/Tk and does some stuff in a noticeable amount of time. Tcl caller has to get the result of that function so it waits until it's finished. To avoid blocking of GUI, that C++ function has some kind of event loop implemented in its body:

while (m_curSyncProc.isRunning()) {
    const clock_t tm = clock();
    while (Tcl_DoOneEvent(TCL_ALL_EVENTS | TCL_DONT_WAIT) > 0) {}  // <- stuck here in case of tkwait/vwait
    // Pause for 10 ms to avoid 100% CPU usage
    if (double(clock() - tm) / CLOCKS_PER_SEC < 0.005) {
        nanosleep(10000);
    }
}

Everything works great unless tkwait/vwait is in action in Tcl code.

For example, for dialogs the tkwait variable someVariable is used to wait Ok/Close/<whatever> button is pressed. I see that even standard Tk bgerror uses the same method (it uses vwait).

The problem is that once called Tcl_DoOneEvent does not return while Tcl code is waiting in tkwait/vwait line, otherwise it works well. Is it possible to fix it in that event loop without total redesigning of C++ code? Because that code is rather old and complicated and its author is not accessible anymore.

Monocyte

Answer 1

Beware! This is a complex topic!

The Tcl_DoOneEvent() call is essentially what vwait, tkwait and update are thin wrappers around (passing different flags and setting up different callbacks). Nested calls to any of them create nested event loops; you don't really want those unless you're supremely careful. An event loop only terminates when it is not processing any active event callbacks, and if those event callbacks create inner event loops, the outer event loop will not get to do anything at all until the inner one has finished.

As you're taking control of the outer event loop (in a very inefficient way, but oh well) you really want the inner event loops to not run at all. There's three possible ways to deal with this; I suspect that the third (coroutines) will be most suitable for you and that the first is what you're really trying to avoid, but that's definitely your call.

1. Continuation Passing

You can rewrite the inner code into continuation-passing style — a big pile of procedures that hands off from step to step through a state machine/workflow — so that it doesn't actually call vwait (and friends). The only one of the family that tends to be vaguely safe is update idletasks (which is really just Tcl_DoOneEvent(TCL_IDLE_EVENTS | TCL_DONT_WAIT)) to process Tk internally-generated alterations.

This option was your main choice up to Tcl 8.5, and it was a lot of work.

2. Threads

You can move to a multi-threaded application. This can be easy… or very difficult; the details depend on an examination of what you're doing throughout the application.

If going this route, remember that Tcl interpreters and Tcl values are totally thread-bound; they internally use thread-specific data so that they can avoid big global locks. This means that threads in Tcl are comparatively expensive to set up, but actually use multiple CPUs very efficiently afterwards; thread pooling is a very common approach.

3. Coroutines

Starting in 8.6, you can put the inner code in a coroutine. Almost everything in 8.6 is coroutine-aware (“non-recursive” in our internal lingo) by default (including commands you wouldn't normally think of, such as source) and once you've done that, you can replace the vwait calls with equivalents from the Tcllib coroutine package and things will typically “just work”. (For example, vwait var becomes coroutine::vwait var, and after 123 becomes coroutine::after 123.)

The only things that don't have direct replacements are tkwait window and tkwait visibility; you'll need to simulate those with waiting for a <Destroy> or <Visibility> event (the latter is uncommon as it is unsupported on some platforms), which you do by binding a trivial callback on those that just sets a variable that you can coroutine::vwait on (which is essentially all that tkwait does internally anyway).

Coroutines can become messy in a few cases, such as when you've got C code that is not coroutine-aware. The main places in Tcl where these come into play are in trace callbacks, inter-interpreter calls, and the scripted implementations of channels; the issue there is that the internal APIs these sit behind are rather complicated already (especially channels) and nobody's felt up to wading in and enabling a non-recursive implementation.

Minute

Coroutines (which in Tcl are deep coroutines, unlike in some other common languages) allow you to take ordinary code and turn it internally into continuation-passing code without needing to write it out explicitly; the language runtime effectively does the heavy lifting for you. The result is very much like a lightweight cooperative thread. – Minute

Thank you for the options, So, it's impossible to fix the problem on C++ side, right? Anyway, is my assuming correct that if I choose coroutine way it will be impossible to get the result from a function that will use coroutine::vwait instead of regular vwait? Because it will return to its caller in that line. As an example, the result of tk_messageBox. – Monocyte

Doing things with threads would be how you fix things on the C++ side. You'd end up with Tcl/Tk effectively running in one thread and your C++ code running in another (except for some bits of glue). – Minute