Execute Python function in Main thread from call in Dummy thread

Question

I have a Python script that handles aynchronous callbacks from .NET Remoting. These callbacks execute in a dummy (worker) thread. From inside my callback handler, I need to call a function I've defined in my script, but I need the function to execute in the main thread.

The Main thread is a remote client that sends commands to a server. Some of these commands result in asynchronous callbacks.

Basically, I need the equivalent of .NET's Invoke method. Is this possible?

Nozicka

what does the main thread do? you need to signal it somehow and get it to call the function. if the main thread generally waits for things to happen, then signal it. if the main thread does processing, then put the function to be called on a queue and have the main thread check and process the queue occasionally. – Jarrett

Hmmm...thinking more....if the callback occurs while Main is waiting for the server to return control from a call, there is no way to get Main to do anything else, is there? – Nozicka

Hmm no, you'd have to make Main wait on one of two things: either server to return control, or a scheduled function call. Have you considered using Twisted? It's really, really good for dealing with asynchronous stuff. It provides a main loop and, for example, you can use reactor.callFromThread to do exactly what you want - run a function in the main (reactor) thread. – Jarrett

Twisted definitely looks cool. I'm not sure I can make it work with .NET Remoting, though, since Twisted seems to want to manage endpoints and that is what .NET Remoting does for me. After establishing 2-way comm, I simply subscribe to an event on the server and get my callbacks. Anyway, putting aside the deadlock case, how would I set up a queue of function names that gets written to by a Dummy thread and consumed by the Main thread? – Nozicka

I'll put that in an answer – Jarrett

Answer 1

You want to use the Queue (now queue from python 3) class to set up a queue that your dummy threads populate with functions and that your main thread consumes.

import Queue

#somewhere accessible to both:
callback_queue = Queue.Queue()

def from_dummy_thread(func_to_call_from_main_thread):
    callback_queue.put(func_to_call_from_main_thread)

def from_main_thread_blocking():
    callback = callback_queue.get() #blocks until an item is available
    callback()

def from_main_thread_nonblocking():
    while True:
        try:
            callback = callback_queue.get(False) #doesn't block
        except Queue.Empty: #raised when queue is empty
            break
        callback()

Demo:

import threading
import time

def print_num(dummyid, n):
    print "From %s: %d" % (dummyid, n)
def dummy_run(dummyid):
    for i in xrange(5):
        from_dummy_thread(lambda: print_num(dummyid, i))
        time.sleep(0.5)
    
threading.Thread(target=dummy_run, args=("a",)).start()
threading.Thread(target=dummy_run, args=("b",)).start()

while True:
    from_main_thread_blocking()

Prints:

From a: 0
From b: 0
From a: 1
From b: 1
From b: 2
From a: 2
From b: 3
From a: 3
From b: 4
From a: 4

and then blocks forever

Jarrett

Very cool. Python makes delegate passing and storage trivial. I appreciate your very detailed answer. – Nozicka

@JimC: that's part of why it's my favorite language =). cheers – Jarrett