Python .NET, multithreading and the windows event loop

Question

I'm building a Python API around a black box .NET DLL using Python .NET. The DLL is only doing networking operations. The DLL require me to run a windows message pumping loop, otherwise the network operations get stuck after a while. I run the windows message loop using System.Windows.Forms.Application.Run() in the main thread. This works fine for only receiving data. My program starts to behave weirdly though when I start to do calls from an other Python thread to the DLL. I think it is related to threading since the problems are accuring very irregular - networking events disappear or come in very late. As far as I know Python and C# are thread safe, but maybe because of the multiple layers of wrapping something goes wrong.

So I have a couple of questions:

• Is it a bad idea to do calls to the DLL's from multiple Python theads? Or does this depend on the DLL internals? I thought from the DLL's perspective the Python threads are seen as one single agents due to the GIL.

• Does every Python thread using this DLL need a message pump?

• It is probably a good idea to keep all DLL interactions in one thread. I have difficulties accomplishing this, since I give control to the message pump loop in the main thread. My naive approach would be to put new outgoing network messages generated in my worker thread on a Python queue, create a custom message pump loop in the main thread that does the windows event handling but also monitors my queue and if there is message would do a call to the DLL. But this all feels quite clunky and a lot of work for such a simple task. Is this the right approach?

• Is there an other way to put a function in the main windows event loop that monitors the previously described queue and takes action? Or should I dive into the .NET specifics and start using .NET events or dispatchers?

Aguila

Answer 1

I have found answers to most of my own questions.

• I think we can't assume the DLL is thread safe, so probably best to isolate all interaction with the DLL to one thread.

• It looks like the Windows message system is per thread and not per process, so yes, every thread using the Windows message system is required to have a Windows message processing loop.

• One can insert execution in the a windows event loop using Form.Invoke. running a non-ui main loop, you can get a dispatcher using Dispatcher.CurrentDispatcher which can used for 'invoking' a function. The invoked function is then executed on the thread where the dispatcher is created. The called function needs to be delegated though, which is a C# specific thing to pass references to functions.

in the end I did something like this for a non-ui main loop:

import clr
import threading

# we need to get access to the threading assembly
clr.AddReference("c:\\Program Files\\Reference Assemblies\\Microsoft\\Framework\\v3.0\\WindowsBase.dll")

import System
from System.Windows.Threading import Dispatcher
from System.Windows.Forms import Application

# now get the dispatcher for the current thread
dispatcher = Dispatcher.CurrentDispatcher

def display():
    print(threading.current_thread())

# now make this a deligate
deligate = System.Action(display)

def other_thread(dispatcher):
    # now you can use the dispatcher from the main thread to
    # to schedule a run from an other thread
    dispatcher.Invoke(deligate)

thread = threading.Thread(target=other_thread, args=(dispatcher,))
thread.start()
Application.Run()

some related links:

• more info about the Dispatcher
• more about the Action delegate

Aguila

I fixed the problems I had when using python threads in winforms applications with pythonnet thanks to this answer. Many thanks! – Bratcher

Answer 2

I have had a similar problem: using a pythonnet library DLL (import clr, et.) from multiple Python threads at the same time.

I had errors when multiple threads were communicating with the same functions (managing a hardware device) in parallel.

Solution:

1. create a global lock:

   lock = threading.Lock()
   

2. replace each API call e.g. device.move(x=1234) by:

   with lock:
       device.move(x=1234)
   

Brown