How does a threading.Thread yield the rest of its quantum in Python?

Question

I've got a thread that's polling a piece of hardware.

while not hardware_is_ready():
    pass
process_data_from_hardware()

But there are other threads (and processes!) that might have things to do. If so, I don't want to burn up cpu checking the hardware every other instruction. It's been a while since I've dealt with threading, and when I did it wasn't Python, but I believe most threading libraries have a yield function or something that allows a thread to tell the scheduler "Give the other threads a chance."

while not hardware_is_ready():
    threading.yield()          # This function doesn't exist.
process_data_from_hardware()

But I can't find any reference to something like this in the threading documentation. Python does have a yield statement, but I'm pretty sure that's something else entirely (to do with generators).

What's the correct thing to do here?

Storm

Answer 1

time.sleep(0) is sufficient to yield control -- no need to use a positive epsilon. Indeed, time.sleep(0) MEANS "yield to whatever other thread may be ready".

Dewan

Is this documented somewhere? I thought this might be true -- I've seen it in other languages -- but I couldn't find a mention of it in the Python docs. – Storm

for example, docs.python.org/library/sched.html "delayfunc will also be called with the argument 0 after each event is run to allow other threads an opportunity to run in multi-threaded applications." (delayfunc is usually time.sleep here). I agree it would be nicer if it was documented more directly in module time;-) – Dewan

Windows users: setting to 0 didn't work for me, time.sleep( 0.0001 ) did. – Hackett

This is not working on Windows 7 with python 3.6.4 for me. – Stuckey

Answer 2

Read up on the Global Interpreter Lock (GIL).

For example: http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/

Also: http://www.pyzine.com/Issue001/Section_Articles/article_ThreadingGlobalInterpreter.html

Do this in your code if you must do Busy Waiting (e.g. polling a device).

time.sleep( 0.0001 )

This will yield to the thread scheduler.

Also, I collected some notes and references in http://homepage.mac.com/s_lott/iblog/architecture/C551260341/E20081031204203/index.html

Lazurite

I'm no expert, but I've read a bit on the GIL. Could you give me a hint what in particular relates to this question? I'll go with time.sleep(epsilon) for now, but it's kind of unsatisfying, isn't it? There isn't any particular interval I want to sleep for. – Storm

Its the only solution if your hardware cannot send interrupts. – Archimandrite

Without using time.sleep() how does Python perform pre-emptive yielding? On what conditions does a yield occur? – Clarkson

All three links seem broken. – Okajima

Answer 3

Why the yield in time.sleep(0) might not be enough:

I had a similar problem and ultimately time.sleep(0) did not work in all cases whereas time.sleep(0.0001) worked. In my case I use the standard vanilla cpython - which can be found at python.org. Looking at the implementation of the sleep function in the C code there is a tiny difference between a call of the sleep(0) and sleep(something_other_than_null).

Looking at pysleep in timemodule.c you will see that in the first case

if (ul_millis == 0 || !_PyOS_IsMainThread()) {
  Py_BEGIN_ALLOW_THREADS
  Sleep(ul_millis);
  Py_END_ALLOW_THREADS
  break;
}

is called. The "allow threads" releases the GIL, Sleep(0) in that case is equivalent to std::this_thread::yield(), after that the GIL is claimed again. In the code that deals with a non-zero value, apart from the actual sleep/wait there is an additional call to PyErr_CheckSignals - which as stated in its documentation "checks whether a signal has been sent to the processes and if so, invokes the corresponding signal handler". That would explain why in some cases when working with hardware a time.sleep(0) is not enough. So this might be a slip on cpython implementation side. And that is why time.sleep(0.0001) magically works.

Tetrapody

Answer 4

If you're doing this on *nix, you might find the select library useful. Kamaela also has a few components you may find useful, but it may require a bit of a paradigm change.

Meghan