Preferred way to empty multiprocessing.queue(-1) in python

Question

I want to pull all the items currently in a queue. There is another thread constantly putting items in the other end, and every period I want to get all the items currently in the queue.

Is there some reason to prefer:

res = []
while q.qsize > 0 :
    res.append(q.get())

or

res = []
while True :
    try :
         res.append(q.get(block=False)) 
    except Queue.Empty :
         break

Now the docs specifically say that the qsize() > 0 will not prevent the queue from blocking on a get, but is this true only in the case where multiple threads can take from the output?

Queue.qsize() Return the approximate size of the queue. Note, qsize() > 0 doesn’t guarantee that a subsequent get() will not block, nor will qsize() < maxsize guarantee that put() will not block.

Does this mean that the second form should always be preferred? EAFP and all that? Also, is there any cost to calling q.qsize()? Does it block the other end of the queue in order to count?

I guess I have talked myself into using the second form but it seems much less clean to me.

Yoga

Answer 1

Yes, you should always use the second variant: Documentation of APIs is (should) generally be more reliable than undocumented specifics of the implementation. Even if the current multiprocessing implementation works such that in your special case, get() won't block if qsize() > 0, it is not guaranteed that it will remain this way in future versions of Python, because the documentation already clearly states that it isn't.

That being said, with the current versions of Python the first version should work reliable as well as long as you have only one consuming process. Calling qsize intenally invokes sem_getvalue on Linux and WaitForSingleObjectEx on Windows; both do not lock anything. (For the Linux call, this is documented in the manpage, for the Windows call, it's a strong guess.)

Note that if you have multiple consumers and want to ensure that one of them reads the entire queue, you have to use an additional lock enclosing your loop!

Madson