Pathos Pool not running

Question

I was getting PicklingError: Can't pickle <type 'obj'> while using multiprocessing so I switched from multiprocessing to pathos.

My code looks like this:

from pathos.multiprocessing import ProcessingPool as Pool
pool = Pool(processes=9)
tuples = [(session, query, parse_query, filter_values, i) for i in range(32)]

responses = pool.uimap(execute_queries, [session]*32, [query]*32,[parse_query]*32, [filter_values]*32, [i for i in range(32)]  )
pool.close() 
pool.join()
                
response = reduce(reduce_dic, responses)

Upon running the code I get the following error: Pool not running On one post I read that I need to clear the pool, but I am not sure how would that work for me.

Warga

Answer 1

I'm the pathos author. I assume that you are running on a closed pool. In that case, all you need to do is to restart the pool.

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> p = Pool()
>>> responses = p.uimap(lambda x:x*x, range(32))
>>> p.close()
>>> p.join()
>>> x = list(responses)
>>> x
[0, 1, 16, 4, 9, 36, 25, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]
>>> 
>>> responses = p.uimap(lambda x:x*x, range(32))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.9.dev0-py3.7.egg/pathos/multiprocessing.py", line 149, in uimap
    return _pool.imap_unordered(star(f), zip(*args)) # chunksize
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess-0.70.13.dev0-py3.7.egg/multiprocess/pool.py", line 332, in imap_unordered
    raise ValueError("Pool not running")
ValueError: Pool not running
>>>
>>>
>>> p.restart()
<multiprocess.pool.Pool object at 0x105031c50>
>>> responses = p.uimap(lambda x:x*x, range(32))
>>> list(responses)
[0, 1, 4, 16, 9, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]
>>> 

Drusie

Thank you for the reply. Is there a way I have different pools in different threads. I am using this in a threaded server and if I send a second request while the first one is still running I get the above mentioned error. – Warga

Yes, it's possible to have a ProcessPool launch a ThreadPool, or vice versa. Pathos pools are, by default, singletons... so don't close/join the pool if you intend to reuse it (unless you restart it). – Drusie