Both multiprocessing.map and joblib use only 1 cpu after upgrade from Ubuntu 10.10 to 12.04

Question

I had some perfectly working python code which used multiprocessing module and loaded all 8 CPUs on my machine at 100%.

After I upgraded from Ubuntu 10.10 to 12.04 (the most evident thing, maybe I did something else that broke everything), it stopped working. After lots of debugging, I found that even in the simplest use case, both modules are only using 1 CPU:

from pylab import *
import multiprocessing as mp
from joblib import Parallel, delayed

def f(i):
    # Slow calculation
    x = 1
    for j in range(100000): x = cos(x)
    print i, x

if __name__ == '__main__':
    # Try to multiprocess with multiprocessing
    mp.Pool(processes=8).map(f, range(100))
    # Try to multiprocess with joblib
    Parallel(n_jobs=8)(delayed(f)(i) for i in range(100))

I need to use all 8 CPUs in my system. Any ideas of what I should look at to fix the issue?

Starbuck

could it be due to your Python version changing? I know that the default Python version changes from 2.6 -> 2.7 when going from Ubuntu 10 -> 12 – Downtrend

You might be able to fix this problem using taskset - see my answer here – Granger

Which value does multiprocessing.cpu_count() return? If it is not 8, there is a problem. – Oid

FWIW, on my 12.10 running python 2.7.3, the mp.Pool().map() call consumes 100% of all 4 cpus in my system. – Jaunty

I can't reproduce this on 13.04. – Vulpine

If you have solved the problem then you really ought to write it as an answer and mark it as solved. This helps de-clutter the place and also helps the search engines. – Gregarine

I tried the same procedure for a Jupyter notebook, and it does not work. In placed that os command as my first cell before I import any libraries. Any ideas on how to get this to work on Jupyter notebooks? – Norenenorfleet

@Starbuck Please post your answer as an answer – Dumfries

Answer 1

Solution moved from @user1084871's question post.

As @ali_m pointed out in a comment here and in the answer to Why does multiprocessing use only a single core after I import numpy? the problem is related to numpy messing up with cpu affinity. Calling

os.system('taskset -p 0xffffffff %d' % os.getpid())

before I do any multiprocessing solved the problem for me.

Vindicate