Is there any reason to use Ipyparallel for common python script (not ipython notebook) over multiprocessing module?

Question

Is there any reason to use Ipyparallel for common python script (not ipython notebook)?

Cullet

Answer 1

There are a few reasons why you might choose IPython parallel, which may or may not be relevant to you:

1. There are some things IPython parallel can serialize efficiently (numpy arrays) that multiprocessing doesn't do as well because it pickles everything
2. IPython parallel can distribute work across many machines, which multiprocessing cannot.
3. IPython parallel manages persistent interactive namespaces on each engine (a full IPython session), which can be useful for composing work in pieces and debugging.

In general, if you are just trying to parallelize small bits of code on your multi-core computer, IPython parallel doesn't offer you much over multiprocessing, and the burden of starting and connecting to an IPython cluster isn't worth it. But if you might want to distribute it across more machines, IPython parallel will let you do that. And since it works the same way whether you are using one computer or one hundred, you can prototype on your laptop and then run the exact same code on a larger scale without any changes.

Saunders