How to use multiprocess in python on a class object

Question

I am fairly new to Python, and my experience is specific to its use in Powerflow modelling through the API provided in Siemens PSS/e. I have a script that I have been using for several years that runs some simulation on a large data set.

In order to get to finish quickly, I usually split the inputs up into multiple parts, then run multiple instances of the script in IDLE. Ive recently added a GUI for the inputs, and have refined the code to be more object oriented, creating a class that the GUI passes the inputs to but then works as the original script did.

My question is how do I go about splitting the process from within the program itself rather than making multiple copies? I have read a bit about the mutliprocess module but I am not sure how to apply it to my situation. Essentially I want to be able to instantiate N number of the same object, each running in parallel.

The class I have now (called Bot) is passed a set of arguments and creates a csv output while it runs until it finishes. I have a separate block of code that puts the pieces together at the end but for now I just need to understand the best approach to kicking multiple Bot objects off once I hit run in my GUI. Ther are inputs in the GUI to specify the number of N segments to be used.

I apologize ahead of time if my question is rather vague. Thanks for any information at all as Im sort of stuck and dont know where to look for better answers.

Reinhold

Answer 1

Create a list of configurations:

configurations = [...]

Create a function which takes the relevant configuration, and makes use of your Bot:

def function(configuration):
    bot = Bot(configuration)
    bot.create_csv()

Create a Pool of workers with however many CPUs you want to use:

from multiprocessing import Pool
pool = Pool(3)

Call the function multiple times which each configuration in your list of configurations.

pool.map(function, configurations)

For example:

from multiprocessing import Pool
import os

class Bot:
    def __init__(self, inputs):
        self.inputs = inputs

    def create_csv(self):
        pid = os.getpid()
        print('TODO: create csv in process {} using {}'
              .format(pid, self.inputs))


def use_bot(inputs):
     bot = Bot(inputs)
     bot.create_csv()


def main():
    configurations = [
        ['input1_1.txt', 'input1_2.txt'],
        ['input2_1.txt', 'input2_2.txt'],
        ['input3_1.txt', 'input3_2.txt']]

    pool = Pool(2)
    pool.map(use_bot, configurations)

if __name__ == '__main__':
    main()

Output:

TODO: create csv in process 10964 using ['input2_1.txt', 'input2_2.txt']
TODO: create csv in process 8616 using ['input1_1.txt', 'input1_2.txt']
TODO: create csv in process 8616 using ['input3_1.txt', 'input3_2.txt']

Antihelix

Thanks for the reply. I tried to implement this method but it seems to get hung up at the pool.map(use_bot, configurations) part. It doesnt send an error, it just sits. It does create the n number of interpreter windows, but fails to instantiate the Bot object. Also, when I go to close one of the newly spawned interpreters, it creates another one. It basically creates double the number I specify, but only shows the specified number at a time. – Reinhold

@Reinhold the if __name__ == '__main__' is really important as it stops main being called by the sub-processes. If your script is all at the top level each sub-process will create a Pool and call map on it. See the multiprocessing programming guidelines for Windows – Antihelix

You're going to need to post some simplified code. It's not easy to debug in comments. See how to create a minimal reproducible example. – Antihelix

Answer 2

If you'd like to make life a little less complicated, you can use multiprocess instead of multiprocessing, as there is better support for classes and also for working in the interpreter. You can see below, we can now work directly with a method on a class instance, which is not possible with multiprocessing.

>>> from multiprocess import Pool
>>> import os
>>> 
>>> class Bot(object):
...   def __init__(self, x): 
...     self.x = x
...   def doit(self, y):
...     pid = os.getpid()
...     return (pid, self.x + y)
... 
>>> p = Pool()
>>> b = Bot(5)
>>> results = p.imap(b.doit, range(4))
>>> print dict(results)
{46552: 7, 46553: 8, 46550: 5, 46551: 6}
>>> p.close()
>>> p.join()

Above, I'm using imap, to get an iterator on the results, which I'll just dump into a dict. Note that you should close your pools after you are done, to clean up. On Windows, you may also want to look at freeze_support, for cases where the code otherwise fails to run.

>>> import multiprocess as mp
>>> mp.freeze_support 

Antihero