Update variable while working with ProcessPoolExecutor

Question

if __name__ == '__main__':

    MATCH_ID = str(doc_ref2.id)

    MATCH_ID_TEAM = doc_ref3.id

    with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
        results = list(executor.map(ESPNlayerFree, teamList1))
    
    MATCH_ID_TEAM = str(doc_ref4.id)

    with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
        results = list(executor.map(ESPNlayerFree, teamList2))

When I print the MATCH_ID_TEAM it prints the value. But in the process, it shows up the default value which I set empty at the top.

How do I update the value of my variables to all the processes?

ESPNPlayerFree is a class that takes `id` as an argument. So `teamList1` and `teamList2` are list of ids to initialize my objects.

MATCH_ID and MATCH_ID_TEAM are variables that are used in my Class ESPNPlayerFree

OS Windows 10 64bit

IDE Pycharm

Python Version 3.6.1

Splendent

what variables? what is teamList1? what is ESPNPlayerFree? can you add a more complete code regarding your process? – Schmidt

@ranifisch updated explaining everything. I can't add more code as there isn't any more of it, just the class ESPNPlayerFree which is huge – Splendent

so you want to pass MATCH_ID and MATCH_ID_TEAM to the process and get the updated value in the processes? I mean update them from your "main" and have the updated values on the processes? – Schmidt

Yes exactly that's what I want this to work – Splendent

convert list teamLIst1 to list [(MATCH_ID_TEAM, item1), (MATCH_ID_TEAM, item2), etc.] and use this list with processes. And every process has to unpack argument to variables - match_id_team, item = arg – Melessa

@Melessa can you pls provide a working code with my case? – Splendent

@Splendent can you please share part of your ESPNPlayerFree class/func? – Schmidt

@ranifisch sorry I can't – Splendent

@Melessa can you have a look at this question, I almost solved it #59095585 – Splendent

@Splendent the default value which I set empty at the top. Please show the code where you (i) set the default value, and (ii) use MATCH_ID_TEAM in ESPNPlayerFree. – Husain

Answer 1

I'm picking up where @furas left in his comment some days ago. The simplest approach is indeed to pass just everything you need in your class along with .map(). executor.map() is expecting iterables, which get zipped to an argument-tuple for each function call to be made in your workers.

You obviously need both MATCH_ID and MATCH_ID_TEAM to remain the same for a whole job, that is one call to executor.map(). Your challenge is that both are iterables (strings), but you need them replicated as a whole and often enough to match with every item of your teamlist-iterable.

So what you do is simply wrap these strings with itertools.repeat() when you pass them to .map() together with the team-id list. itertools.repeat() by default returns an infinite iterator of the passed object. ProcessPoolExecutor internally then uses zip() to combine items from all iterables as arguments.

import concurrent.futures
import multiprocessing
from itertools import repeat


class ESPNPlayerFree:
    def __init__(self, team_id, match_id, match_id_team):
        self.teams_id = team_id
        self.match_id = match_id
        self.match_id_team = match_id_team
        print(
            multiprocessing.current_process().name,
            self.teams_id, self.match_id, self.match_id_team
        )


if __name__ == '__main__':

    teams1 = [f"id{i}" for i in range (10)]
    teams2 = [f"id{i}" for i in range(10, 20)]

    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:

        MATCH_ID = 'doc_ref2.id'
        MATCH_ID_TEAM = 'doc_ref3.id'

        results = list(
            executor.map(
                ESPNPlayerFree,
                teams1,
                repeat(MATCH_ID),
                repeat(MATCH_ID_TEAM),
            )
        )

        print("--- new MATCH_ID_TEAM ---")
        MATCH_ID_TEAM = 'doc_ref4.id'

        results = list(
            executor.map(
                ESPNPlayerFree,
                teams2,
                repeat(MATCH_ID),
                repeat(MATCH_ID_TEAM),
            )
        )

Output:

ForkProcess-1 id0 doc_ref2.id doc_ref3.id
ForkProcess-2 id1 doc_ref2.id doc_ref3.id
ForkProcess-3 id2 doc_ref2.id doc_ref3.id
ForkProcess-4 id3 doc_ref2.id doc_ref3.id
ForkProcess-1 id4 doc_ref2.id doc_ref3.id
ForkProcess-3 id5 doc_ref2.id doc_ref3.id
ForkProcess-2 id6 doc_ref2.id doc_ref3.id
ForkProcess-4 id7 doc_ref2.id doc_ref3.id
ForkProcess-3 id8 doc_ref2.id doc_ref3.id
ForkProcess-1 id9 doc_ref2.id doc_ref3.id
--- new MATCH_ID_TEAM ---
ForkProcess-1 id10 doc_ref2.id doc_ref4.id
ForkProcess-3 id11 doc_ref2.id doc_ref4.id
ForkProcess-2 id12 doc_ref2.id doc_ref4.id
ForkProcess-4 id13 doc_ref2.id doc_ref4.id
ForkProcess-1 id14 doc_ref2.id doc_ref4.id
ForkProcess-3 id15 doc_ref2.id doc_ref4.id
ForkProcess-2 id16 doc_ref2.id doc_ref4.id
ForkProcess-4 id17 doc_ref2.id doc_ref4.id
ForkProcess-2 id18 doc_ref2.id doc_ref4.id
ForkProcess-1 id19 doc_ref2.id doc_ref4.id

Process finished with exit code 0

For the second job, with new MATCH_ID_TEAM you then don't have to recreate the ProcessPoolExecutor again, you just use the existing again by staying within the context-manager as long as you need it.

Fagaceous

Followed exactly but I get this error BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. To make things clear, I also get this error following @Melessa solution. Data is stored for the first part and then it throws the error, never gets to the 2nd part – Splendent

@Splendent Your OS is killing a worker, in that case the whole Pool shuts down. Might be that you're using too much memory, have a try with less workers and keep an eye on memory consumption. – Fagaceous

Tried with max_workers=4, same error. I also tried with a manual list with only 2 items, same error. It completes the processing and then throws the error – Splendent

@Splendent Do you get the error already with my unchanged example code? – Fagaceous

@Splendent That shouldn't be. Can you please specify your OS, how you run your code (which IDE or terminal), which exact Python version you use and if you run the code on a server or local machine. – Fagaceous

Windows 10 64bit, Pycharm IDE, I'm running it on PC, Python version 3.6.1 – Splendent

@Splendent This setup doesn't sound suspicious to me. Please have a try with starting from terminal. Upgrading your version of Python would also be an idea. – Fagaceous