Value update in manager.dict not reflected

Question

I expect the following code to print [{0: 100}], since it do the plus in updateList for a hundred time In turn, it prints [{0: 0}] what's the problem and how to rectify it?

from multiprocessing import Process, Lock, Value,Manager
class myWorker:
    def __init__(self, lock, driver, i):
        self.idx=i
        self.driver=driver
        self.lock=lock

    def run(self):
        self.driver.updateList(self.lock,self.idx)

class driver(object):

    def __init__(self):
        manager=Manager()
        self.lock=Lock()
        self.lst=manager.list()
        self.dict1=manager.dict({0:0})
        self.lst.append(self.dict1)

    def workerrun(self,lock, i):
        worker1=myWorker(lock,self,i)
        worker1.run()

    def run(self):
        D=[Process(target=self.workerrun,args=(self.lock,i)) for i in range(10)]
        for d in D:
            d.start()
        for d in D:
            d.join()

    def updateList(self,l,i):
        with self.lock: # acquire lock
            for j in range(10):
                self.lst[0][0]+=1

            print ("update from", i)

if __name__=='__main__':
    dr=driver()
    dr.run()
    print(dr.lst)

Showpiece

Of note, this is no longer a problem in python 3.6+. If you store managed lists/dictionaries inside other managed lists/dictionaries, then any changes to the nested items will automatically be reflected in the outer list/dictionary. This means, this code above will execute as expected on python 3.6+. Check this issue github.com/python/cpython/issues/51015 – Claudeclaudel

Answer 1

 def updateList(self,l,i):
        with self.lock: # acquire lock
            for j in range(10):
                d = self.lst[0]
                d[0] += 1
                self.lst[0]=d
                print ("update from", i,self.lst[0])

from the docs

Note Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy:

Seeing as you already have dict1, you can update directly:

   def updateList(self,l,i):
        with self.lock: # acquire lock
            for j in range(10):
                self.dict1[0]+=1
                print ("update from", i,self.lst[0])

Montiel

Answer 2

multiprocessing.Manager() returns a multiprocessing.managers.SyncManager if you see the when you created the list and the dictionary, you actually got their proxies. That means, what you appended to the list not the {0:0} that you created inside the manager but a proxy(copy).

self.dict1=manager.dict({0:0})
#this:
self.lst.append(self.dict1)
#is the same as doing this:
self.lst.append({0:0})   

So, in the updatelist method:

def updateList(self,l,i):
    with self.lock: # acquire lock
        for j in range(10):
            self.lst[0][0]+=1

            # is the same as doing:
            # example: self.lst == [{0:x}]
            proxy = self.lst
            # you get a copy of actual list
            # proxy == [{0:x}]
            dict = proxy[0]
            dict[0]+=1
            # proxy == [{0:x+1}]
            # BUT
            # self.lst == [{0:x}]

That means you are making copies, changing them, and then not using them. You need to change to assign the list with the new value so that it changes for all processes:

def updateList(self,l,i):
    with self.lock: # acquire lock
        dict0 = self.lst[0]
        for j in range(10):
            dict0[0]+=1
        self.lst[0] = dict0

Telescopium

Answer 3

You can rectify it by updating the dictionary directly.

def updateList(self,l,i):
    with self.lock: # acquire lock
        for j in range(10):
            self.dict1[0] += 1
        print ("update from", i)

Although, why loop to add 10? Get rid of the loop and just do self.dict1[0] += 10.

Boysenberry

Answer 4

make sure the shared object itself isnt being altered by any of your processes, and by altered i mean object overwriting/assignment and not item updation/insertion.

in my case, the changes werent being reflected because, my parent process was re-assigning my manager dict with a json response which thereby changed the datatype to a normal dict instead of the manager dict. performing a key-by-key item insertion instead of blind assignment was the fix.

in order to debug this, i first off verified that the same manager dict was being shared throughout processes, u can grab an object's id by simply printing id(your_object) and match it with all your processes. secondly, ensuring the datatype is <class 'mulitprocessing.managers.DictProxy'> and not <class 'dict'>

Zeba