TypeError: AutoProxy object is not iterable - multiprocessing

Question

consider the following server code :

from multiprocessing.managers import BaseManager, BaseProxy

def baz(aa) :
    print "aaa"
    l = []
    for i in range(3) :
      l.append(aa)
    return l

class SolverManager(BaseManager): pass

manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', baz)

server = manager.get_server()
server.serve_forever()

and the associated client :

import sys
from multiprocessing.managers import BaseManager, BaseProxy

class SolverManager(BaseManager): pass

def main(args) :
    SolverManager.register('solver')
    m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
    m.connect()

    for i in m.solver(args[1]):
        print i

if __name__ == '__main__':
    sys.exit(main(sys.argv))

I think I'm missing something important here. My guess is that I have to subclass the BaseProxy class to provide an iterable object, but so far I haven't managed to get it right.

when I run the client I get this error :

Traceback (most recent call last):
  File "mpmproxy.py", line 17, in <module>
    sys.exit(main(sys.argv))
  File "mpmproxy.py", line 13, in main
    for i in m.solver(args[1]):
TypeError: 'AutoProxy[solver]' object is not iterable

however if I try to print it, the list is there ... Maybe it has also something to do with the way data is serialized between client and server ...

in the documentation there is a similar case (with a generator) and they use the following class to access the data :

class GeneratorProxy(BaseProxy):
    _exposed_ = ('next', '__next__')
    def __iter__(self):
        return self
    def next(self):
        return self._callmethod('next')
    def __next__(self):
        return self._callmethod('__next__')

shall I do something similar ? Can anybody give me an example and explain to me how this works ?

update

To clarify : suppose I add the class :

class IteratorProxy(BaseProxy):
    def __iter__(self):
        print self
        return self

and in the client I register the function as

SolverManager.register('solver', proxytype=IteratorProxy)

the error I get is :

$python mpmproxy.py test
['test', 'test', 'test']
Traceback (most recent call last):
  File "mpmproxy.py", line 22, in <module>
    sys.exit(main(sys.argv))
  File "mpmproxy.py", line 18, in main
    for i in m.solver(args[1]):
TypeError: iter() returned non-iterator of type 'IteratorProxy'

I have the impression I'm missing something stupid here ...

update 2

I think I solved this problem:

The point was to get the real value :

for i in m.solver(args[1])._getvalue():
    print i

gosh !!! I'm not sure if this is the correct answer or just a workaround ...

Vida

Kudos for figuring it out! I have no idea how you figured this out. You just saved me a whole heap of time, thanks! This very curious behaviour of returning the BS proxy object seems to be specific to lists. dicts are fine, and even dicts containing lists are fine somehow. – Trophic

In fact, just the update 2 in this question solved my problem, which was quite similar to this one. It wasn't necessary to use any proxy (e.g., BaseProxy, GeneratorProxy, IteratorProxy). I've just added ._getvalue() to retrieve the object in the client. – Medusa

you can also use .copy() – Jillane

Answer 1

Indeed, to be iterable your class needs to define the __iter__ method that BaseProxy defines, so I guess inheritance is the right way to go !

Substance

Answer 2

While the solution in OPs 2nd update may work, the method _getvalue() is private and therefore shouldn't really be accessed in this way. Why don't you try using the list proxy type instead (from multiprocessing.managers import ListProxy)? While the result returned from the server still won't be iterable, as far as I've seen, you should be able to run the well known .pop() or .pop(i) (where i is an index) on it, which is a publicly available method for the result.

Bowleg

Answer 3

This works, and eliminates the need to use ._getvalue():

from multiprocessing.managers import BaseManager, MakeProxyType, ListProxy, DictProxy

class MyManager(BaseManager):
    pass

BaseSetProxy = MakeProxyType('BaseSetProxy', ([
'__and__', '__contains__', '__iand__', '__ior__', '__iter__', 
'__isub__', '__ixor__', '__len__', '__or__', '__rand__', '__ror__', '__rsub__',
'__rxor__', '__sub__', '__xor__', 'add', 'clear', 'copy', 'difference',
'difference_update', 'discard', 'intersection', 'intersection_update', 'isdisjoint',
'issubset', 'issuperset', 'pop', 'remove', 'symmetric_difference', 
'symmetric_difference_update', 'union', 'update']
))

class SetProxy(BaseSetProxy):
    pass

MyManager.register('set', set, SetProxy)
MyManager.register('list', list, ListProxy)
MyManager.register('dict', dict, DictProxy)

Hurleigh