Why can't cython memory views be pickled?

Question

I have a cython module that uses memoryview arrays, that is...

double[:,:] foo

I want to run this module in parallel using multiprocessing. However I get the error:

PicklingError: Can't pickle <type 'tile_class._memoryviewslice'>: attribute lookup tile_class._memoryviewslice failed

Why can't I pickle a memory view and what can I do about it.

Buehrer

Answer 1

Maybe passing the actual array instead of the memory view can solve your problem. If you want to execute a function in parallel, all of it parameters have to be picklable if i recall correctly. At least that is the case with python multiprocessing. So you could pass the array to the function and create the memoryview inside your function.

def some_function(matrix_as_array):
    cdef double[:,:] matrix = matrix_as_array
    ...

I don't know if this helps you, but I encountered a similar problem. I use a memoryview as an attribute in a cdef class. I had to write my own __reduce__ and __setstate__ methods to correctly unpickle instances of my class. Pickling the memory view as an array by using numpy.asarray and restoring it in __setstate__ worked for me. A reduced version of my code:

import numpy as np

cdef class Foo:
    cdef double[:,:] matrix

    def __init__(self, matrix):
        '''Assign a passed array to the typed memory view.'''
        self.matrix = matrix

    def __reduce__(self):
        '''Define how instances of Foo are pickled.'''
        d=dict()
        d['matrix'] = np.asarray(self.matrix)
        return (Foo, (d['matrix'],), d)

    def __setstate__(self, d):
        '''Define how instances of Foo are restored.'''
        self.matrix = d['matrix']

Note that __reduce__ returns a tuple consisting of a callable (Foo), a tuple of parameters for that callable (i.e. what is needed to create a 'new' Foo instance, in this case the saved matrix) and the dictionary with all values needed to restore the instance.

Cyclosis