multiprocessing and ctypes with pointers

Question

I have multiProcessing.Process objects whose target functions take input and output queue.

To the output queue they put some data, that is a wrapped ctypes structure with internal pointers. Of course, the pickle module, that should serialize the data, breaks:

ValueError: ctypes objects containing pointers cannot be pickled

Can I somehow get my ctypes structures with pointers out of my child processes without dumping them to files?

The code is below

# -*- coding: utf-8 -*-
import multiprocessing as mp

from liblinear import *
from liblinearutil import *


def processTarget(inQueue, outQueue):
    while(not inQueue.empty()):
        inVal = inQueue.get()

        #training model
        y, x = [1,-1], [{1:inVal, 3:3*inVal}, {1:-1,3:-1}]
        prob  = problem(y, x)
        param = parameter('-c 4 -B 1')
        m = train(prob, param)


        outQueue.put((inVal * 2, m))
        print "done", inVal
        inQueue.task_done()

def Main():
    processes = []
    inQueue = mp.JoinableQueue()
    for i in xrange(10):
        inQueue.put(i)

    outQueue = mp.JoinableQueue()
    for i in xrange(5):
        process = mp.Process(target=processTarget, args=(inQueue, outQueue))
        print "starting", i
        process.start()
        print "started", i
    inQueue.join()

    print "JOINED"

    while(not outQueue.empty()):
        print outQueue.get()



if __name__ == '__main__':
    Main()

Chopfallen

Answer 1

When you use multiprocessing every process has its own address space. The pointer would not be valid in another process.

translate the object to a python object or to a ctypes type without pointers and it should work.

Keep in mind that changes to the object that occur in the other process will not be reflected in the parent unless you send the object back on the queue.

Wallie

what does "translate the object to a python object or to a ctypes type without pointers" mean in practice? – Offshoot

@onurcanbektas The op stated they have a ctypes object that currently has pointers. In practice it means creating a normal python class that has the same content as the ctypes object or making sure that the ctypes object does not have any references to other objects. This can be done by replacing pointers with having the object actually contain the other one. – Wallie