What's the best way to transfer data from python to another application in windows?

Question

I'm developing an application with a team in .Net (C++) and provide a COM interface to interact with python and other languages.

What we've found is that pushing data through COM turns out to be pretty slow.

I've considered several alternatives:

• dumping data to a file and sending the file path through com
• Shared Memory via mmap?
• Stream data through a socket directly?

From your experience what's the best way to pass around data?

Aguedaaguero

Is it huge amounts of data? Are you serializing it? What is the nature of the data? – Advertence

In most cases basically lat/lon point data. And at the moment it's been passed directly through COM, something like. AddPoint(lat, lon). – Aguedaaguero

Answer 1

Staying within the Windows interprocess communication mechanisms, we had positive experience using windows named pipes. Using Windows overlapped IO and the win32pipe module from pywin32.

You can learn much about win32 and python in the Python Programming On Win32 book.

The sending part simply writes to r'\\.\pipe\mypipe'.

A listener (ovpipe) object holds an event handle, and waiting for a message with possible other events involves calling win32event.WaitForMultipleObjects.

rc = win32event.WaitForMultipleObjects(
    eventlist,    # Objects to wait for.
    0,            # Wait for one object
    timeout)      # timeout in milli-seconds.

Here is part of the python overlapped listener class:

import win32event
import pywintypes
import win32file
import win32pipe

class ovpipe:
"Overlapped I/O named pipe class"
def __init__(self):
    self.over=pywintypes.OVERLAPPED()
    evt=win32event.CreateEvent(None,1,0,None)
    self.over.hEvent=evt
    self.pname='mypipe'
    self.hpipe = win32pipe.CreateNamedPipe(
        r'\\.\pipe\mypipe',             # pipe name 
        win32pipe.PIPE_ACCESS_DUPLEX|   # read/write access
        win32file.FILE_FLAG_OVERLAPPED,
        win32pipe.PIPE_TYPE_MESSAGE|    # message-type pipe 
        win32pipe.PIPE_WAIT,            # blocking mode 
        1,                              # number of instances 
        512,                            # output buffer size 
        512,                            # input buffer size 
        2000,                           # client time-out
        None)                           # no security attributes
    self.buffer = win32file.AllocateReadBuffer(512)
    self.state='noconnected'
    self.chstate()

def execmsg(self):
    "Translate the received message"
    pass

def chstate(self):
    "Change the state of the pipe depending on current state"
    if self.state=='noconnected':
        win32pipe.ConnectNamedPipe(self.hpipe,self.over)
        self.state='connectwait'
        return -6

    elif self.state=='connectwait':
        j,self.strbuf=win32file.ReadFile(self.hpipe,self.buffer,self.over)
        self.state='readwait'
        return -6

    elif self.state=='readwait':
        size=win32file.GetOverlappedResult(self.hpipe,self.over,1)
        self.msg=self.strbuf[:size]
        ret=self.execmsg()
        self.state = 'noconnected'
        win32pipe.DisconnectNamedPipe(self.hpipe)
        return ret

Disfranchise

Great! Thanks, I've heard of named pipes but have never worked with them, I'll have to check it out! – Aguedaaguero

This answer is unclear. Can you explain how to use these classes ? an example would be welcome. – Cirrhosis

@MikeTex The code is just a snippet of a larger program, not an example. You can write and (read) asynchronously to (from) the named pipe, using events with win32event.WaitForMultipleObjects() to handle synchronization. – Disfranchise

why use pywin32 which doesn't always work, and not the native _winapi and msvcrt which always works?? – Bifurcate

Answer 2

XML/JSON and a either a Web Service or directly through a socket. It is also language and platform independent so if you decide you want to host the python portion on UNIX you can, or if you want to suddenly use Java or PHP or pretty much any other language you can.

As a general rule proprietary protocols/architectures like COM offer more restrictions than they do benefits. This is why the open specifications appeared in the first place.

HTH

Unifoliolate

Answer 3

+1 on the named pipes but I would also like to add that from your comments it seems that your application is very chatty. Every time you make a remote call no matter how fast the underlying transport is you have a fixed cost of marshaling the data and making a connection. You can save a huge amount of overhead if you change the addpoint(lat, long) method to a addpoints(point_array) method. The idea is similar to why we have database connection pools and http-keep-alive connections. The less actual calls you make the better. Your existing COM solution may even be good enough if you can just limit the number of calls you make over it.

Inflatable

We've looked into this some, but we've hit memory issues when sending large amounts of data. The memory issue can be mitigated to some degree by sending data in blocks. Is there a better way? – Aguedaaguero

Answer 4

Upgrading answer of gimel on Python. It was really hard to make it work with events.

import multiprocessing

import win32event
import pywintypes
import win32file
import win32pipe

import client


class ovpipe:
    "Overlapped I/O named pipe class"

    def __init__(self):
        self.over = pywintypes.OVERLAPPED()
        evt = win32event.CreateEvent(None, 1, 0, None)
        self.over.hEvent = evt
        self.pname = 'mypipe'
        self.hpipe = win32pipe.CreateNamedPipe(
            r'\\.\pipe\mypipe',  # pipe name
            win32pipe.PIPE_ACCESS_DUPLEX |  # read/write access
            win32file.FILE_FLAG_OVERLAPPED,
            win32pipe.PIPE_TYPE_MESSAGE |  # message-type pipe
            win32pipe.PIPE_WAIT,  # blocking mode
            2,  # number of instances
            512,  # output buffer size
            512,  # input buffer size
            2000,  # client time-out
            None)  # no security attributes
        self.buffer = win32file.AllocateReadBuffer(512)
        self.state = 'noconnected'
        self.chstate()

    def execmsg(self):
        "Translate the received message"
        return bytes(self.msg).decode()

    def chstate(self):
        "Change the state of the pipe depending on current state"
        if self.state == 'noconnected':
            win32pipe.ConnectNamedPipe(self.hpipe, self.over)
            self.state = 'connectwait'
            return -6

        elif self.state == 'connectwait':
            j, self.strbuf = win32file.ReadFile(self.hpipe, self.buffer, self.over)
            self.state = 'readwait'
            return -6

        elif self.state == 'readwait':
            size = win32file.GetOverlappedResult(self.hpipe, self.over, 1)
            self.msg = self.strbuf[:size]
            ret = self.execmsg()
            self.state = 'noconnected'
            win32file.WriteFile(self.hpipe, b'Hello, client!')
            win32pipe.DisconnectNamedPipe(self.hpipe)
            return ret


def client_main():
    PIPE_NAME = r'\\.\pipe\MyPipe'

    # open a named pipe for writing
    pipe = win32file.CreateFile(
        PIPE_NAME,
        win32file.GENERIC_READ | win32file.GENERIC_WRITE,
        0,  # no sharing
        None,  # default security attributes
        win32file.OPEN_EXISTING,
        0,  # default attributes
        None  # no template file
    )

    # send data to the server
    data = b'Hello, server!'
    win32file.WriteFile(pipe, data)
    print(f'Sent Message to Server: \"{data.decode()}\". Waiting for response')

    # receive a response from the server
    response = win32file.ReadFile(pipe, 65536)
    print(f'Received response from server: {response}')

    # close the named pipe
    win32file.CloseHandle(pipe)


def main():
    ov = ovpipe()
    rc = win32event.WaitForMultipleObjects(
        [ov.over.hEvent],  # Objects to wait for.
        0,  # Wait for one object
        500)  # timeout in milli-seconds.
    print(rc)
    p = multiprocessing.Process(target=client_main, daemon=True)
    p.start()
    for _ in range(5):
        rc = win32event.WaitForMultipleObjects(
            [ov.over.hEvent],  # Objects to wait for.
            0,  # Wait for one object
            2000)  # timeout in milli-seconds.
        print(rc)
        win32event.ResetEvent(ov.over.hEvent)
        if rc == 0:
            res = ov.chstate()
            if type(res) is str:
                print(f"Received message from client: {res}")
                break
    p.join(10000)


if __name__ == '__main__':
    main()

Toolmaker

Answer 5

It shouldn't be too complicated to set up a test for each of your alternatives and do a benchmark. Noting beats context sensitive empirical data... :)

Oh, and if you do this I'm sure a lot of people would be interested in the results.

Onomatopoeia

Until we know more information this is a very sensible answer. – Advertence