Flexible IPC solution for Python on Linux? [closed]

Question

I'm writing a program in Python for which I'm considering a local client-server model, but I am struggling to figure out the best way for the server to communicate with the client(s). A simple, canned solution would be best--I'm not looking to reinvent the wheel. Here are my needs for this program:

• Runs on Linux
• Server and clients are on the same system, so I don't need to go over a network.
• Latency that's not likely to be annoying to an interactive user.
• Multiple clients can connect to the same server.
• Clients are started independently of the server and can connect/disconnect at any time.
• The number of clients is measurable in dozens; I don't need to scale very high.
• Clients can come in a few different flavors:
  1. Stream readers - Reads a continuous stream of data (in practice, this is all text).
  2. State readers - Reads some state information that updates every once in a while.
  3. Writers - Sends some data to the server, receives some response each time.

Client type 1 seems simple enough; it's a unidirectional dumb pipe. Client type 2 is a bit more interesting. I want to avoid simply polling the server to check for new data periodically since that would add noticeable latency for the user. The server needs some way to signal to all and only the relevant clients when the state information is updated so that the client can receive the updated state from the server. Client type 3 must be bidirectional; it will send user-supplied data to the server and receive some kind of response after each send.

I've looked at Python's IPC page (http://docs.python.org/2/library/ipc.html), but I don't think any of those solutions are right for my needs. The subprocess module is completely inappropriate, and everything else is a bit more low-level than I'd like.

The similar question Efficient Python to Python IPC isn't quite the same; I don't need to transfer Python objects, I'm not especially worried about CPU efficiency for the number of clients I'll have, I only care about Linux, and none of the answers to that question are especially helpful to me anyway.

Update:

I cannot accept an answer that just points me at a framework/library/module/tool without actually giving an explanation of how it can be used for my three different server-client relationships. If you say, "All of this can be done with named pipes!" I will have to ask "How?" Code snippets would be ideal, but a high-level description of a solution can work too.

Merth

What about PyRo (Python Remote Objects)? – Drip

@BartekBanachewicz That seems designed for passing shorter, specific messages back and forth in the form of Python objects, so it would work for client type 3 (send message, receive response), but seems poorly suited for client types 1 (an open-ended stream) and 2 (which would require the periodic polling I'm trying to avoid). – Merth

Answer 1

Have you already looked into ZeroMQ? It has excellent Python support, and the documented examples already cover your use cases.

It's easy to use on a single platform, single machine setup, but it can be very easily extended to a network.

Carhart

At a glance, this has a lot going for it. I'm not seeing the documented examples covering my use cases, though. I see a heaping pile of examples, but it would take me some time to dig through and find which might be appropriate for me. Would you point me at the relevant examples you've spotted so that I may more quickly evaluate this library? – Merth

I will update the answer as soon as I work through the pile myself. :) Meanwhile this could give you a good introduction: slideshare.net/fcrippa/… I see pub/sub for "state readers" and push/pull for the other clients (there's a whole chapter on request-response style). – Carhart

Thank you. While reading more about ZeroMQ, I discovered this StackOverflow answer: https://mcmap.net/q/63905/-activemq-or-rabbitmq-or-zeromq-or-closed which makes the claim that "Zmq supports many advanced messaging scenarios but...you’ll have to implement most of them yourself by combining various pieces of the framework..." That might make ZeroMQ less than ideal. Of course, if the examples implement something close enough to what I want, that may be a moot point. – Merth