Efficient Python IPC [closed]

Question

I'm making an application in Python3, which will be divided in batch and gui parts. Batch is responsible for processing logic and gui is responsible for displaying it.

Which inter-process communication (IPC) framework should I use with the following requirements:

1. The GUI can be run on other device than batch (GUI can be run on the same device, on smartphone, tablet etc, locally or over network).
2. The batch (Python3 IPc library) should work with no problem on Linux, Mac, Windows, ...
3. The IPC should support GUI written in different languages (Python, Javascript, ...)
4. The performance of IPC is important - it should be as "interactive" as possible, but without losing information.
5. Several GUI could be connected to the same batch.

additional: Will the choice be other if the GUI will be guaranteed to be written in Python also?

Edit: I have found a lot of IPC libraries, like here: Efficient Python to Python IPC or ActiveMQ or RabbitMQ or ZeroMQ or.

The best looking options I have found so far are:

• rabbitmq
• zeromq
• pyro

Are they appropriate slutions to this problem? If not why? And if something is better, please tell me why also.

Gassaway

Use a SocketServer and connect the GUI through sockets. In this way you are sure it works on every platform and even through the net. – Superhuman

Thank you. Could you pleae justyfi why a SocketServer would be better choise than for example zeromq or rabbitmq? – Gassaway

assuming local IPC, using sockets is a horrible choice for numerous reasons. I personally recommend a non-standard anonymous pipe duplex using _winapi.ReadFile() and _winapi.WriteFile() with _winapi.DuplicateHandle() from the child process on the parent handles passed to it as that works directly in RAM. – Pithos

Answer 1

The three you mentioned seem a good fit and will uphold your requirements. I think you should go on with what you feel most comfortable\familiar with.

From my personal experience, I do believe ZeroMQ is the best combination between efficiency, ease of use and inter-operability. I had an easy time integrating zmq 2.2 with Python 2.7, so that would be my personal favorite. However as I said I'm quite sure you can't go wrong with all 3 frameworks.

Half related: Requirements tend to change with time, you may decide to switch framework later on, therefore encapsulating the dependency on the framework would be a good design pattern to use. (e.g. having a single conduit module that interacts with the framework and have its API use your internal datastructures and domain language)

Ageratum

Answer 2

I've used the Redis engine for this. Extremely simple, and lightweight.

Server side does:

import redis
r = redis.Redis() # Init
r.subscribe(['mychannel']) # Subscribe to "channel"
for x in r.listen():
  print "I got message",x

Client side does:

import redis
r = redis.Redis() # Init
r.publish('mychannel',mymessage)

"messages" are strings (of any size). If you need to pass complex data structures, I like to use json.loads and json.dumps to convert between python dicts/arrays and strings - "pickle" is perhaps the better way to do this for python-to-python communication, though JSON means "the other side" can be written in anything.

Now there are a billion other things Redis is good for - and they all inherently are just as simple.

Typhoid

redis no longer works like that. None of the commands are in the library anymore and from what i have gathered on the internet it basically is a very nice dictionary. Introductory blog post: realpython.com/python-redis – Extinct

Answer 3

You are asking for a lot of things from the framework; network enabled, multi-platform, multi-language, high performance (which ideally should be further specified - what does it mean, bandwidth? latency? what is "good enough"; are we talking kB/s, MB/s, GB/s? 1 ms or 1000 ms round-trip?) Plus there are a lot of things not mentioned which can easily come into play, e.g. do you need authentication or encryption? Some frameworks give you such functionality, others rely on implementing that part of the puzzle yourself.

There probably exists no silver bullet product which is going to give you an ideal solution which optimizes all those requirements at the same time. As for the 'additional' component of your question - yes, if you restrict language requirements to python only, or further distinguish between key vs. nice-to-have requirements, there would be more solutions available.

One technology you might want to have a look at is Versile Python (full disclosure: I am one of the developers). It is multi-platform and supports python v2.6+/v3, and java SE6+. Regarding performance, it depends on what are your requirements. If you have any questions about the technology, just ask on the forum.

Vitellin

Answer 4

The solution is dbus

It is a mature solution and availiable for a lot of languages (C, Python, ..., just google for dbus + your favorite language), though not as fast as shared memory, but still fast enough for pretty much everything not requiring (hard) realtime properties.

Upgrowth

But will it work over network? (additional is dbus working good on windows and macosx?) - I've added right now these requirements to the question. – Gassaway

Answer 5

I'll take a different tack here and say why not use the de facto RPC language of the Internet? I.e. HTTP REST APIs?

With Python Requests on the client side and Flask on the server side, you get these kinds of benefits:

• Existing HTTP REST tools like Postman can access and test your server.
• Those same tools can document your API.
• If you also use JSON, then you get a lot of tooling that works with that too.
• You get proven security practices and solutions (session based security and SSL).
• It's a familiar pattern for a lot of different developers.

Burly