What is the preferred method for TCP/IP IPC in stackless Python?

Question

Stackless python allows you to serialize a task (pickling) for later execution that need not be on the same machine: http://www.stackless.com/wiki/Pickling

My question is does stackless python provide any kind of IPC, middleware, service broker, or DDS technology for moving these pickled tasks between processes and or machines? Is it really that we need to use a socket here?

They have this nice concept of a Channel: http://www.stackless.com/wiki/Pickling

That would be awesome if channels worked across machines and you could simple register a channel with a service broker on a network. In essence, allowing you to move your tasks to different stackless python services located on different machines.

Rasia

Celery seems to be one option from cpython. Was hoping for something more stackless specific though. I believe both have a "task" concept so would need to explore how those would integrate. – Rasia

maybe Pyro is more like what you need – Alumina

Pyro is helpful. I feel it is more of a general solution though like celery for the middleware piece. My intent with this post was to make sure I wasn't missing something from stackless that enabled some kind of distributed task scheduling out of the box. If nothing native in stackless is provided, I would accept an answer showing a simple python middleware (celery, pyro, etc) integration with stackless tasks. Personally, I decided to not use stackless. – Rasia

pyro is a good solution,it uses pickle for serialization. – Mitchel

The fact that it uses pickle is sort of a draw back if you want dynamic code loading for jobs. Pickle doesn't include object source. sPickle claims too but I couldn't get it to work: ep2012.europython.eu/conference/talks/… – Rasia

sPickle claims it is experimental. Consider, you can easy get automatically the source for a module, if the source exists. It is not necessary to copy the source tree structure for it. All sources can be saved to the same directory with a little mangled names. A big advantage of source being present are clear error messages. – Ninnette

Answer 1

The stacklessexamples project lists a number of approaches to do networking; most closely related to your question might be the rpc example.

Shaefer

Going to give you a +1 for the research. I am looking for something that is a little more example oriented in an answer and more directly distributed computing in nature... in contrast to just an RPC. I want something that can bundle code for execution remotely and is reasonable simple, i.e. not marshal and socket but more Channel/Broker/Task based API. Personally, I decided not to use stackless but would like to leave this open for other input or maybe someone does a lib for this kind of feature and wants to comment. – Rasia

After two years of leaving this open I am going to give it to you. – Rasia