Using Akka actors to invoke or pass messages to Python code

Question

I've a some analysis code that's written in Python. Currently I'm using Storm to process streams. Because Storm allows invocation of python code using message serialization I an invoke Python code from Java/Scala in my Storm bolts.

I found Pykka which is a Python implementation of the actor model. I was wondering if there is a way to invoke Python code from Akka actors? For example, is it possible to pass message from Akka actors to Pykka actors ?

Hadwyn

Answer 1

I doubt that the wire protocols for the two remote actor model implementations could be easily bridged, but you could use 0MQ between the Scala code and a Python app. Akka allows message passing over 0MQ, so after some setup the Scala code could deal with the Python app just like any other actor, although I'm not sure what that would mean on the Python side.

Another possibility worth considering is to run your Python analysis code on the JVM using Jython. You could have a Scala actor call a Jython function/method. But if your Python code makes use of C extension modules, you'd have to find alternatives.

Yet another possibility (also assuming you aren't using C extension modules) is to give the converter py2scala a try; for data analysis code it might do a decent job. Presumably this would give you the most performant solution.

Introversion

Would you please give a word here #28665537 – Endora

Answer 2

You could also use Jep to run embedded CPython in the actor JVM for full access to the native (C) Python packages.

See slide 5 here Mixing Python and Java.

If Python packages you need are not thread safe then can use remote Akka actors. See https://github.com/alpinedatalabs/alpine-r/blob/35209af47f896450e4f8f8745a80595d01035ca3/server/src/main/resources/application.conf#L4

To make the Scala to Python smoother try ScalaPy - also built on top of Jep.

Technically something similar should also work for R using Renjin - GPL, unfortunately so must be careful with running embedded.

Schizoid