The "right" way to add python scripting to a non-python application

Question

I'm currently in the process of adding the ability for users to extend the functionality of my desktop application (C++) using plugins scripted in python.

The naive method is easy enough. Embed the python static library and follow any number of the dozens of tutorials scattered around the web describing how to initialize and call python files, and you're pretty much done.

However...

What I'm looking for is more like what Blender does. Blender is completely customizable through python scripts, and it requires an external python executable. (Ie. python isn't actually embedded in the blender executable at all.) So, naturally, you can include any modules you already have in your site-packages directory when you are writing blender scripts. Not that that's advised, since that would limit the portability of your script.

So, what I want to know is if there is already a way to have your cake and eat it too. I want a plugin system that uses:

• An embedded python interpreter.

  The downside of Blender's approach is that it forces you to have a specific, possibly outdated version of python installed globally on your system. Having an embedded interpreter allows me to control what version of python is being used.

• Firewall plugins.

  Some equivalent of a virtualenv for each plugin; allowing them to install all the modules they need or want, but keeping them seperated from possible conflicts in other plugins. Maybe zc.buildout is a better candidate here, but, again, I'm very open to suggestion. I'm a bit at a loss as to the best way to accomplish this.

• As painless as possible...

  For the user. I'm willing to go the extra mile, just so long as most of the above is as transparent to the plugin writer as possible.

---

If any of you folks out there have any experience with this sort of thing, your help would be much appreciated. :)

---

Edit: Basically, the short version of what I want is the simplicity of virtualenv, but without the bundled python interpreter, and a way to activate a specific "virtual environment" programmatically, like zc.buildout does with sys.path manipulation (the sys.path[0:0] = [...] trick).

Both virtualenv and zc.buildout contain portions of what I want, but neither produce relocatable builds that I, or a plugin developer can simply zip up and send to another computer.

Simply manipulating .pth files, or manipulating sys.path directly in a script, executed from my application gets me half-way there. But it is not enough when compiled modules are necessary, such as the PIL.

Express

When you mention virtualenv, are you implying that you want to make it easy for plugin authors to build in external modules? After all, if you simply give each plugin its own sys.path entry, prebuilt inside that plugin's interpreter before plugin load, it's quite possible to for them to package most pure-Python modules without any real difficulty. – Jeopardy

Maybe I'm asking for the moon, but it would be great to have a way to bundle compiled modules, like the PIL, in the same way that you described bundling pure-python modules. – Express

Not exactly an answer to the question here, but I would not embed Python at all. Why limit yourself to a single scrip language? Instead expose the scripting API using COM, DBUS or another remote call protocol and write Python to that API. – Buiron

Answer 1

One effective way to accomplish this is to use a message-passing/communicating processes architecture, allowing you to accomplish your goal with Python, but not limiting yourself to Python.

------------------------------------
| App  <--> Ext. API <--> Protocol | <--> (Socket) <--> API.py <--> Script
------------------------------------

This diagram attempts to show the following: Your application communicates with external processes (for example Python) using message passing. This is efficient on a local machine, and can be portable because you define your own protocol. The only thing that you have to give your users is a Python library that implements your custom API, and communicates using a Send-Receive communication loop between your user's script and your application.

Define Your Application's External API

Your application's external API describes all of the functions that an external process must be able to interact with. For example, if you wish for your Python script to be able to draw a red circle in your application, your external API may include Draw(Object, Color, Position).

Define A Communication Protocol

This is the protocol that external processes use to communicate with your application through it's external API. Popular choices for this might be XML-RPC, SunRPC, JSON, or your own custom protocol and data format. The choice here needs to be sufficient for your API. For example, if you are going to be transferring binary data then JSON might require base64 encoding, while SunRPC assumes binary communication.

Build Your Application's Messaging System

This is as simple as an infinite loop receiving messages in your communication protocol, servicing the request within your application, and replying over the same socket/channel. For example, if you chose JSON then you would receive a message containing instructions to execute Draw(Object, Color, Position). After executing the request, you would reply to the request.

Build A Messaging Library For Python (or whatever else)

This is even simpler. Again, this is a loop sending and receiving messages on behalf the library user (i.e. your users writing Python scripts). The only thing this library must do is provide a programmatic interface to your Application's External API and translate requests into your communication protocol, all hidden from your users.

Using Unix Sockets, for example, will be extremely fast.

Plugin/Application Rendezvous

A common practice for discovering application plugins is to specify a "well known" directory where plugins should be placed. This might be, for example:

~/.myapp/plugins

The next step is for your application to look into this directory for plugins that exist. Your application should have a some smarts to be able to distinguish between Python scripts that are, and are not, real scripts for your application.

Let's assume that your communication protocol specifies that each script will communicate using JSON over StdInput/StdOuput. A simple, effective approach is to specify in your protocol that the first time a script runs it sends a MAGIC_ID to standard out. That is, your application reads the first, say, 8 bytes, and looks for a specific 64-bit value that identifies it as a script.

Additionally, you should include in your External API methods that allow your scripts to identify themselves. For example, a script should be able to inform the application through the External API things such as Name, Description, Capabilities, Expectations, essentially informing the application what it is, and what it will be doing.

Unbidden

This looks excellent, but I'm a little bit unfamiliar with this type of architecture. How would I go about "discovering" user plugins? – Express

@Express added application/plugin rendezvous suggestion. – Unbidden

This feels complicated. What is the bonus compared to other solutions that overcomes the need to write so many layers ? (except that you can extend your app with several different languages) – Leathaleather

Thanks, man! Yes, this is slightly more complicated, but being able to use several different languages interchangeably would be an enormous boon. One last question: In "Build A Messaging Library For Python" you mention that there would be, again, another message loop. I'm not sure I understand that bit. I was assuming that the python/lua/whatever messaging library would just be a collection of one-shot functions that fire a message over the specified protocol. Not a loop. A separate message loop implies a daemon of some-sort? – Express

Ahhh. So, in your messaging library (per-supported language) you need some active component that is capable of reading incoming messages, firing "one-shot functions" and returning the response. So, your application will fork off "python script.py" and immediately begin reading and responding over standard-in/out. It is not technically a daemon, but absolutely needs an active component. Does that answer your question? – Unbidden

Ahhh, indeed. For some reason when I was thinking it through in my mind I was only considering sending messages from scripts, not receiving messages in scripts. What you said makes perfect sense. Have a bonus, sir. You've earned it. :) – Express

@Noah : Did you already implement this in a concurrent context (different threads needing the same service) ? Did you have problems ? Is there any 'tip' you could share ? – Leathaleather

@Calvin1602: Sorry to but in, but on a related note, the only reason I'm considering this at all is because I'm using Qt, which, amongst other awesome things: A) Supports dbus out-of-the-box. And, B) Is completely reentrant. So, the "hard" stuff is already taken care of for me. My services are mostly thread-safe, even though I was not considering this kind of architecture when I started writing them. – Express

Answer 2

I don't see the problem with embedding Python with, for example, Boost.Python. You'll get everything you ask for:

• It will be embedded, and it will be an interpreter (with enough access to implement auto-completion and such)
• You can create a new interpreter per-script and have completely separated python environments
• ... and as transparent as it's possible

I mean, you'll still have to expose and implement an API, but 1) this is a good thing, 2) Blender does it too and 3) I really can't think of another way that leverages you from this work...

PS: I have little experience with Python / Boost.Python but have worked extensively with Lua / LuaBind, which is kinda the same

Leathaleather

This may be exactly what I end up doing, but creating an new interpreter per script is easy enough even without boost. The main problem as listed above is the inability to shield plugin's external modules from other plugin's external modules. – Express

Maybe you could have one directory for each plugin, with all the plugin depencencies in that directory ? But let's wait for less hack-ish solutions – Leathaleather

Answer 3

If you really want to be as painless as possible for you and your users, consider extending python rather than embedding.

• embedding doesn't allow easy integration with other software -- only one program that embeds python can be used at once by a python script. OTOH extending means the user can use your software anywhere python runs;
• To make stuff available for the script writer, you don't have to initialize an interpreter. the interpreter will be already initialized for you, saving you work.
• You don't have to create special built-in variables and fake modules to inject into your embedded interpreter. Just give them a real extension module instead, and you can initialize everything when your module is first imported.
• You can use distutils to distribute your software
• Tools like virtualenv can be used as-is -- you or the user don't have to come up with new tools. Your user can also use her IDE/debug tool/testing framework of choice

Embedding really buys nothing to you and your users.

Carburetor

What do you mean by extending ? – Leathaleather

@Calvin1602: By embedding and extending I mean what is mentioned here docs.python.org/extending/index.html (chapters 1-4 are about extending, and chapter 5 is about embedding) – Carburetor

ok. But : "If the main program (the Python interpreter)..." -> it seems to mean that it adds functionality to Python, not extendability to the app. Where do I go wrong ? Which program is started first ? – Leathaleather

Yes, I'm missing something here too. If I have a GUI application, using the "extended python" approach, then the event loop would be in a python script, correct? If so, then this is not what I want. – Express

@kurige: The event loop doesn't need to be in a python script. The event loop can be a function callable from Python, and need not return. It can then call out to callbacks registered through other PyObjects . gtk.main() from PyGTK and Tkinter.Tk.mainloop() -- from the python distribution itself! -- both do this. – Carburetor