How do I make a nasty C++ program scriptable with Python and/or Lua? [closed]

Question

I'm confronted with the task of making a C++ app scriptable by users. The app has been in development for several years with no one wasting a thought on this before. It contains all sorts of niceties like multithreading, template wizardry and multiple inheritance. As the scripting language, Python is preferred, but Lua might be accepted if it is significantly easier to implement.

Question 1

From what I have learned so far, there are broadly speaking two ways to integrate Python/Lua with C++ : "extending" and "embedding".

In this case, it looks like I need both. The scripting language need access to objects, methods and data from the app but needs to be called by the app once the user has written the script - without restarting anything.

How is this usually done in the real world?

Question 2

There seems to be a bewildering array of of manual solutions and binding generators out there, all of them less than perfect.

• swig, pyste, Py++, ctypes, Boost.Python sip, PyCXX, pybindgen, robin, (Cython/Pyrex, Weave)
• CppLua, Diluculum, Luabind, Luabridge, LuaCpp, Luna/LunaWrapper, MLuaBind, MultiScript, OOLua, SLB, Sweet Lua, lux (this list from the lua wiki)
• CPB, tolua, tolua++, toLuaxx, luna and again swig

Most commments on these found on the web are a little out of date. For example, swig is said to be difficult in non-trivial cases and to generate incomprehensible code. OTOH, it has recently gone to v2.0.

Some of the above use pygccxml to let gcc analyze the C++ code and then genarate the binding. I find this idea appealing, as gcc probably understands the code better than i do :-). Does this work well?

Testing them all might easily cost me half of the time allocated for the whole project.

So, which ones do you recommend?

Francinafrancine

This would work much better when actually posting these as 3 separate questions - one new Q per language for best binder. For that there are probably also duplicates already. – Truscott

“no one wasting a thought on this before” — Heh! I’ve worked on a few projects that had clearly been developed by someone very frugal with their thoughts :) – Diurnal

@ Georg Fritzsche I felt that separate questions might be frowned upon, a bit like crossposting in newsgroups. Duplicates: yes, but i can't find much about the specific problem of making things user scriptable. Do you know of an easily grokkable example of how to do this (extend and embed)? – Francinafrancine

It depends on what language you are talking about. Separate questions are much better here - they can potentially be answered by different people, are easier found when searching, ... In short, the more focused the better. (sidenote: That space after the @ makes notifications not work and you don't need the full name - see here.) – Truscott

Answer 1

I wouldn't recommend swig as it's hard to get it to generate satisfactory binding in complex situations: been there, done that. I had to write a horrible script that "parsed" the original C++ code to generate some acceptable C++ code that swig could chew and generate acceptable bindings. So, in general: avoid ANY solution that relies on parsing the original C++ program.

Between Lua and Python: I have found Lua MUCH, MUCH better documented and more cleanly implemented. Python has a GIL (global lock), whereas with Lua, you can have an interpreter instance in each thread, for example. So, if you can choose, I'd recommend Lua. It is smaller language, easier to comprehend, easier to embed (much cleaner and smaller API, with excellent documentation). I have used luabind for a small project of mine and found it easy to use.

Bureaucrat

GIL stands for global interpreter lock. If you have separate interpreters, I don't see how the GIL will be relevant unless you want threading inside each interpreter. – Eun

I'd go with Lua as well. The C API is very straight-forward. I almost always create Lua bindings by hand. – Electromechanical

@Eun This is a cite from Python's C API documentation at docs.python.org/c-api/init.html "The global interpreter lock is also shared by all threads, regardless of to which interpreter they belong." – Bureaucrat

@zcrba - Right, I stand corrected. Thanks for the reference. – Eun

Answer 2

Regarding Question 1 - yes, you need to do both.

You would expose your applications scripting interface to an embedded interpreter, which then runs the user script in question.

The Python manuals section on embedding includes a section Extending Embedded Python and there are tutorials for similar things for Lua, see e.g. this article. Of course there are probably easier ways to be found when you have decided with what scripting language and binding mechanism you want to go.

Truscott

Answer 3

Some of the above use pygccxml to let gcc analyze the C++ code and then genarate the binding. I find this idea appealing, as gcc probably understands the code better than i do :-). Does this work well?

A similar approach is used in the Lua Qt binding - lqt. It uses cpptoxml to generate an XML file containing all classes and methods with parameters, and generated binding C++ code according to that.

It works really well, the generator is able to bind almost all Qt classes and methods, including virtual methods, overloaded operators, selected template classes, signal/slot mechanism etc.

Although it was created specially for Qt, it has a "noqt" mode, in which it can serve as a generic binder, unfortunately I have no experience with it, so I cannot tell how much work it would be to bind your code.

Elroy

Answer 4

My experience may not be much, but I figure it's at least worth what you paid for it ;)

I've done some basic "hello world" python modules, and I couldn't really get into swig - it seemed like a lot of overhead for what I was doing. Of course it's also possible that it's just the right amount for your needs.

Lionel

Answer 5

Try Boost::Python, it has somewhat of a learning curve associated with it but it is the best tool for the job in my view, we have a huge real time system and developed the scripting library for the QA in Boost::Python.

Nylanylghau