Python vs Lua for embedded scripting/text processing engine

Question

For a project I'm currently working on, I'm looking to embed a scripting engine into my C++ code to allow for some extensibility down the line. The application will require a fair amount of text processing and the use of regular expressions within these scripts.

I know Lua is generally the industry darling when it comes to embedded scripting, but I also know it doesn't support regular expressions (at least out of the box). This is causing me to lean toward python for my language to embed, as it seems to have the best support behind Lua and still offers powerful regex capabilities.

Is this the right choice? Should I be looking at another language? Is there a reason I should give Lua a second look?

Unavailing

The key to this question is balancing what you get against the hoops the language will make you jump through, something that only seems to be explicitly addressed in Roger's answer. – Arlindaarline

Lua does have pattern matching out of the box although it is not POSIX reg exps - lua.org/pil/20.2.html – Lempres

Answer 1

if you need specifically what is commonly known as 'regular expressions' (which aren't regular at all), then you have two choices:

1. go with Python. it's included regexp is similar enough to Perl's and sed/grep
2. use Lua and an external PCRE library

if, on the other hand, you need any good pattern matching, you can stay with Lua and either:

1. use Lua's included pattern matching, which aren't in the grep tradition but are quite capable. The missing functionality is subpattern alternatives (|)
2. use LPEG, which are far more powerful than regexps, and usually faster too.

As you can tell, i'm a big fan of the last. It not only lets you define very complex but deterministic patterns, it's a full grammar tool that you can use to create a whole parser. If you wish, the grammar can be described in a single multi-line string constant, with your own defined hooks to capture data and build your structures.

i've used it to quickly hack a JSON parser, a C call-tree, an xPath library, etc.

Kadiyevka

The built-in patterns include a trick that PCRE and the like have a lot of trouble doing: balanced parenthesis matching with %b(). Any pair of distinct characters can be used; the item matches strings that start with (, end with ), and where the ( and ) are balanced. – Moonraker

Having never programmed anything more than a simple World of Warcraft extension in Lua myself, can you speak to the abilities to design advanced classes and functions? I know it supports object oriented programming, but I've only heard of it being used for fairly simple applications. I can foresee my developers using this for advanced machine learning functions, as well as network communication. I've done all of these in Python with no problem. Is Lua as straightforward for these things? – Unavailing

@Wade Tandy: that's a whole big subject. In fact Lua doesn't 'come with' OOP facilities, but it's syntax supports many different OO styles. for most uses, just a couple of lines to create appropriate metatables is enough. but if you need full static-like class-based inheritance, it quickly becomes a non-trivial project by itself. in my case, i've very seldom wanted class-based OOP. encapsulation and polymorphism are the real advantages of OOP, and you get them almost automatically with a little code discipline – Kadiyevka

Answer 2

Python and C++ integration is greatly helped with boost.python. You may find this much more convenient if those familiar with your C++ source are primarily the ones writing scripts.

Even if the scripters aren't familiar with your particular source, if they are more familiar with C-like syntax (C, C++, etc.), they should find Python easier to use—perhaps only slightly, Lua isn't hard. Good programmers can use a multitude of languages anyway, but you've not given any information about your audience.

Lua is much easier to sandbox than Python, so if you must restrict what scripts can do (e.g. spawn additional processes, read files), that may rule out Python.

Borneol

boost.python? Sounds interesting. How good is it at freeing you from worrying about Python's state? – Arlindaarline

@Charles: Internal interpreter state, such as you have to worry about when using Python's C API? I've not yet had to even think about that with boost.python, but I've also not used it extensively. – Borneol

Right, and boundary issues with the memory managers. The boost.python FAQ says it doesn't work with multiple threads without a fair bit of care. – Arlindaarline

Does anything work with multiple threads without a fair bit of care? :) – Borneol

Answer 3

Having incorporated Lua in one of my C projects myself, I'll suggest Lua, as this is easier.

But that depends on what your scripting language needs to be capable of. Lua rose to the de-facto scripting language of games. If you need advanced scripting capabilities, you might use Python, but if it's just for easy scripting support, take Lua. From what I've seen, Lua is easier to learn for newbees, that aren't used to scripting.

I'd argue, that Lua is lighter, if you need to have external packages, you can add them, but the point is, the atomic part of Lua, is much smaller than that of Python.

Predicable

Answer 4

dont forget the grand-daddy of them all - tcl

there is a c++ wrapper for tcl which makes it incredibly easy to embed

i am using it in a current project

in previous (c#) project I used lua over python. In older c# projects I had used python; I chose lua because the syntax is more normal for average scripter (used to vbscript or javascript). However I will change back to (iron)python for next c# project; lua is just too obscure

For c++ I will always use tcl from now on

EDIT: My new favorite is jint (.net javascriptt interpreter) v easy to use, nice interface. And nobody can complain about the language given that js is the cool language at the moment

Senghor

Tcl was designed from the ground up for this kind of work, and it's still the best. I can only assume that the original authors of Lua simply hadn't heard of it! – Radioluminescence

@Radioluminescence Tcl is at best kind of "quirky" as a language and that's especially true of early versions that would have been around back when Lua was originally created (early '90s). Even if they had heard of tcl, they may simply not have liked it... (its quirkiness made tcl the subject of much debate from those who would rather have a more "real" language) – Vespid

@Gaius: Actually, I did some checking, and the Lua authors did know about tcl when they first made Lua: “In 1993, the only real contender was Tcl, which had been explicitly designed to be embedded into applications. However, Tcl had unfamiliar syntax, did not offer good support for data description, and ran only on Unix platforms.” [from the paper The evolution of Lua ( lua.org/doc/hopl.pdf )] – Vespid

IMHO TCL is syntactically antiquated. Go with Python, if ease of written expression or readability is important. – Gesso