Non-C++ languages for generative programming?

Question

C++ is probably the most popular language for static metaprogramming and Java doesn't support it.

Are there any other languages besides C++ that support generative programming (programs that create programs)?

Lindeman

I have complained about StackOverflow moderation before, but let me refrain here, and merely point out an observation ... do you see an overload of "debate, arguments, polling or extended discussion" here? If that's what we're afraid of, let's assess just how bad it is in this question's responses. If, instead of a horrid Jerry Springer episode, we instead see intelligent, valuable information shared respectfully and appreciated, then perhaps we should leave the question open. Otherwise, we're worrying too much about problems we don't even have. – Mohler

I have edited this question to remove requests for "best". Please consider reopening. BTW: it was through the answers to this question that I first heard of Clojure. So it was constructive for me. Thanks to those who answered and commented! – Lindeman

Answer 1

The alternative to template style meta-programming is Macro-style that you see in various Lisp implementations. I would suggest downloading Paul Graham's On Lisp and also taking a look at Clojure if you're interested in a Lisp with macros that runs on the JVM.

Macros in Lisp are much more powerful than C/C++ style and constitute a language in their own right -- they are meant for meta-programming.

Bellinzona

Hmm, how are Lisp macros a language in their own right? It's all Lisp, that's the beauty of it. – Amiamiable

Macros are a little different than Lisp since they don't get evaluated and a bunch of special operators. I don't think you can build macros with Lisp (in the sense that you can't add macros to your lisp with just a bunch of functions). – Bellinzona

You can't add functions (or variables, or special operators, or...) to your Lisp with a bunch of macros, either. So functions are a language in their own right as well? – Ossuary

adding macros to a lisp that didn't have macros before is actually quite simple and does not involve changing meta-level constructs like special operators. the devil is in the details of course, but a v0.1 is a matter of a page or two... – Stolon

Would I be right in thinking that Lisp's meta-programming is not static (that is, at compile-time)? – Doviedow

Lisp macros can be plugged into the various phases of the compiler. I am not an expert, but there are reader macros that get called during parsing and compile time macros that generate code, for example. I think there are others. – Bellinzona

There are "read macros", these are used when reading source code (from the REPL or from file) and transform source to internal representation. There are "normal" macros, they're used during "macro-expansion time" (this is done in, essentially, "the minimal compilation phase", so pretty close to function definition). There are "compiler macros", that work as whole-function compilation replacement things, where you can emit different code depending on what you have in the way of constants, variables, known type restrictions and the like. – Cheboksary

Answer 2

let me list a few important details about how metaprogramming works in lisp (or scheme, or slate, or pick your favorite "dynamic" language):

• when doing metaprogramming in lisp you don't have to deal with two languages. the meta level code is written in the same language as the object level code it generates. metaprogramming is not limited to two levels, and it's easier on the brain, too.
• in lisp you have the compiler available at runtime. in fact the compile-time/run-time distinction feels very artificial there and is very much subject to where you place your point of view. in lisp with a mere function call you can compile functions to machine instructions that you can use from then on as first class objects; i.e. they can be unnamed functions that you can keep in a local variable, or a global hashtable, etc...
• macros in lisp are very simple: a bunch of functions stuffed in a hashtable and given to the compiler. for each form the compiler is about to compile, it consults that hashtable. if it finds a function then calls it at compile-time with the original form, and in place of the original form it compiles the form this function returns. (modulo some non-important details) so lisp macros are basically plugins for the compiler.
• writing a lisp function in lisp that evaluates lisp code is about two pages of code (this is usually called eval). in such a function you have all the power to introduce whatever new rules you want on the meta level. (making it run fast is going to take some effort though... about the same as bootstrapping a new language... :)

random examples of what you can implement as a user library using lisp metaprogramming (these are actual examples of common lisp libraries):

• extend the language with delimited continuations (hu.dwim.delico)
• implement a js-to-lisp-rpc macro that you can use in javascript (which is generated from lisp). it expands into a mixture of js/lisp code that automatically posts (in the http request) all the referenced local variables, decodes them on the server side, runs the lisp code body on the server, and returns back the return value to the javascript side.
• add prolog like backtracking to the language that very seamlessly integrates with "normal" lisp code (see screamer)
• an XML templating extension to common lisp (includes an example of reader macros that are plugins for the lisp parser)
• a ton of small DSL's, like loop or iterate for easy looping

Stolon

Thanks for an excellent answer. As per another answer, I've been checking out Clojure (a Lisp variant that runs on the JVM). You might find it interesting. A suggestion: add more links (e.g., slate, cl-delico, screamer, etc.) search is fine but a link saves readers' time. – Lindeman

to me Clojure is not interesting. it's just more approachable by the 80% crowd and therefore becoming more popular... the kernel programming language, the FONC project at VPRI.org are genuinely interesting, though. – Stolon

Answer 3

Template metaprogramming is essentially abuse of the template mechanism. What I mean is that you get basically what you'd expect from a feature that was an unplanned side-effect --- it's a mess, and (although tools are getting better) a real pain in the ass because the language doesn't support you in doing it (I should note that my experience with state-of-the-art on this is out of date, since I essentially gave up on the approach. I've not heard of any great strides made, though)

Messing around with this in about '98 was what drove me to look for better solutions. I could write useful systems that relied on it, but they were hellish. Poking around eventually led me to Common Lisp. Sure, the template mechanism is Turing complete, but then again so is intercal.

Common Lisp does metaprogramming `right'. You have the full power of the language available while you do it, no special syntax, and because the language is very dynamic you can do more with it.

There are other options of course. No other language I've used does metaprogramming better than Lisp does, which is why I use it for research code. There are lots of reasons you might want to try something else though, but it's all going to be tradeoffs. You can look at Haskell/ML/OCaml etc. Lots of functional languages have something approaching the power of Lisp macros. You can find some .NET targeted stuff, but they're all pretty marginal (in terms of user base etc.). None of the big players in industrially used languages have anything like this, really.

Condonation

Thanks for your answer! MP with C++ templates is definitely messy (that's one motivation for my question). When/if C++0x is available from compiler vendors it should get a little cleaner as many of the new language features are specifically to support MP. – Lindeman

Abuse or not abuse but in fact you have no other choice to write generic code in C++. – Antechoir

Answer 4

Nemerle and Boo are my personal favorites for such things. Nemerle has a very elegant macro syntax, despite its poor documentation. Boo's documentation is excellent but its macros are a little less elegant. Both work incredibly well, however.

Both target .NET, so they can easily interoperate with C# and other .NET languages -- even Java binaries, if you use IKVM.

Edit: To clarify, I mean macros in the Lisp sense of the word, not C's preprocessor macros. These allow definition of new syntax and heavy metaprogramming at compiletime. For instance, Nemerle ships with macros that will validate your SQL queries against your SQL server at compiletime.

Formalize

Thanks for your answers, perhaps you could edit to add links for Nemerle and Boo? – Lindeman

The site for nemerle is nemerle.org. Boo is located at boo.codehaus.org – Trude

Answer 5

Nim is a relatively new programming language that has extensive support for static meta-programming and produces efficient (C++ like) compiled code.

http://nim-lang.org/

It supports compile-time function evaluation, lisp-like AST code transformations through macros, compile-time reflection, generic types that can be parametrized with arbitrary values, and term rewriting that can be used to create user-defined high-level type-aware peephole optimizations. It's even possible to execute external programs during the compilation process that can influence the code generation. As an example, consider talking to a locally running database server in order to verify that the ORM definition in your code (supplied through some DSL) matches the schema of the database.

Raji

Answer 6

The "D" programming language is C++-like but has much better metaprogramming support. Here's an example of a ray-tracer written using only compile-time metaprogramming:

Ctrace

Additionally, there is a gcc branch called "Concept GCC" that supports metaprogramming contructs that C++ doesn't (at least not yet).

Concept GCC

Intercalation

Answer 7

Common Lisp supports programs that write programs in several different ways.

1) Program data and program "abstract syntax tree" are uniform (S-expressions!)

2) defmacro

3) Reader macros.

4) MOP

Of these, the real mind-blower is MOP. Read "The Art of the Metaobject Protocol." It will change things for you, I promise!

Brogan

Thanks for your answer, I'll definitely check out MOP! A suggestion: to save readers' time, edit answer to add links or publication information for your references. – Lindeman

MOP rocks; Kizcales is now into Aspect Oriented Programming, which can be seen as MOP for Java, C#, etc. – Underdrawers

Answer 8

I recommend Haskell. Here is a paper describing its compile-time metaprogramming capabilities.

Millesimal

Answer 9

Lots of work in Haskell: Domain Specific Languages (DSL's), Executable Specifications, Program Transformation, Partial Application, Staged Computation. Few links to get you started:

• http://haskell.readscheme.org/appl.html
• http://www.cse.unsw.edu.au/~dons/papers/SCKCB07.html
• http://www.haskell.org/haskellwiki/Research_papers/Domain_specific_languages

Underdrawers

Answer 10

'metaprogramming' is really a bad name for this specific feature, at least when you're discussing more than one language, since this feature is only needed for a narrow slice of languages that are:

• static
• compiled to machine language
• heavily optimised for performance at compile time
• extensible with user-defined data types (OOP in C++'s case)
• hugely popular

take out any one of these, and 'static metaprogramming', just doesn't make sense. therefore, i would be surprised if any remotely mainstream language had something like that, as understood on C++.

of course, dynamic languages, and several functional languages support totally different concepts that could also be called metaprogramming.

Ichnology

Answer 11

The ML family of languages were designed specifically for this purpose. One of OCaml's most famous success stories is the FFTW library for high-performance FFTs that is C code generated almost entirely by an OCaml program.

Cheers, Jon Harrop.

Tuddor

Answer 12

Most people try to find a language that has "ultimate reflection" for self-inspection and something like "eval" for reifying new code. Such languages are hard to find (LISP being a prime counterexample) and they certainly aren't mainstream.

But another approach is to use a set of tools that can inspect, generate, and manipulate program code. Jackpot is such a tool focused on Java. http://jackpot.netbeans.org/

Our DMS software reengineering toolkit is such a tool, that works on C, C++, C#, Java, COBOL, PHP, Javascript, Ada, Verilog, VHDL and variety of other languages. (It uses production quality front ends to enable it to read all these langauges). Better, it can do this with multiple languages at the same instant. See http://www.semdesigns.com/Products/DMS/DMSToolkit.html

DMS succeeds because it provides a regular method and support infrastructure for complete access to the program structure as ASTs, and in most cases additional data such a symbol tables, type information, control and data flow analysis, all necessary to do sophisticated program manipulation.

Hastings

Answer 13

Lisp supports a form of "metaprogramming", although not in the same sense as C++ template metaprogramming. Also, your term "static" could mean different things in this context, but Lisp also supports static typing, if that's what you mean.

Adieu

I understand “static metaprogramming” as meaning compile-time code transformation as opposed to dynamically intercepting/changing behaviour at run-time (as with metaclasses or EVAL, for instance). In this sense, Lisp macros are a kind of static metaprogramming facility. OP, is this what you mean? – Ossuary

There is a good reason that lisp has been called the "programmable programming language". It allows/encourages a much more general form of "metaprogramming" than things like the c++ template approach. – Condonation

Answer 14

The Meta-Language (ML), of course: http://cs.anu.edu.au/student/comp8033/ml.html

Wellfed

Answer 15

It does not matter what language you are using -- any of them is able to do Heterogeneous Generative Metaprogramming. Take any dynamic language such as Python or Clojure, or Haskell if you are a type-fan, and write models in this host language that are able to compile themself into some mainstream language you want or forced to use by your team/employer.

I found object graphs a good model for internal model representation. This graph can mix attributes and ordered subgraphs in a single node, which is native to attribute grammar and AST. So, object graph interpretation can be an effective layer between your host and target languages and can act itself as some sort of no-syntax language defined over data structures.

The closest model is an AST: describe AST trees in Python (host language) targets to C++ syntax (target language):

# from metaL import *
class Object:
    def __init__(self, V):
        self.val = V
        self.slot = {}
        self.nest = []


class Module(Object):
    def cc(self):
        c = '// \ %s\n' % self.head(test=True)
        for i in self.nest:
            c += i.cc()
        c += '// / %s\n' % self.head(test=True)
        return c

hello = Module('hello')

# <module:hello> #a04475a2

class Include(Object):
    def cc(self):
        return '#include <%s.h>\n' % self.val

stdlib = Include('stdlib')
hello // stdlib

# <module:hello> #b6efb657
#     0: <include:stdlib> #f1af3e21

class Fn(Object):
    def cc(self):
        return '\nvoid %s() {\n}\n\n' % self.val

main = Fn('main')
hello // main

print(hello.cc())

// \ <module:hello>
#include <stdlib.h>

void main() {
}

// / <module:hello>

But you are not limited with the level of abstraction of constructed object graph: you not only can freely add your own types but object graph can interpret itself, can thus can modify itself the same way as lists in a Lisp can do.

Ugo