When is a new language the right tool for the job?

Question

For a long time I've been trying different languages to find the feature-set I want and I've not been able to find it. I have languages that fit decently for various projects of mine, but I've come up with an intersection of these languages that will allow me to do 99.9% of my projects in a single language. I want the following:

• Built on top of .NET or has a .NET implementation
• Has few dependencies on the .NET runtime both at compile-time and runtime (this is important since one of the major use cases is in embedded development where the .NET runtime is completely custom)
• Has a compiler that is 100% .NET code with no unmanaged dependencies
• Supports arbitrary expression nesting (see below)
• Supports custom operator definitions
• Supports type inference
• Optimizes tail calls
• Has explicit immutable/mutable definitions (nicety -- I've come to love this but can live without it)
• Supports real macros for strong metaprogramming (absolute must-have)

The primary two languages I've been working with are Boo and Nemerle, but I've also played around with F#.

Main complaints against Nemerle: The compiler has horrid error reporting, the implementation is buggy as hell (compiler and libraries), the macros can only be applied inside a function or as attributes, and it's fairly heavy dependency-wise (although not enough that it's a dealbreaker).
Main complaints against Boo: No arbitrary expression nesting (dealbreaker), macros are difficult to write, no custom operator definition (potential dealbreaker).
Main complaints against F#: Ugly syntax, hard to understand metaprogramming, non-free license (epic dealbreaker).

So the more I think about it, the more I think about developing my own language.

Pros:

• Get the exact syntax I want
• Get a turnaround time that will be a good deal faster; difficult to quantify, but I wouldn't be surprised to see 1.5x developer productivity, especially due to the test infrastructures this can enable for certain projects
• I can easily add custom functionality to the compiler to play nicely with my runtime
• I get something that is designed and works exactly the way I want -- as much as this sounds like NIH, this will make my life easier

Cons:

• Unless it can get popularity, I will be stuck with the burden of maintenance. I know I can at least get the Nemerle people over, since I think everyone wants something more professional, but it takes a village.
• Due to the first con, I'm wary of using it in a professional setting. That said, I'm already using Nemerle and using my own custom modified compiler since they're not maintaining it well at all.
• If it doesn't gain popularity, finding developers will be much more difficult, to an extent that Paul Graham might not even condone.

So based on all of this, what's the general consensus -- is this a good idea or a bad idea? And perhaps more helpfully, have I missed any big pros or cons?

Edit: Forgot to add the nesting example -- here's a case in Nemerle:

def foo = 
    if(bar == 5)
        match(baz) { | "foo" => 1 | _ => 0 }
    else bar;

Edit #2: Figured it wouldn't hurt to give an example of the type of code that will be converted to this language if it's to exist (S. Lott's answer alone may be enough to scare me away from doing it). The code makes heavy use of custom syntax (opcode, :=, quoteblock, etc), expression nesting, etc. You can check a good example out here: here.

Stately

Answer 1

Sadly, there's no metrics or stories around failed languages. Just successful languages. Clearly, the failures outnumber the successes.

What do I base this on? Two common experiences.

1. Once or twice a year, I have to endure a pitch for a product/language/tool/framework that will Absolutely Change Everything. My answer has been constant for the last 20 or so years. Show me someone who needs support and my company will support them. And that's that. Never hear from them again. Let's say I've heard 25 of these.

2. Once or twice each year, I have to work with a customer who has orphaned technology. At some point in the past, some clever programming built a tool/framework/library/package that was used internally for several projects. Then that programmer left. No one else can figure that darn thing out, and they want us to replace/rewrite it. Sadly, we can't figure it out either, and our proposal is to rewrite from scratch. And they complain that their genius built the set of apps in a period of weeks, it can't take us months to rewrite them in Java/Python/VB/C#. Let's say I've written 25 or so of these kinds of proposals.

That's just me, one consultant.

Indeed one particularly sad situation was a company who's entire IT software portfolio was written by one clever guy with a private language and tools. He hadn't left, but he'd realized that his language and toolset had fallen way behind the times -- the state of the art had moved on, and he hadn't.

And the move was -- of course -- in an unexpected direction. His language and tools were okay, but the world had started to adopt relational databases, and he had absolutely no way to upgrade his junk to move away from flat files. It was something he had not foreseen. Indeed, it was something he could not possibly foresee. [You won't fall into this trap, will you?]

So, we talked. He rewrote a lot of the applications in Plain-Old VAX Fortran (yes, this is a long time ago.) And he rewrote it to use plain old relational SQL stuff (Ingres, at the time.)

After a year of coding, they were having performance problems. They called me back to review all the great stuff they'd done in replacing the home-built language. Sadly, they'd done the worst possible relational database design. Worst possible. They'd taken their file copies, merges, sorts, and what-not, and implemented each low-level file system operation using SQL, duplicating database rows left, right and center.

He was so mired in his private vision of the perfect language, that he couldn't adapt to a relatively common, pervasive new technology.

Snailfish

Although I can't see these things happening (primarily due to the similarity to existing languages and the fact that this'll be used by everything in my company by a number of my developers), I guess that's the point :) Definitely going to keep this in mind. Thanks very much. – Stately

This is the best explanation, better than any I could give. @Cody: it sounds like you don't really want our advice. You just want to make a language. – Umbles

@Marcin, I'm sorry if my comment came off that way. What I was saying is that me not being able to see this far ahead is the same trap everyone else has fallen into, and why it's so important TO pay attention to his advice. – Stately

Good points! Point #1 has a name in finance and the mutual fund industry. See the <a href="en.wikipedia.org/wiki/Survivorship_bias">Survivorship Bias</a> – Hysterotomy

Answer 2

I say go for it.

• It would be an awesome experience regardless of weather it makes it to production or not.
• If you make it compile down to IL then you do not have to worry about not being able to re-use your compiled assemblies with C#
• If you believe that you have valid complaints about the languages you listed above, it is likely that many will think like you. Of course, for every 1000 interested person there might be 1 willing to help you maintain it - but that is always the risk

But here are a few things to be cautioned about:

• Get your language specification IN STONE before development. Make sure any and all language features are figured out before hand - even things that you may only want in the future. In my opinion, C# is slowly falling into the "oh-just-one-more-language-extension" trap that will lead to its eventual doom.
• Be sure to make it optimized. I dont know what you already know; but if you dont know then learn ;) Nobody will want a language that has nice syntax but runs as slow as IE's javascript implementation.

Good luck :D

Geophagy

Answer 3

When I first started my career in the early 90s, there seemed to be this craze of everyone developing their own in-house languages. My first 3 jobs were with companies that had done this. One company had even developed their own operating system!

From experience, I'd say this is a bad idea for the following reasons:

1) You will spend time debugging the language itself in addition to the code base on top of it
2) Any developers you hire will need to go through the learning curve of the language
3) It will be hard to attract and keep developers since working in a proprietary language is a dead-end for someone's career

The main reason I left those three jobs was because they had proprietary languages and you'll notice that not many companies take this route any more :).

An additional argument I'd make is that most languages have entire teams whose full time job it is to develop the language. Maybe you'd be an exception, but I'd be very surprised if you'd be able to match that level of development by only working on the language part-time.

Beadle

Answer 4

Main complaints against Nemerle: The compiler has horrid error reporting, the implementation is buggy as hell (compiler and libraries), the macros can only be applied inside a function or as attributes, and it's fairly heavy dependency-wise (although not enough that it's a dealbreaker).

I see your post has been written more than two years ago. I advise you trying Nemerle language today. The compiler is stable. There are no blocker bugs for today. The VS integration has a lot of improvements , also there is SharpDevelop integration.

If you give it a chance, you won't be disappointed.

Carinacarinate

Btw, there is VS2010 support too. – Carinacarinate

Answer 5

NEVER EVER develop your own language.

Developing your own language is a fool's trap, and worse it will limit you to what your imagination can provide, as well demanding that you work out both your development environment and the actual programme you're writing.

The cases in which this doesn't apply are pretty much if you're Larry Wall, the AWK guys, or part of a substantial group of people dedicated to testing the boundaries of programming. If you're in any of those categories, you don't need my advice, but I strongly doubt that you're targeting a niche where there is no suitable programming language for the task AND the characteristics of the people doing the task.

Umbles

Well, this language (if it happens) will be born out of the high-performance emulation/compilation/code transformation (all tightly tied together) space. Nemerle is currently being used for the majority of it, but the limitations have been showing and patched for a while. [...] – Stately

That said, thanks for your input. This is definitely not something to go into lightly, which is why I want as many opinions as possible :) Thanks. – Stately

Answer 6

If you are as clever as you seem to be (a likely possibility), my advice is to go ahead and do the design of the language first, iterate a couple of times over it, ask some smart fellows you trust in smart programming language related communities about the concrete design you came up with and then take the decision.

You might realize in the process of creating the design that just a quick hack on Nemerle would give it all you need, for example. Many things can happen just when thinking hard about a problem, and the final solution might not be what you actually had in mind when beginning the project.

Worst case scenario, you're stuck with actually implementing the design, but by then you will have it proof read and mature, and you'll know with a high degree of certainty that it was a good path to take.

A related piece of advice, start small, just define the features you absolutely need and then build on them to get the rest.

Kandrakandy

I've been considering this route (that is, modifying Nemerle), especially because the Nemerle implementation is so out of maintenance. It may even be worth it to take Nemerle and make a better implementation, since that's 99% of my problem with it. Thanks for the input. – Stately

While you are at it, you can add the couple of missing things and everybody lived happily ever after. – Kandrakandy

Answer 7

Writing your own language is not a easy project.. Especially one to be used in any kind of "professional setting"

It is a huge amount of work, and I would doubt you could write your own language, and still write any big projects that use it - you will spend so long adding features that you need, fixing bugs, and general language-design stuff.

I would strongly recommend choosing a language that is closest to what you want, and extending it to do what you need. It'll never be exactly what you want, but compared to the time you'll spend writing your own language, I would say that's a small compromise..

Tarragon

I'm actually not worried about the core language implementation time (I do recompiler work all the time and can reuse a ton of code) -- I'm really only worried about ongoing maintenance (there are always bugs) and the difficulty of finding developers. Thanks for your input, though :) – Stately

Answer 8

Scala has a .NET compiler. I don't know the status of this though. It's kind of a second class citizen in the Scala world (which is more focused on the JVM). But it might be a good tradeof to adopt the .NET compiler instead of creating a new language from scratch.

Scala is kind of weak in the meta-programming department ATM. It's possible that the need for metaprogramming is somewhat reduced by other language features. In any case I don't think anyone would be sad if you were to implement metaprogramming features for it. Also there is a compiler plug-in infrastructure on the way.

Chemar

Answer 9

I think most languages will never fit all of the bill.

You might want to combine your 2 favourite languages (in my case C# and Scheme) and use them together.

From a professional point of view, this probably not a good idea though.

Dehorn

Answer 10

It would be interesting to hear some of the things you feel you can't do in existing languages. What kind of projects are you working on that can't be done in C#?

I'm just curios!

Battiste

You should read Cody's whole post. The stuff about macros and metaprogramming is vital. – Olen

This is being used for high-performance emulation, compilation, and code transformation. Good case of why C# won't work worth a damn for this is the custom opcode syntax seen here: ironbabel.svn.sourceforge.net/viewvc/ironbabel/trunk/CPU/IA32/… – Stately