Creating a scripting language to be used to create web pages

Question

I am creating a scripting language to be used to create web pages, but don't know exactly where to begin.

I have a file that looks like this:

---

mylanguagename(main) {
       OnLoad(protected) {
              Display(img, text, link);
       }

       Canvas(public) {
              Image img: "Images\my_image.png";
                    img.Name: "img";
                    img.Border: "None";
                    img.BackgroundColor: "Transparent";
                    img.Position: 10, 10;

              Text text: "This is a multiline str#ning. The #n creates a new line.";
                   text.Name: text;
                   text.Position: 10, 25;

              Link link: "Click here to enlarge img.";
                   link.Name: "link";
                   link.Position: 10, 60;

                   link.Event: link.Clicked;
       }

       link.Clicked(sender, link, protected) {
              Image img: from Canvas.FindElement(img);
                    img.Size: 300, 300;
       }
}

... and I need to be able to make that text above target the Windows Scripting Host. I know this can be done, because there used to be a lot of Docs on it around the net a while back, but I cannot seem to find them now.

Can somebody please help, or get me started in the right direction?

Thanks

Lysine

Answer 1

You're making a domain-specific language which does not exist. You want to translate to another language. You will need a proper scanner and parser. You've probably been told to look at antlr. yacc/bison, or gold. What went wrong with that?

And as an FYI, it's a fun exercise to make new languages, but before you do for something like this, you might ask a good solid "why? What does my new language provide that I couldn't get any other (reasonable) way?"

Blase

Thank you for your answer. This is just an exercise for me. I do understand that it's quite a learning curve, BUT it's something I've always wanted to do and I guess there are many languages out there that fulfill my needs, but I just have so many complaints with other languages and I would just like to create one myself, because aleast then, I'd be able to add/remove/mod features etc. – Lysine

@BenjaminCox indeed they were. I gave up after a few weeks. But I'm back at it now, with much more experience. Wish me luck! – Ferullo

Answer 2

The thing to understand about parsing and language creation is that writing a compiler/interpreter is primarily about a set of data transformations done to an input text.

Generally, from an input text you will first translate it into a series of tokens, each token representing a concept in your language or a literal value.

From the token stream, you will generally then create an intermediate structure, typically some kind of tree structure describing the code that was written.

This tree structure can then be validated or modified for various reasons, including optimization.

Once that's done, you'll typically write the tree out to some other form - assembly instructions or even a program in another language - in fact, the earliest versions of C++ wrote out straight C code, which were then compiled by a regular C compiler that had no knowledge of C++ at all. So while skipping the assembly generation step might seem like cheating, it has a long and proud tradition behind it :)

I deliberately haven't gotten into any suggestions for specific libraries, as understanding the overall process is probably much more important than choosing a specific parser technology, for instance. Whether you use lex/yacc or ANTLR or something else is pretty unimportant in the long run. They'll all (basically) work, and have all been used successfully in various projects.

Even doing your own parsing by hand isn't a bad idea, as it will help you to learn the patterns of how parsing is done, and so then using a parser generator will tend to make more sense rather than being a black box of voodoo.

Maharanee

Thank you very much for your answer kyoryu :). Could you kindly suggest any references/links that would help me get started parsing stuff by hand? – Lysine

Answer 3

Languages similar to C# are not easy to parse - there are some naturally left-recursive rules. So you have to use a parser generator that can deal with them properly. ANTLR fits well.

If PEG fits better, try this: http://www.meta-alternative.net/mbase.html

Jovitta

Left recursive doesn't make it hard to parse. LR parsers handle this all the time. (I've built a full C# 4.0 parser using GLR rather than LR [because its easier than even LR!] but none of the rules except those that form lists are left recursive. And list-forming rules are pretty easy to handle with every parser generator I've ever seen. – Afterclap

Yes, that's true if you are using a parser generator. But this is not a default option - normally one would expect a hand-written recursive descent parser. And that is exactly the case when C#-like syntax is not the best choice for a DSL. Of course it is very easy to handle if you know what you are doing. Not if it's your first DSL ever. And as for left recursive rules: expr = expr . method ( ...) | expr [ index ] | expr . field | ... - they're the evil side of C++-ish syntax. Unless you've got a GLR, of course. And thanks for -1 :) – Jovitta

P.S. of course the rules I've mentioned above needs not to be left recursive unless you want to actually use the resulting syntax tree. If it's only a parsing and pretty-printing you're after, it's ok to make them right-recursive. – Jovitta

Normally one would expet a hand-written recursive descent parser?Not in 2010. The expectation is as much automation as you can get, and even you suggested ANTLR and PEG. As for "evil" left recursion, LALR/LR/GLR all do your example fine. For LL, expr = expr A | expr = expr B is trivially refactored to expr = expr ( A | B). I don't get it, none of this is hard. – Afterclap

I use (and implement) parser generators. You do so too. But you certainly forgot what "normal" people do when they first encounter a parsing problem. I tried to convince many into using ANTLR or a similar technology - and failed. In 2009-2010, yes. I know your business is all about parsing (and so as mine, too), so your view is definitely biased. It's not hard for you - and it's hard for those who never implemented even an infix arithmetics before. And yes - it's certainly "trivially refactored" - at a cost of a more complex syntax tree that needs additional transform before it's usable. – Jovitta

btw, jfyi: expr = expr (A|B) is still left-recursive. Refactoring is a little bit more complex here, you'll end up with a right-associated sequence of postfixes. – Jovitta

Answer 4

So you want to translate C# programs to JavaScript? Script# can do this for you.

Ingoing

Sort of. The code above that I wrote is like a prototype. It's not actually c#, it's my own language. I'd like to be able to code in the way that I have above, and then convert it to Javascript. – Lysine

(Just to prevent confusion) the reason I have c# as a tag is simply because I would like to create a parser for this new language of mine, in c#. (If a parser is what would be needed for this type of thing.) – Lysine

Why do you want to develop a new language? Isn't there any that suits your needs? – Silvers

What goal do you want to achieve? Your sample suggests that you are heavily embedding data in your code - why do you want to do this instead of getting the data from a data store like a database or XML files? – Silvers

This is going to be a scripting language that can be used to create webpages. :) – Lysine

Answer 5

Rather than write your own language and then run a translator to convert it into Javascript, why not extend Javascript to do what you want it to do?

Take a look at jQuery - it extends Javascript in many powerful ways with a very natural and fluent syntax. It's almost as good as having your own language. Take a look at the many extensions people have created for it too, especially jQuery UI.

Countrywide

Answer 6

Assuming you are really dedicated to do this, here is the way to go. This is normally what you should do: source -> SCANNER -> tokens -> PARSER -> syntax tree

1) Create a scanner/ parser to parse your language. You need to write a grammar to generate a parser that can scan/parse your syntax, to tokenize/validate them.

I think the easiest way here is to go with Irony, that'll make creating a parser quick and easy. Here is a good starting point

http://www.codeproject.com/KB/recipes/Irony.aspx

2) Build a syntax tree - In this case, I suggest you to build a simple XML representation instead of an actual syntax tree, so that you can later walk the XML representation of your DOM to spit out VB/Java Script. If your requirements are complex (like you want to compile it or so), you can create a DLR Expression Tree or use the Code DOM - but here I guess we are talking about a translator, and not about a compiler.

But hey wait - if it is not for educational purposes, consider representing your 'script' as an xml right from the beginning, so that you can avoid a scanner/parser in between, before spitting out some VB/Java script/Html out of that.

Comminate

Thanks for your answer amazedsaint! much appreciated ;) – Lysine

Answer 7

You obviously need machinery designed to translate langauges: parsing, tree building, pattern matching, target-language tree building, target-language prettyprinting. You can try to do all of this with YACC (or equivalents), but you'll discover that parsing is only a small part of a full translator. This means there's a lot more work to do than just parsing, and that takes time and effort.

Our DMS Software Reengineering Toolkit is a commercial solution to building full translators for relatively modest costs.

If you want to do it on your own from the ground up as an exercise, that's fine. Just be prepared for the effort it really takes.

One last remark: designing a complete language is hard if you want to get a nice result.

Afterclap

Hey, Ira, don't you work for them? Shouldn't you put in a disclaimer that you're plugging for your company? – Maharanee

I pretty much am them. Check my bio. Doesn't change the validity of the answer. – Afterclap

I'm not doubting that, and frankly, your product looks very interesting to me - if I had an idea of how much it would cost, I'd be tempted to give it a shot. From my view, knowing that it's your product up front with the answer seems more honest, vs. potentially finding it out after investigation. Whether or not it's intended (and you've been pretty up-front when asked about this, so I'm not accusing you of any mal-intent), it can give an impression of dishonesty. Or, to put it another way - being honest about it makes you more likely to get my money :) – Maharanee

Answer 8

I don't wan to be rude... but why are you doing this?

Creating a parser for a regular language is a non-trivial task. Just don't do it.

Why don't you just use html, javascript and css (and jquery as someone above suggested)

If you don't know where to begin, then you probably don't have any experience of this kind and probably you don't have a good reason, why to do this.

I want to save you the pain. Forget it. It's probably a BAD IDEA!

M.

Transfer

Answer 9

1. Check out Constructing Language Processors for Little Languages. It's a very good intro I believe. In fact I just consulted my copy 2 days ago when I was having trouble with my template language parser.

2. Use XML if at all possible. You don't want to fiddle with a lexer and parser by hand if you want this thing in production. I've made this mistake a few times. You end up supporting code that you really shouldn't be. It seems that your language is mainly a templating language. XML would work great there. Just as ASPX files are XML. Your server side blocks can be written in Javascript, modified if necessary. If this is a learning exercise then do it all by hand, by all means.

I think writing your own language is a great exercise. So is taking a college level compiler writing class. Good luck.

Clinkstone

Answer 10

Personally I think that every self-imposed challenge is good. I do agree with the other opinions that if what you want is a real solution to a real life problem, it's probably better to stick with proved solutions. However, if as you said yourself, you have an academic interest into solving this problem, then I encourage you to keep on. If this is the case, I might point a couple of tips to get you on the track.

Parsing is not really an easy task, that is way we take at least a semester of it. However, it can be learned. I would recommend starting with Terrence Parr's book on language implementation patterns. There are many great books about compiling and parsing, probably the most loved and hated been the Dragon Book.

This is pretty heavy stuff, but if you are really into this, and have the time, you should definitely take a look. This would be the Robisson Crusoe's "i'll make it all by myself approach". I have recently written an LR parser generator and it took me no more than a long weekend, but that after reading a lot and taking a full two-semesters course on compilers.

If you don't have the time or simply don't want to learn to make a parser "like men do", then you can always try a commercial or academic parser generator. ANTLR is just fine, but you have to learn its meta-language. Personally I think that Irony is a great tool, specially because it stays inside C# and you can take a look at the source code and learn for yourself. Since we are here, and I'm not trying to make any advertisement at all, I have posted a tiny tool in CodePlex that could be useful for this task. Take a look for yourself, it's open-source and free.

As a final tip, don't get scared if someone tells you it cannot be done. Parsing is a difficult theoretical problem but it's nothing that can't be learned, and it really is a great tool to have in your portfolio. I think it speaks very good of a developer that he can write an descent-recursive parser by hand, even if he never has to. If you want to pursuit this goal to its end, take a college-level compilers course, you'll thank me in a year.

Chrystal