Custom programming language: how?

Question

Hopefully this question won't be too convoluted or vague. I know what I want in my head, so fingers crossed I can get this across in text.

I'm looking for a language with a syntax of my own specification, so I assume I will need to create one myself. I've spent the last few days reading about compilers, lexers, parsers, assembly language, virtual machines, etc, and I'm struggling to sort everything out in terms of what I need to accomplish my goals (file attached at the bottom with some specifications). Essentially, I'm deathly confused as to what tools specifically I will need to use to go forward.

A little background: the language made would hopefully be used to implement a multiplayer, text-based MUD server. Therefore, it needs easy inbuilt functionality for creating/maintaining client TCP/IP connections, non-blocking IO, database access via SQL or similar. I'm also interested in security insofar as I don't want code that is written for this language to be able to be stolen and used by the general public without specialist software. This probably means that it should compile to object code

So, what are my best options to create a language that fits these specifications

My conclusions are below. This is just my best educated guess, so please contest me if you think I'm heading in the wrong direction. I'm mostly only including this to see how very confused I am when the experts come to make comments.

1. For code security, I should want a language that compiles and is run in a virtual machine. If I do this, I'll have a hell of a lot of work to do, won't I? Write a virtual machine, assembler language on the lower-level, and then on the higher-level, code libraries to deal with IO, sockets, etc myself, rather than using existing modules?

2. I'm just plain confused.

3. I'm not sure if I'm making sense.

If anyone could settle my brain even a little bit, I'd sincerely appreciate it! Alternatively, if I'm way off course and there's a much easier way to do this, please let me know!

Sherrillsherrington

This would be a great undertaking, Why not go with an interpreter instead and add security and make use of existing programming language? – Kilocycle

"For code security, I should want a language that compiles and is run in a virtual machine.": How a virtual machine is going to help with security? – Mcgarry

@Albireo, forgive him. @Sherrillsherrington To explain a bit, you may want to learn from Java: It's easily decompiled because it uses a virtual machine. – Kilocycle

Tell me if I'm wrong! That's just the conclusion I made, and I'm rather confused right now heh. My understanding is that if I use an interpreter and interpret to, say, C, then anyone could just run away with any files I produce for that game (after they're interpreted to C), and they'd have a useable copy of my game. However, if I have files that only run in a virtual machine/runtime environment, they files and code would be useless to anyone without the software. – Sherrillsherrington

Okay, sure, I get that the code itself would be easily read... But you wouldn't find it very easy to run a java source code file without the virtual machine, would you? This is more what I meant by security than seeing the actual raw code. – Sherrillsherrington

Sam, since you need to provide your user with the virtual machine, they can simple tamper the VM and obtain the same result. To obtain security you must check every action to check if it's legal or not server side, not client side. BTW, not passing informations not needed by the client (e.g. other players positions if they can't "see" them) is another thing you must do. – Mcgarry

@Sam: Unless your game is great, no one will steal it. If it is great, you will be able to afford some experts to help you make it hard to steal. And if you just offer access to the game via a web server, no one can steal the code unless you allow to download the code. The HTML pages which your code renders != your code. – Genoese

What you're trying to achieve is security through obscurity, which will (not if, but when) results in a fsckup. Build your system in a way that even if the user tampers with the client, the user will not be able to get any gain. – Mcgarry

I would recommend putting the security in the game itself. Since most of the people already have internet, the best way I can think of is to use a key that is verified online and focus more on making your online verification secure. – Kilocycle

I think the real question the comments should be asking is, what is it that you want from a custom language that you can't get from ANY other existing language? – Ermine

Answer 1

Designing a custom domain-specific programming language is the right approach to a problem. Actually, almost all the problems are better approached with DSLs. Terms you'd probably like to google are: domain specific languages and language-oriented programming.

Some would say that designing and implementing a compiler is a complicated task. It is not true at all. Implementing compilers is a trivial thing. There are hordes of high-quality compilers available, and all you need to do is to define a simple transform from your very own language into another, or into a combination of the other languages. You'd need a parser - it is not a big deal nowdays, with Antlr and tons of homebrew PEG-based parser generators around. You'd need something to define semantics of your language - modern functional programming langauges shines in this area, all you need is something with a support for ADTs and pattern matching. You'd need a target platform. There is a lot of possibilities: JVM and .NET, C, C++, LLVM, Common Lisp, Scheme, Python, and whatever else is made of text strings.

There are ready to use frameworks for building your own languages. Literally, any Common Lisp or Scheme implementation can be used as such a framework. LLVM has all the stuff you'd need too. .NET toolbox is ok - there is a lot of code generation options available. There are specialised frameworks like this one for building languages with complex semantics.

Choose any way you like. It is easy. Much easier than you can imagine.

Altissimo

Interesting. Nowdays I develop in the .net world often but for that, I think will check LLVM first. I'll add more after creating first my own language :) – Huygens

@QMaster, if performance or latency requirements are important, you'd certainly have to look at LLVM first - even before considering generating C. If you need batteries included (such as GC, for example), .net or jvm would be a bit easier to deal with. – Altissimo

I develop with Delphi and .net for years and continued nowadays. Exactly performance, clear coding, manageability and cross platforming are important for me. In the second level, I'll look at GC, etc. Thanks for your guides. – Huygens

Answer 2

Writing your own language and tool chain to solve what seems to be a standard problem sounds like the wrong way to go. You'll end up developing yet another language, not writing your MUD.

Many game developers take an approach of using scripting languages to describe their own game world, for example see: http://www.gamasutra.com/view/feature/1570/reflections_on_building_three_.php

Also see: https://stackoverflow.com/questions/356160/which-game-scripting-language-is-better-to-use-lua-or-python for using existing languages (Pythong and LUA) in this case for in-game scripting.

Machination

Answer 3

Since you don't know a lot about compilers and creating computer languages: Don't. There are about five people in the world who are good at it.

If you still want to try: Creating a good general purpose language takes at least 3 years. Full time. It's a huge undertaking.

So instead, you should try one of the existing languages which solves almost all of your problems already except maybe the "custom" part. But maybe the language does things better than you ever imagined and you don't need the "custom" part at all.

Here are two options:

1. Python, a beautiful scripting language. The VM will compile the language into byte code for you, no need to waste time with a compiler. The syntax is very flexible but since there is a good reason for everything in Python, it's not too flexible.

2. Java. With the new Xtext framework, you can create your own languages in a couple of minutes. That doesn't mean you can create a good language in a few minutes. Just a language.

Python comes with a lot of libraries but if you need anything else, the air gets thin, quickly. On a positive side, you can write a lot of good and solid code in a short time. One line of python is usually equal to 10 lines of Java.

Java doesn't come with a lot of frills but there a literally millions of frameworks out there which do everything you can image ... and a lot of things you can't.

That said: Why limit yourself to one language? With Jython, you can run Python source in the Java VM. So you can write the core (web server, SQL, etc) in Java and the flexible UI parts, the adventures and stuff, in Python.

Genoese

Typical CS students are able to tackle the compilation after a one week introduction course. There is literally nothing complicated in it. Some complex things may appear if you do a really harsh optimisation, or implement languages with unusual semantics. All the rest is just a commodity. No need in implementing your own GC, your own standard library, thinking of FFI implementation, etc. Its all available for free. You just have to combine the existing stuff. – Altissimo

Literally nothing complicated in the same way that calculus isn't complicated. If you study it, the details are clear. Most people haven't, so the details are not. Yes, there are tons of machinery lying around, some free, some commercial. But knowing how to design a good langauge is only driven somewhat by knowing how to implement one. And knowing how to do a good implementation is not something a CS grad with 1 week of Aho/Ullman can do. Sorry. – Forlini

If OP wants to do it for fun, that's fine. He should be aware that it will take a huge amount of effort to get something that can run what other people would consider to be a real program. – Forlini

3 years seems a bit exaggerated. I managed to make a JVM language with a ton of features (custom operators, lambda expressions, pattern matching, ...) in less than a year on my own while going to school. So compiler construction is not a big deal (even if you don't use any frameworks except for ASM). – Usa

@Clashsoft: How many people use your language? Do you have a good debugger? How many parts of your language could be better or which would you do different if you started again? I have designed many "simple" languages and then many people started using them and all kinds of surprising problems came up. I stand my be 3 years for a real, Java/Python/C# level language which is used by thousands of people all over the globe. – Genoese

I never stated that it was finished, only that it currently has many usable features :). github.com/Clashsoft/Dyvil – Usa

Answer 4

If you really want to create your own little language, a simpler and often quicker solution is to look at tools like lex and yacc and similar systems (ANTLR is a popular alternative), and then you can generate code either to an existing virtual machine or make a simple one yourself.

Making it all yourself is a great learning-experience, and will help you understand what goes on behind the scenes in other virtual machines.

Raffish

Answer 5

An excellent source for understanding programming language design and implementation concepts is Structure and Interpretation of Computer Programs from MIT Press. It's a great read for anyone wanting to design and implement a language, or anyone looking to generally become a better programmer.

Oedema

Answer 6

From what I can understand from this, you want to know how to develop your own programming language.

If so, you can accomplish this by different methods. I just finished up my own a few minutes ago and I used HTML and Javascript (And DOM) to develop my very own. I used a lot of x.split and x.indexOf("code here")!=-1 to do so... I don't have much time to give an example, but if you use W3schools and search "indexOf" and "split" I am sure that you will find what you might need. I would really like to show you what I did and past the code below, but I can't due to possible theft and claim of my work.

I am pretty much just here to say that you can make your own programming language using HTML and Javascript, so that you and other might not get their hopes too low. I hope this helps with most things....

Fradin