What alternative user input techniques should be adopted in programming?

Question

Programming is particularly different to, for example word processing due to the wealth of special symbols etc that need to be entered.

Of the current crop of new user interface techniques, which are suited to programming and why?

Or is the idea of a language syntax the problem, should we be programming more symbolically, and if so, how would this affect user interface?

Edit: When I specified user interface techniques, i left it open to both using existing hardware (mouse/keyboard) and some other things, like multi-touch, gesture recognition, augmented reality (see HitLabNz for some great examples). I'm interested in how we can apply these to programming.

Grus

All we need is Emacs and C-x M-c M-butterfly ... xkcd.com/378 – Carrie

Answer 1

I was just thinking about this. I was going to write a blog post about it, I may as well get started here. In programming, I think we need more than just a new input method, there needs to be a new metaphor to go with it. It's a three tiered thing. Model-Metaphor-Interface.

I've been thinking more and more lately that language is a poor metaphor for representing a computation. Language is something we use for communication. You could look at a program as a communication to a computer, and simultaneously a communication to other programmers. But there are other ways to communicate other than just the written word. I'm working on a list here, feel free to edit this post to add more stuff to the list.

Methods of Communication

• Speech
• Body Language
• Gestures
• Facial Expressions
• Sign Language
• Painting/Drawing
• dials, buttons, sliders, pointing, dragging (gui)

Another metaphor for programming is building. Here's some possible ways of building functional things, that could form the basis for a programming interface

Methods of building functional things

• Gears/springs and other mechanics
• paper folding/cutting/gluing
• patch cords
• electronic circuits
• hinges, ball bearings, wheels
• fountains valves and pipes
• Archimedes machines: pulleys, levers, screws
• Lego

Yet another way of specifying a computation is by definition.

Methods of Definition - Constraints - Categorization - Set Theory - Properties - Symptoms - Logic tables - Rules - Railroads (as in railroad diagrams)

But keep in mind why we're doing this. There's obviously some weaknesses in the way programming languages work now, (otherwise we wouldn't want to make new languages) so let's keep them in mind while we're designing our new languages

problems with current languages

• The interface is hidden

• the APIs are hidden

• Side effects are a huge cause of bugs- Any part of a program can effect any other part.

• Refactoring- Sometimes you find that you're repeating yourself, so you need an easy way to factor out the repetition into a macro, or a function, or some other metaphor. This is largely done by hand (or semi-automatically in java) by a massive text manipulation effort. Is there a new metaphor that would make such a thing look utterly silly?

• You need an easy way to define your own building blocks, or "words" or idioms, to use to build more complex structures. Your own tools, your own parts of the environment. A lot of languages don't let you do this in a first class way.

• compilers punish the programmer severely for the slightest mistake.

• Variables lack a sense of time- There's no way to query the history of all the values a variable has been set to in the past. In other words, can we have a programming language where we can "rewind" the progress of our program? The fact that a variable can change, frequently to unexpected values is another source of bugs. This is the other half of the side effects problem

• most programming languages have a fairly steep learning curve

• Making reference to library or widget X throughout your code largely marries you to that library- Making it difficult to switch to a similar equivalent library without a lot of refactoring. This is largely to do with the fact that libraries have names, and in order to use a library, we're hardcoding the name of that library and its methods throughout our code. Is there a better way?

• Poor parallelism, multithreading leads to bugs, race conditions, deadlocks. Is there a better approach to parallelism that makes such bugs impossible? This issue alone is causing the creation of many new languages.

• Think beyond the computer screen, people. Maybe the keyboard is the most efficient interface for entering in complex relationships and symbols. Are you sure? There are more alternatives than just a mouse, or a touch screen, or a tablet. Zillions of ways of interacting with a computer- We have just all settled on one or two rather ordinary ways.

Gable

Answer 2

Just about every effort to have a non-textual programming language has fallen flat on its face. It's very hard to be both precise and efficient without a textual language.

Where a lot of the UI effort goes is in making better tools. For example, you can just use a simple text editor to do the programming or you can have full-blown IDEs like Visual Studio or Eclipse. Beyond that, there are visualization and design tools like Rational Rose. These tools offer complementary ways of exploring and/or modifying the underlying code.

Insect

Answer 3

I think that it's very important to have a textual representation of what you program, even if you use a non-textual (e.g. graphical, speech recognition, direct neural interface) input method.

A program is basically something like a recipe: "to achieve this, go through these steps". The textual representation is a write-up of this recipe. If you need a recipe for making the recipe ("click this menu, use this dialog box..."), and a textual interaction is not possible, then you lose contact with what you produce.

I think that a programmer's wish for alternative input/programming methods is correlated to the conceptual flaws in the language he uses. Recently I read about someone who got an assignment to write a bunch of setters and getters -- in a better language, this would be a job for its metaprogramming facilities.

On the subject of graphical programming: I can much faster type the word "for" than draw something like a triangle with the mouse. This is so even if this drawing is "facilitated" by letting me get this triangle from some menu. When programming, you use hundreds of different symbols; how can they be organized to access without typing? Heh, I know, how about keyboard shortcuts ... wait ...

The keyboard is currently the fastest instrument for conveying meaning to a computer, and a piece of text ist the most concise and useful representation for storing meaning on a computer.

Wanettawanfried

Your first point may be true, but the level of abstraction is subjective. Compare PDF with HTML. Most people don't realize that PDF is not a file format- it's a programming language source file, the kind you can open in a text editor. But that aspect is very effectively abstracted away. – Gable

HTML has graphical authoring environments, but quite a lot of the time you still need to know how to make the code by hand to get the more interesting effects, and for proper accessibility. – Gable

As for your second point, it's true- Drawing may not be the most effective metaphor. But there are others which may be more suitable than either drawing or writing. We won't really know until we try them. – Gable

Answer 4

Things like multitouch could potentially increase the mechanical efficiency of code creation, but I don’t think that’s a major concern in programming. When you consider all the work necessary to analyze, design, document, and test algorithms, the proportion of time spent actually typing code is so small that saving a key-press or mouse-click here or there isn’t going to amount to much.

It seems to me the main challenge in programming these days isn’t the special symbols or syntax, which are relatively few, mostly intuitive, and approaching a de facto standard of something close to C conventions. It seems to me the main challenge in programming is understanding the larger units of code, namely the functions and classes of both the API and the program itself: what is available, what each class does, what each function requires and returns, where there are similarities and differences, and how they fit together into an architecture. I believe the most serious problems are due to not understanding subtleties in the classes and functions for all possible combinations of conditions the code may encounter.

Maybe AR visualization techniques could help programmers and analysts see and manipulate the larger picture of the code structure or processes, but frankly a lot could be done with much more conventional UI techniques and controls, such as tables, forms, and menus, that have yet to make it into the coding world. The late Gupta/Centura programming language used a tree-like control to make it easier to see larger code structures. Intellisense is the right idea for prompting code creation, but more could be done to provide the developer the tools to understand and analyze the code at larger scales. Roedy Green’s Source Code in Database is a good start (http://mindprod.com/project/scid.html), allowing the developer to intelligently query the code base. Even better would be a programming UI that pushes analysis on the developer, making explicit what the developer needs to consider for a given program design.

Ung

Answer 5

Well the whole purpose of some kind of symbolic scenario really just boils down to GUI based development, which is a subject Visual Studio slowly flirts with, but we're still many years away from.

Creating some kind of symbol that represents a FOR loop wouldn't speed up development. If you want to code faster just use drag'n'drop code blocks that any decent IDE already supports.

Functionalism

Unless you come up with a programming metaphore that does away with the for loop, and replaces it with something more useful and expressive. – Gable

"for loop" is a language metaphore. If you're working in a different medium you need a different metaphore – Gable

@Travis: We are, in fact, not many years away from GUI development. Have a look at Labview, for instance. I'm not saying it's perfect (in fact, far from it), but you don't need to type a single line of code to implement an entire application. – Dwaindwaine

I think we're thinking a bit narrowly here. The goal may not always be to produce what we currently think of as a windowed gui application. The options for producing applications may be broader than just "use a gui, or use a text editor, them's your choices". look at my list. what would it mean... – Gable

to build a program out of lego blocks? I don't mean on a computer screen. I mean out of real blocks. Just as an example. – Gable

Answer 6

There have been several attempts at this.

APL which used a selection of special symbols to represent each operation. This still exists in the form of the "J" language which -- wait for it -- replaced all the symbols with two or three letter ASCII character combinations.

IBMs Visual Age circa 1998. Had a graphical IDE where you did things like drag a "socket" icons to your work space and join it to a "stream" icon. It just generated C++ though, and, after the intial g-whizz most programmers found the "text view" option and stuck with it.

Suns Fortress -- still textual but it lets you use unicode symbols like √ as operators. Most published examples programs seem to stick with the ASCII character set though.

There are two problems here.

Text is very very good! People have been able to draw pretty good pictures for thousands of years but 99% of the books Amazon sells contain only simple text. There is probably a good reason for this.

While a graphic for a "socket" and the associated "connection" points may be easy to use, its not trivial to develop. Instead of doing a method signature and perhaps a bit of Javadoc you now need a graphic designer to design an "Error message display icon" and define a set of rules as to how and where the graphic can be used.

Norman

The biggest obstacle for non textual programming languages in today's environment is that the computer interfaces which are now common and ubiquitous are optimized for textual input. APL in its original form was incredibly popular among those with access to the appropriate keyboard. – Gable

Answer 7

Wait... You need more than a text editor and a compiler/linker?

Syman