Is it feasible to use Antlr for source code completion?

Question

I don't know, if this question is valid since i'm not very familiar with source code parsing. My goal is to write a source code completion function for one existing programming language (Language "X") for learning purposes. Is Antlr(v4) suitable for such a task or should the necessary AST/Parse Tree creation and parsing be done by hand, assuming no existing solutions exists?

I haven't found much information about that specific topic, except a list of compiler books, except a compiler is not what i'm after for.

Chaff

Intellij IDEA uses ANTLR for lots of different things that it's powerful parsing capabilities provide. Code completion and structural refactoring are just a couple of examples. – Over

Answer 1

The code completion in GoWorks is completely implemented using ANTLR 4. The following video shows the level of completion of this code completion engine. The code completion example runs from 5 minutes through the end of the video.

Intro to Tunnel Vision Labs' GoWorks IDE (Preview Release)

I have been working on code completion algorithms for many years, and strongly believe that there is no better solution (automated or manual) for producing a code completion solution for a new language that meets the requirements for what I would call highly-responsive code completion. If you are not interested in that level of performance or accuracy, other solutions may be easier for you to get involved with (I don't work with those personally, because I am too easily disappointed in the results).

Horrid

Being a fan of Go, this looks very awesome. Going off topic: are you planning on adding support for displaying documentation as well with code completion? – Chaff

Sam, I noticed that some years ago you were of the opinion that ANTLR "is not the correct method for implementing autocompletion". What changed your thinking? Can you please elaborate. – Gastroenteritis

@Gastroenteritis Well that sure was an interesting read! One big thing that changed is I got deeply involved in the development of ANTLR 4, and made sure that the complete rewrite addressed some of the concerns that made ANTLR 3 a challenging tool to use for this task. On the performance front, ANTLR does not parse an entire file in GoWorks, but rather parses from what are called Anchors. ForestParser and CaretToken cover the robustness situation. – Horrid

Answer 2

Xtext uses ANTLR3 and has good autocomplete facilities. The problem is, it generates a seperate parser (again using antlr3) for autocomplete processing which is derived from AbstractInternalContentAssistParser. This multi-thousand line code part shows that the error recovery of ANTLR3 alone found to be insufficient by the xtext team.

Meanwhile ANTLR4 has a function parser.getExpectedTokensWithinCurrentRule() which lists possible token types for given position. It works when used in a ParseTreeListener. Remaining is semantics, scoping etc which is out of ANTLRs scope.

Destiny