Generate source code from AST with Antlr4 and StringTemplates
Asked Answered
B

1

7

If I have an AST and modify it, can I use StringTemplates to generate the source code for the modified AST?

I have successfully implemented my grammar for Antlr4. It generates the AST of a source code and I use the Visitor Class to perform the desired actions. I then modify something in the AST and I would like to generate the source code for that modified AST. (I believe it is called pretty-printing?).

Does Antlr's built in StringTemplates have all the functionality to do this? Where should one start (practical advice is very welcome)?

Balustrade answered 7/7, 2016 at 9:53 Comment(0)
W
4

You can walk the tree and use string templates (or even plain out string prints) to spit out text equivalents that to some extent reproduce the source text.

But you will find reproducing the source text in a realistic way harder to do than this suggests. If you want back code that the original programmer will not reject, you need to:

  • Preserve comments. I don't think ANTLR ASTs do this.
  • Generate layout that preserves the original indentation.
  • Preserve the radix, leading-zero count, and other "format" properties of literal values
  • Renerate strings with reasonable escapes

Doing all of this well is tricky. See my SO answer How to compile an AST back to source code for more details. (Weirdly, the ANTLR guy suggests not using an AST at all; I'm guessing this is because string templates only work on ANTLR parse trees whose structure ANTLR understands, vs. ASTs which are whatever you home-rolled.)

If you get all of this right, what you are likely to discover is that modifying the parse tree/AST is harder than it looks. For almost any interesting task on complex languages, you need information which is not trivial to extract from the tree (e.g., what is the meaning of this identifier?, where is this variable used?,...) I call this the problem of Life After Parsing. My main point is that it takes a lot of machinery to modify ASTs and regenerate code; be aware of the size of your project.

Weal answered 7/7, 2016 at 12:21 Comment(2)
Thanks for your answer. I am aware that this is not an easy task. Would zou also have some practical advice on how to start?Balustrade
If you insist on doing it yourself, the page I linked-to for prettyprinting is pretty good start considering you already have a parser. If you don't want to do all this work yourself, you should get a program transformation system (PTS) in which all the basic work has been done for you, allowing you to focus on your change task rather than building all that infrastructure. See en.wikipedia.org/wiki/Program_transformation for some possible choices.Weal

© 2022 - 2024 — McMap. All rights reserved.