Generating Rewritable Abstract Syntax Trees

A Foundation for the Rapid Development of Source Code Transformation Tools
  • Jeffrey L. Overbey
  • Ralph E. Johnson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5452)


Building a production-quality refactoring engine or similar source code transformation tool traditionally requires a large amount of hand-written, language-specific support code. We describe a system which reduces this overhead by allowing both a parser and a fully rewritable AST to be generated automatically from an annotated grammar, requiring little or no additional hand-written code. The rewritable AST is ideal for implementing program transformations that preserve the formatting of the original sources, including spacing and comments, and the system can be augmented to allow transformation of C-preprocessed sources even when the target language is not C or C++. Moreover, the AST design is fully customizable, allowing it to resemble a hand-coded tree. The amount of required annotation is typically quite small, and the annotated grammar is often an order of magnitude smaller than the generated code.


Abstract Syntax Source Text Concrete Syntax Node Class Original Source Code 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    ASC Center for Astrophysical Thermonuclear Flashes,
  2. 2.
    Bečička, J., Hřebejk, P., Zajac, P.: (Sun Microsystems): Using Java 6 compiler as a refactoring and an anaysis engine. In: 1st Workshop on Refactoring Tools (2007)Google Scholar
  3. 3.
    Bowdidge, R.: (Apple Computer): Personal communicationGoogle Scholar
  4. 4.
    Celentano, A.: Incremental LR parsers. Acta Inf. 10, 307–321 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Cordy, J.: The TXL source transformation language. Sci. Comp. Prog. 61, 190–210 (2006)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Cormen, T., Leiserson, C., Rivest, R., Stein, C.: Introduction to Algorithms. 2/e. MIT Press, Cambridge (2001)zbMATHGoogle Scholar
  7. 7.
    De Jonge, M., Visser, E., Visser, J.: XT: A bundle of program transformation tools. In: Proc. Lang. Descriptions, Tools and Applications 2001. Elec. Notes in Theoretical Comp. Sci, vol. 44, pp. 211–218 (2001)Google Scholar
  8. 8.
    DeRemer, F.: Practical translators for LR(k) languages. PhD thesis, MIT (1969)Google Scholar
  9. 9.
    Eclipse C/C++ Development Tools,
  10. 10.
    Eclipse Java Development Tools,
  11. 11.
    Ernst, M., Badros, G., Notkin, D.: An empirical analysis of C preprocessor use. IEEE Trans. on Software Eng. 28, 1146–1170 (2002)CrossRefGoogle Scholar
  12. 12.
    Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design patterns: Elements of reusable object-oriented software. Addison-Wesley, Reading (1994)zbMATHGoogle Scholar
  13. 13.
    Garrido, A., Johnson, R.: Challenges of refactoring C programs. In: Proc. Intl. Workshop on Principles of Software Evolution, pp. 6–14 (2002)Google Scholar
  14. 14.
    Garrido, A.: Program refactoring in the presence of preprocessor directives. PhD thesis, University of Illinois at Urbana-Champaign (2005)Google Scholar
  15. 15.
    Garrido, A., Johnson, R.: Refactoring C with conditional compilation. In: Proc. 18th IEEE Intl. Conf. on Automated Software Eng., pp. 323–326 (2003)Google Scholar
  16. 16.
    Hopcroft, J., Motwani, R., Ulman, J.: Introduction to automata theory, languages, and computation. 2/e. Addison-Wesley, Reading (2001)Google Scholar
  17. 17.
  18. 18.
    ISO/IEC 9899:1999: Programming Languages – CGoogle Scholar
  19. 19.
    ISO/IEC 14882:2003: Programming Languages – C++Google Scholar
  20. 20.
  21. 21.
    Kort, J., Lämmel, R.: Parse-tree annotations meet re-engineering concerns. In: Proc. Source Code Analysis and Manipulation, pp. 161–172 (2003)Google Scholar
  22. 22.
    Lämmel, R.: Towards generic refactoring. In: Proc. ACM SIGPLAN Workshop on Rule-Based Prog., pp. 15–28 (2002)Google Scholar
  23. 23.
  24. 24.
  25. 25.
    McCloskey, B., Brewer, E.: ASTEC: A new approach to refactoring C. In: Proc. 13th ACM SIGSOFT Intl. Symp. Found. Software Eng., pp. 21–30 (2005)Google Scholar
  26. 26.
    Mens, T., Van Eetvelde, N., Demeyer, S., Janssens, D.: Formalizing refactorings with graph transformations. J. Software Maint. and Evolution 17, 247–276 (2005)CrossRefGoogle Scholar
  27. 27.
    Parr, T.: The definitive ANTLR reference: Building domain-specific languages. Pragmatic Bookshelf, Raleigh (2007)Google Scholar
  28. 28.
  29. 29.
    Platoff, M., Wagner, M., Camaratta, J.: An integrated program representation and toolkit for the maintenance of C programs. In: Proc. Conf. Software Maint., pp. 129–137 (1991)Google Scholar
  30. 30.
    Sellink, M., Verhoef, C.: Scaffolding for software renovation. In: Proc. Conf. Software Maint. Reeng., pp. 161–172 (2000)Google Scholar
  31. 31.
    Stallman, R., Weinberg, Z.: The C preprocessor,
  32. 32.
    Tomita, M.: Generalized LR parsing. Springer, Heidelberg (1991)CrossRefzbMATHGoogle Scholar
  33. 33.
    van den Brand, M.G.J., van Deursen, A., Heering, J., de Jong, H.A., de Jonge, M., Kuipers, T., Klint, P., Moonen, L., Olivier, P.A., Scheerder, J., Vinju, J.J., Visser, E., Visser, J.: The ASF+SDF meta-environment: A component-based language development environment. In: Wilhelm, R. (ed.) CC 2001. LNCS, vol. 2027, pp. 365–370. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  34. 34.
    Visser, E.: Stratego: A language for program transformation based on rewriting strategies. In: Middeldorp, A. (ed.) RTA 2001. LNCS, vol. 2051, pp. 357–361. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  35. 35.
    Vlissides, J.: Pattern hatching: Design patterns applied. Addison-Wesley, Reading (1998)Google Scholar
  36. 36.
    Wang, D., Appel, A., Korn, J., Serra, C.: The Zephyr abstract syntax description language. In: USENIX Workshop on Domain-Specific Langauges (1997)Google Scholar
  37. 37.
    Wile, D.: Abstract syntax from concrete syntax. In: Proc. 19th Intl. Conf. on Software Eng., pp. 472–480 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Jeffrey L. Overbey
    • 1
  • Ralph E. Johnson
    • 1
  1. 1.University of Illinois at Urbana-ChampaignUrbanaUSA

Personalised recommendations