Systematic Testing of Refactoring Engines on Real Software Projects

  • Milos Gligoric
  • Farnaz Behrang
  • Yilong Li
  • Jeffrey Overbey
  • Munawar Hafiz
  • Darko Marinov
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7920)


Testing refactoring engines is a challenging problem that has gained recent attention in research. Several techniques were proposed to automate generation of programs used as test inputs and to help developers in inspecting test failures. However, these techniques can require substantial effort for writing test generators or finding unique bugs, and do not provide an estimate of how reliable refactoring engines are for refactoring tasks on real software projects.

This paper evaluates an end-to-end approach for testing refactoring engines and estimating their reliability by (1) systematically applying refactorings at a large number of places in well-known, open-source projects and collecting failures during refactoring or while trying to compile the refactored projects, (2) clustering failures into a small, manageable number of failure groups, and (3) inspecting failures to identify non-duplicate bugs. By using this approach on the Eclipse refactoring engines for Java and C, we already found and reported 77 new bugs for Java and 43 for C. Despite the seemingly large numbers of bugs, we found these refactoring engines to be relatively reliable, with only 1.4% of refactoring tasks failing for Java and 7.5% for C.


Refactoring engines Systematic testing Test clustering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bettenburg, N., Premraj, R., Zimmermann, T., Kim, S.: Duplicate bug reports considered harmful... really? In: ICSM, pp. 337–345 (2008)Google Scholar
  2. 2.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on Java predicates. In: ISSTA, pp. 123–133 (2002)Google Scholar
  3. 3.
  4. 4.
    Cinnéide, M.Ó., Tratt, L., Harman, M., Counsell, S., Moghadam, I.H.: Experimental assessment of software metrics using automated refactoring. In: ESEM, pp. 49–58 (2012)Google Scholar
  5. 5.
    Coker, Z., Hafiz, M.: Program transformations to fix C integers. In: ICSE, pp. 792–801 (2013)Google Scholar
  6. 6.
    Counsell, S., Hassoun, Y., Loizou, G., Najjar, R.: Common refactorings, a dependency graph and some code smells: an empirical study of Java OSS. In: ISESE, pp. 288–296 (2006)Google Scholar
  7. 7.
    Daniel, B., Dig, D., Garcia, K., Marinov, D.: Automated testing of refactoring engines. In: FSE, pp. 185–194 (2007)Google Scholar
  8. 8.
    Dig, D., Johnson, R.: How do APIs evolve? A story of refactoring. J. Softw. Maint. Evol. 18(2), 83–107 (2006)CrossRefGoogle Scholar
  9. 9.
    Dig, D., Marrero, J., Ernst, M.D.: Refactoring sequential Java code for concurrency via concurrent libraries. In: ICSE, pp. 397–407 (2009)Google Scholar
  10. 10.
    Ebraert, P., D’Hondt, T.: Dynamic refactorings: Improving the program structure at run-time. In: RAM-SE, pp. 101–110 (2006)Google Scholar
  11. 11.
    Fowler, M.: Refactoring: Improving the Design of Existing Code. Addison-Wesley, Boston (1999)Google Scholar
  12. 12.
    Gligoric, M., Gvero, T., Jagannath, V., Khurshid, S., Kuncak, V., Marinov, D.: Test generation through programming in UDITA. In: ICSE, pp. 225–234 (2010)Google Scholar
  13. 13.
    GMP home page,
  14. 14.
    Godefroid, P., Kiezun, A., Levin, M.Y.: Grammar-based whitebox fuzzing. In: PLDI, pp. 206–215 (2008)Google Scholar
  15. 15.
    Hafiz, M., Overbey, J.: OpenRefactory/C: An infrastructure for developing program transformations for C programs. In: SPLASH, pp. 27–28 (2012)Google Scholar
  16. 16.
    Han, J.: Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers Inc. (2005)Google Scholar
  17. 17.
    Jagannath, V., Lee, Y.Y., Daniel, B., Marinov, D.: Reducing the costs of bounded-exhaustive testing. In: Chechik, M., Wirsing, M. (eds.) FASE 2009. LNCS, vol. 5503, pp. 171–185. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  18. 18.
  19. 19.
  20. 20.
    JUnit home page,
  21. 21.
    Eclipse Juno home page,
  22. 22.
  23. 23.
    Apache log4j home page,
  24. 24.
    Lucene home page,
  25. 25.
    Marinov, D., Khurshid, S.: TestEra: A novel framework for testing Java programs. In: ASE, pp. 22–31 (2001)Google Scholar
  26. 26.
    Commons Math home page,
  27. 27.
    Misherghi, G., Su, Z.: HDD: Hierarchical delta debugging. In: ICSE, pp. 142–151 (2006)Google Scholar
  28. 28.
    Murphy-Hill, E., Parnin, C., Black, A.P.: How we refactor, and how we know it. In: ICSE, pp. 287–297 (2009)Google Scholar
  29. 29.
    Negara, S., Chen, N., Vakilian, M., Johnson, R.E., Dig, D.: A comparative study of manual and automated refactorings. In: Castagna, G. (ed.) ECOOP 2013. LNCS, vol. 7920, pp. 552–576. Springer, Heidelberg (2013)Google Scholar
  30. 30.
    OpenRefactory/C - A refactoring infrastructure for C,
  31. 31.
    Overbey, J.L.: A Toolkit For Constructing Refactoring Engines. PhD thesis, University of Illinois at Urbana Champaign (2011)Google Scholar
  32. 32.
  33. 33.
    Regehr, J., Chen, Y., Cuoq, P., Eide, E., Ellison, C., Yang, X.: Test-case reduction for C compiler bugs. In: PLDI, pp. 335–346 (2012)Google Scholar
  34. 34.
    Schäfer, M.: Specification, Implementation and Verification of Refactorings. PhD thesis, Oxford University Computing Laboratory (2010)Google Scholar
  35. 35.
    Schäfer, M., Ekman, T., de Moor, O.: Sound and extensible renaming for Java. In: OOPSLA, pp. 277–294 (2008)Google Scholar
  36. 36.
    Schäfer, M., Sridharan, M., Dolby, J., Tip, F.: Refactoring Java programs for flexible locking. In: ICSE, pp. 71–80 (2011)Google Scholar
  37. 37.
    Schäfer, M., Thies, A., Steimann, F., Tip, F.: A comprehensive approach to naming and accessibility in refactoring Java programs. IEEE Trans. Soft. Eng. 38(6), 1233–1257 (2012)CrossRefGoogle Scholar
  38. 38.
    Soares, G.: Making program refactoring safer. In: ICSE, pp. 521–522 (2010)Google Scholar
  39. 39.
    Soares, G., Catao, B., Varjao, C., Aguiar, S., Gheyi, R., Massoni, T.: Analyzing refactorings on software repositories. In: SBES, pp. 164–173 (2011)Google Scholar
  40. 40.
    Song, Y., Wang, X., Xie, T., Zhang, L., Mei, H.: JDF: Detecting duplicate bug reports in Jazz. In: ICSE, pp. 315–316 (2010)Google Scholar
  41. 41.
    Spinellis, D.: CScout: A refactoring browser for C. Sci. of Comp. Prog. 75(4), 216–231 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Steimann, F., Kollee, C., von Pilgrim, J.: A refactoring constraint language and its application to eiffel. In: Mezini, M. (ed.) ECOOP 2011. LNCS, vol. 6813, pp. 255–280. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  43. 43.
    Sun, B., Shu, G., Podgurski, A., Ray, S.: CARIAL: Cost-aware software reliability improvement with active learning. In: ICST, pp. 360–369 (2012)Google Scholar
  44. 44.
    Thies, A., Steimann, F.: Systematic testing of refactoring tools. In: AST (poster) (2010),
  45. 45.
    Vakilian, M., Chen, N., Negara, S., Rajkumar, B.A., Bailey, B.P., Johnson, R.E.: Use, disuse, and misuse of automated refactorings. In: ICSE, pp. 233–243 (2012)Google Scholar
  46. 46.
    Visual Assist X home page,
  47. 47.
    Wang, X., Zhang, L., Xie, T., Anvik, J., Sun, J.: An approach to detecting duplicate bug reports using natural language and execution information. In: ICSE, pp. 461–470 (2008)Google Scholar
  48. 48.
    Wloka, J., Hirschfeld, R., Hänsel, J.: Tool-supported refactoring of aspect-oriented programs. In: AOSD, pp. 132–143 (2008)Google Scholar
  49. 49.
  50. 50.
    Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. Soft. Eng. 28(2), 183–200 (2002)CrossRefGoogle Scholar
  51. 51.
    zlib home page,

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Milos Gligoric
    • 1
  • Farnaz Behrang
    • 2
  • Yilong Li
    • 1
  • Jeffrey Overbey
    • 2
  • Munawar Hafiz
    • 2
  • Darko Marinov
    • 1
  1. 1.University of Illinois at Urbana-ChampaignUrbanaUSA
  2. 2.Auburn UniversityAuburnUSA

Personalised recommendations