Evaluating the Strategies of Statement Selection in Automated Program Repair

  • Deheng Yang
  • Yuhua QiEmail author
  • Xiaoguang Mao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11293)


Automated program repair has drawn significant attention in recent years for its potential to alleviate the heavy burden of debugging activities. Fault localization, which generally provides a rank list of suspicious statements, is necessary for automated repair tools to identify the fault. With such rank list, existing repair tools have two statement selecting strategies for statement modification: suspiciousness-first algorithm (SFA) based on the suspiciousness of statements and rank-first algorithm (RFA) relying on the rank of statements. However, despite the extensive application of SFA and RFA in repair tools and different selecting methods between both strategies, little is known about the performance of the two strategies in automated program repair.

In this paper we conduct an empirical study to compare the effectiveness of SFA and RFA in automated program repair. We implement SFA and RFA as well as 6 popular spectrum-based fault localization techniques into 4 automated program repair tools, and then use these tools to perform repair experiments on 44 real-world bugs from Defects4J to compare SFA against RFA in automated program repair. The results suggest that: (1) RFA performs statistically significantly better than SFA in 64.76% cases according to the number of candidate patches generated before a valid patch is found (NCP), (2) SFA achieves better performance in parallel repair and patch diversity, (3) The performance of SFA can be improved by increasing the suspiciousness accuracy of fault localization techniques. These observations provide directions for future research on the usage of statement selecting strategies and a new avenue for improving the effectiveness of fault localization techniques in automated program repair.


Automated program repair Fault localization Empirical study 



This research was partially supported by National Natural Science Foundation of China (61502015, 61672529, 61379054, 61602504).


  1. 1.
    Gazzola, L., Micucci, D., Mariani, L.: Automatic software repair: a survey. In: ICSE (2018)Google Scholar
  2. 2.
    Qi, Y., et al.: Using automated program repair for evaluating the effectiveness of fault localization techniques. In: ISSTA (2013)Google Scholar
  3. 3.
    Wong, W.E., et al.: A survey on software fault localization. IEEE Trans. Softw. Eng. (TSE) 42(8), 707–740 (2016)CrossRefGoogle Scholar
  4. 4.
    Assiri, F.Y., Bieman, J.M.: Fault localization for automated program repair: effectiveness, performance, repair correctness. Softw. Qual. J. (SQJ) 25(1), 171–199 (2017)CrossRefGoogle Scholar
  5. 5.
    Monperrus, M.: Automatic software repair: a bibliography. ACM Comput. Surv. (CSUR) 51(1), 17 (2018)CrossRefGoogle Scholar
  6. 6.
    Chen, L., Pei, Y., Furia, C.A.: Contract-based program repair without the contracts. In: ASE (2017)Google Scholar
  7. 7.
    Saha, R.K., et al.: ELIXIR: effective object oriented program repair. In: ASE (2017)Google Scholar
  8. 8.
    Mechtaev, S., Yi, J., Roychoudhury, A.: Angelix: scalable multiline program patch synthesis via symbolic analysis. In: ICSE (2016)Google Scholar
  9. 9.
    Le, X.-B.D., et al.: JFIX: semantics-based repair of Java programs via symbolic PathFinder. In: ISSTA (2017)Google Scholar
  10. 10.
    Martinez, M., Monperrus, M.: Astor: a program repair library for Java. In: ISSTA (2016)Google Scholar
  11. 11.
    Xiong, Y., et al.: Precise condition synthesis for program repair. In: ICSE (2017)Google Scholar
  12. 12.
    Xuan, J., et al.: Nopol: automatic repair of conditional statement bugs in Java programs. IEEE Trans. Softw. Eng. (TSE) 43(1), 34–55 (2017)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Pearson, S., et al.: Evaluating and improving fault localization. In: ICSE (2017)Google Scholar
  14. 14.
    Yang, D., Qi, Y., Mao, X.: An empirical study on the usage of fault localization in automated program repair. In: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME NIER Track). IEEE (2017)Google Scholar
  15. 15.
    Kim, D., et al.: Automatic patch generation learned from human-written patches. In: ICSE (2013)Google Scholar
  16. 16.
    Le Goues, C., et al.: Genprog: a generic method for automatic software repair. TSE 38(1), 54–72 (2012)Google Scholar
  17. 17.
    Qi, Y., et al.: The strength of random search on automated program repair. In: ICSE (2014)Google Scholar
  18. 18.
    Le, X.B.D., Lo, D., Le Goues, C.: History driven program repair. In: SANER (2016)Google Scholar
  19. 19.
    Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: ISSTA (2014)Google Scholar
  20. 20.
    Keller, F., et al.: A critical evaluation of spectrum-based fault localization techniques on a large-scale software system. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE (2017)Google Scholar
  21. 21.
    Xie, X., et al.: A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. TOSEM 22(4), 31 (2013)CrossRefGoogle Scholar
  22. 22.
    Parnin, C., Orso, A.: Are automated debugging techniques actually helping programmers? In: ISSTA (2011)Google Scholar
  23. 23.
    Kochhar, P.S., et al.: Practitioners’ expectations on automated fault localization. In: ISSTA (2016)Google Scholar
  24. 24.
    Moon, S., et al.: Ask the mutants: mutating faulty programs for fault localization. In: ICST (2014)Google Scholar
  25. 25.
    Xiong, Y., et al.: Identifying patch correctness in test-based program repair. In: ICSE (2018)Google Scholar
  26. 26.
    Ye, H., Martinez, M., Monperrus, M.: A comprehensive study of automatic program repair on the QuixBugs benchmark. arXiv preprint arXiv:1805.03454 (2018)
  27. 27.
    Martinez, M., et al.: Automatic repair of real bugs in Java: a large-scale experiment on the Defects4J dataset. Empir. Softw. Eng. 22(4), 1936–1964 (2017)CrossRefGoogle Scholar
  28. 28.
    Tanikado, A., et al.: New strategies for selecting reuse candidates on automated program repair. In: COMPSAC (2017)Google Scholar
  29. 29.
    Xin, Q., Reiss, S.P.: Leveraging syntax-related code for automated program repair. In: ASE (2017)Google Scholar
  30. 30.
    Qi, Z., et al.: An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In: ISSTA (2015)Google Scholar
  31. 31.
    Durieux, T., et al.: The patches of the Nopol automatic repair system on the bugs of Defects4J version 1.1. 0. Dissertation, Université Lille 1-Sciences et Technologies (2017)Google Scholar
  32. 32.
    Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: ICSE (2011)Google Scholar
  33. 33.
    Xin, Q., Reiss, S.P.: Identifying test-suite-overfitted patches through test case generation. In: ISSTA (2017)Google Scholar
  34. 34.
    Tao, Y., et al.: Automatically generated patches as debugging aids: a human study. In: FSE (2014)Google Scholar
  35. 35.
    Mechtaev, S., et al.: Semantic program repair using a reference implementation. In: ICSE (2018)Google Scholar
  36. 36.
    Jia Y., et al.: Finding and fixing software bugs automatically with SapFix and Sapienz. (2018)
  37. 37.
    Wong, W.E., et al.: The DStar method for effective software fault localization. IEEE Trans. Reliab. 63, 290–308 (2014)CrossRefGoogle Scholar
  38. 38.
    Wen, M., et al.: Context-aware patch generation for better automated program repair. In: ICSE (2018)Google Scholar
  39. 39.
    Just, R., et al.: Comparing developer-provided to user-provided tests for fault localization and automated program repair. In: ISSTA (2018)Google Scholar
  40. 40.
    Jiang, J., et al.: Shaping program repair space with existing patches and similar code. In: ISSTA (2018)Google Scholar
  41. 41.
    Yang, D.: SFA-RFA. (2018)
  42. 42.
    Le, X.-B.D., et al.: S3: syntax-and semantic-guided repair synthesis via programming by examples. In: FSE (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.National University of Defense TechnologyChangshaChina
  2. 2.Beijing Institute of Tracking and Communication TechnologyBeijingChina
  3. 3.Laboratory of Software Engineering for Complex SystemsNational University of Defense TechnologyChangshaChina

Personalised recommendations