Skip to main content

Refining Fitness Functions for Search-Based Automated Program Repair

A Case Study with ARJA and ARJA-e

  • 360 Accesses

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12914)


Automated Program Repair (APR) strives to automatically fix faulty software without human-intervention. Search-based APR iteratively generates possible patches for a buggy software, guided by the execution of the patched program on a test suite (i.e., a set of test cases). Search-based approaches have generally only used Boolean test case results (i.e., pass or fail), but recently more fined-grained fitness evaluations have been investigated with promising yet unsettled results. Using the most recent extension of the very popular Defects4J bug dataset, we conduct an empirical study using ARJA and ARJA-e, two state-of-the-art search-based APR systems using a Boolean and a non-Boolean fitness function, respectively. We aim to both extend previous results using new bugs from Defects4J v2.0 and to settle whether refining the fitness function helps fixing bugs present in large software.

In our experiments using 151 non-deprecated and not previously evaluated bugs from Defects4J v2.0, ARJA was able to find patches for 6.6% (\(\frac{10}{151}\)) of bugs, whereas ARJA-e found patches for 8% (\(\frac{12}{151}\)) of bugs. We thus observe only a small advantage in using the refined fitness function. This contrasts with the previous work using Defects4J v1.0.1 where ARJA was able to find adequate patches for 26.3% (\(\frac{59}{224}\)) of the bugs and ARJA-e for 47.3% (\(\frac{106}{224}\)). These results may indicate a potential overfitting of the tools towards the previous version of the Defects4J dataset.


  • Search-based automated program repair
  • Empirical study
  • Software engineering

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-88106-1_12
  • Chapter length: 7 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-88106-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Fig. 1.


  1. 1.

  2. 2.


  1. Bian, Z., Blot, A., Petke, J.: Refining fitness functions for search-based program repair. In: ICSE Workshops (2021)

    Google Scholar 

  2. Gazzola, L., Micucci, D., Mariani, L.: Automatic software repair: a survey. IEEE Trans. Softw. Eng. 45(1), 34–67 (2019)

    CrossRef  Google Scholar 

  3. Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: ISSTA, pp. 437–440 (2014)

    Google Scholar 

  4. Kechagia, M., Mechtaev, S., Sarro, F., Harman, M.: Evaluating automatic program repair capabilities to repair API misuses. IEEE Trans. Softw. Eng. (2021)

    Google Scholar 

  5. Le Goues, C., Nguyen, T., Forrest, S., Weimer, W.: GenProg: a generic method for automatic software repair. IEEE Trans. Softw. Eng. 38(1), 54–72 (2012)

    CrossRef  Google Scholar 

  6. Monperrus, M.: The living review on automated program repair. Technical report, hal-01956501, HAL (2018)

    Google Scholar 

  7. Motwani, M., Brun, Y.: Automatically repairing programs using both tests and bug reports. CoRR abs/2011.08340 (2020)

    Google Scholar 

  8. Smith, E.K., Barr, E.T., Goues, C.L., Brun, Y.: Is the cure worse than the disease? Overfitting in automated program repair. In: SIGSOFT FSE, pp. 532–543 (2015)

    Google Scholar 

  9. de Souza, E.F., Le Goues, C., Camilo-Junior, C.G.: A novel fitness function for automated program repair based on source code checkpoints. In: GECCO, pp. 1443–1450 (2018)

    Google Scholar 

  10. Trujillo, L., Villanueva, O.M., Hernandez, D.E.: A novel approach for search-based program repair. IEEE Softw. 38(4), 36–42 (2021)

    CrossRef  Google Scholar 

  11. Yuan, Y., Banzhaf, W.: ARJA: automated repair of java programs via multi-objective genetic programming. IEEE Trans. Softw. Eng. 46(10), 1040–1067 (2020)

    CrossRef  Google Scholar 

  12. Yuan, Y., Banzhaf, W.: Toward better evolutionary program repair: an integrated approach. ACM Trans. Softw. Eng. Methodol. 29(1), 5:1–5:53 (2020)

    Google Scholar 

  13. Zhu, Q., et al.: A syntax-guided edit decoder for neural program repair. In: ESEC/FSE (2021)

    Google Scholar 

Download references


Funded by ERC 741278 and EPSRC EP/P023991/1.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Giovani Guizzo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Guizzo, G., Blot, A., Callan, J., Petke, J., Sarro, F. (2021). Refining Fitness Functions for Search-Based Automated Program Repair. In: O'Reilly, UM., Devroey, X. (eds) Search-Based Software Engineering. SSBSE 2021. Lecture Notes in Computer Science(), vol 12914. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88105-4

  • Online ISBN: 978-3-030-88106-1

  • eBook Packages: Computer ScienceComputer Science (R0)