Skip to main content
SpringerLink
Log in

Test case selection using multi-criteria optimization for effective fault localization

  • Published:
Computing Aims and scope Submit manuscript

Abstract

As spectra-based fault localization techniques report suspicious statements by analyzing the coverage of test cases, the effectiveness of the results is highly dependent on the composition of test suites. This paper proposes an approach for selecting a subset of the passed test suite when a failure revealed by a failed test case. The goal is to obtain a more effective fault localization using a minimal number of test cases than using the originally given large number of test cases. A novelty is that a prioritization criterion and a selection criterion are defined. Different from previous studies, the failed trace is fully considered. The prioritization criterion partitions statements in the failed trace into more suspicious and less suspicious, and then ranks passed test cases by their ability in distinguishing the more suspicious statements from the less suspicious ones. The selection criterion selects the minimal passed test suite which can maximize the number of coverage equivalent classes in the failed trace, so as to distinguish the suspicious statements and meanwhile reduce the size of the test suite. Another novelty is that our approach turns the test case selection into a multi-criteria optimization to make the prioritization and the selection criteria complement each other. This approach was evaluated with 5 fault localization techniques, 8 subject programs and 35,392 test cases. The results show that the fault localization effectiveness can be significantly improved with less than 5% passed test cases. Our approach has advantages over the statement- based and vector-based test suite reduction approaches in both fault localization effectiveness and test suite reduction rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Kochhar PS, Xia X, Lo D, Li S (2016) Practitioners’ expectations on automated fault localization. In: International symposium on software testing and analysis, pp 165–176

  2. Wong WE, Gao R, Li Y, Rui A, Wotawa F (2016) A survey on software fault localization. IEEE Trans Softw Eng 42(8):707–740

    Article  Google Scholar 

  3. Keller B (2017) Evaluating and improving fault localization. In: International conference on software engineering, pp 609–620

  4. Naish L, Lee HJ, Ramamohanarao K (2011) A model for spectra-based software diagnosis. ACM Trans Softw Eng Methodol 20(3):11

    Article  Google Scholar 

  5. Xie X, Liu Z, Song S, Chen Z, Xuan J, Xu B (2016) Revisit of automatic debugging via human focus-tracking analysis. In: vol 40, no 6, pp 808–819

  6. Tang CM, Chan WK, Yu YT, Zhang Z (2017) Accuracy graphs of spectrum-based fault localization formulas. IEEE Trans Reliab 66(99):1–22

    Google Scholar 

  7. Xie X, Kuo FC, Chen TY, Yoo S, Harman M (2013) Provably optimal and human-competitive results in SBSE for spectrum based fault localisation. Springer, Berlin

    Book  Google Scholar 

  8. Jones JA, Harrold MJ, Stasko JT (2002) Visualization of test information to assist fault localization. In: Proceedings of the 22rd international conference on software engineering, ICSE , 19–25 May 2002, Orlando, Florida, USA, pp 467–477

  9. Abreu R, Zoeteweij P, van Gemund AJC (2007) On the accuracy of spectrum-based fault localization. In: Proceedings testing: academic and industrial conference practice and research techniques-MUTATION TAICPART-MUTATION 2007, pp 89–98

  10. Wong WE, Qi Y, Zhao L, Cai KY (2007) Effective fault localization using code coverage. In: Proceedings of the 31st annual international computer software and applications conference COMPSAC 2007, vol 1, pp 449–456

  11. Bandyopadhyay A, Ghosh S (2011) Proximity based weighting of test cases to improve spectrum based fault localization. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering (ASE), pp 420–423

  12. Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation. In: Sarkar V, Hall MW (eds) Proceedings of the ACM SIGPLAN 2005 conference on programming language design and implementation, Chicago, IL, USA, 12–15 June 2005, pp 15–26

  13. Harrold MJ, Gupta R, Soffa ML (1993) A new methodology for controlling the size of test suite. ACM Trans Softw Eng Methodol 2:270–285

    Article  Google Scholar 

  14. Wong WE, Debroy V (2010) Software fault localization. Encycl Softw Eng 1:1147–1156

    Google Scholar 

  15. Renieris M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: 18th IEEE international conference on automated software engineering (ASE 2003), 6–10 Oct 2003, Montreal, Canada, pp 30–39

  16. Liu C, Zhang X, Han J (2008) A systematic study of failure proximity. IEEE Trans Softw Eng 34(6):826–843

    Article  Google Scholar 

  17. Abreu R, Zoeteweij P, van Gemund AJC (2006) An evaluation of similarity coefficients for software fault localization. In: 12th IEEE Pacific rim international symposium on dependable computing (PRDC), 18–20 Dec 2006, University of California, Riverside, USA, IEEE Computer Society, pp 39–46

  18. Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization. In: Wermelinger M, Gall H (eds) Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering, 2005, Lisbon, Portugal, 5–9 Sept 2005, ACM, pp 286–295

  19. Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering, ASE ’05, ACM, New York, NY, USA, pp 273–282

  20. Yu Y, Jones JA, Harrold MJ (2008) An empirical study of the effects of test-suite reduction on fault localization. In: Proceedings ACM/IEEE 30th international conference on software engineering ICSE ’08, pp 201–210

  21. Jiang B, Zhang Z, Chan WK, Tse TH, Chen TY (2012) How well does test case prioritization integrate with statistical fault localization? Inf Softw Technol 54(7):739–758

    Article  Google Scholar 

  22. Jiang B, Chan WK, Tse TH (2011) On practical adequate test suites for integrated test case prioritization and fault localization. In: Proceedings of the 11th international conference on quality software (QSIC), pp 21–30

  23. Jiang B, Zhang Z, Chan WK, Tse TH (2009) Adaptive random test case prioritization. In: ASE, pp 233–244

  24. Hao D, Xie T, Zhang L, Wang X, Sun J, Mei H (2010) Test input reduction for result inspection to facilitate fault localization. Autom Softw Eng 17(1):5–31

    Article  Google Scholar 

  25. González-Sanchez A, Piel É, Abreu R, Groß HG, van Gemund AJC (2011) Prioritizing tests for software fault diagnosis. Softw Pract Exp 41(10):1105–1129

    Google Scholar 

  26. Hutchins M, Foster H, Goradia T, Ostrand T (1994) Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In: Proceedings of the 16th international conference on software engineering ICSE-16, pp 191–200

  27. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Mahwah

    MATH  Google Scholar 

  28. Hao D, Pan Y, Zhang L, Zhao W, Mei H, Sun J (2005) A similarity-aware approach to testing based fault localization. In: 20th IEEE/ACM international conference on automated software engineering (ASE 2005), 7–11 Nov 2005, Long Beach, CA, USA, pp 291–294

  29. Wong WE, Horgan JR, Mathur AP, Pasquini A (1999) Test set size minimization and fault detection effectiveness: a case study in a space application. J Syst Softw 48(2):79–89

    Article  Google Scholar 

  30. Artzi S, Dolby J, Tip F, Pistoia M (2010) Directed test generation for effective fault localization. In: Proceedings of the nineteenth international symposium on software testing and analysis, ISSTA 2010, Trento, Italy, 12–16 July 2010, pp 49–60

  31. Masri W, Assi RA (2010) Cleansing test suites from coincidental correctness to enhance fault-localization. In: Third international conference on software testing, verification and validation (ICST), pp 165–174

  32. Hao D, Zhang L, Zhong H, Mei H, Sun J (2005) Eliminating harmful redundancy for testing-based fault localization using test suite reduction: an experimental study. In: Proceedings of the 21st IEEE international conference on software maintenance, ICSM’05, pp 683–686

  33. Baudry B, Fleurey F, Traon YL (2006) Improving test suites for efficient fault localization. In: ICSE, pp 82–91

  34. Wang X, Jiang S, Gao P, Ju X, Wang R, Zhang Y (2017) Cost-effective testing based fault localization with distance based test-suite reduction. Sci China 60(9):92–112

    Google Scholar 

  35. Zhang Z, Chan WK, Tse TH, Jiang B, Wang X (2009) Capturing propagation of infected program states. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT international symposium on foundations of software engineering, 2009, Amsterdam, The Netherlands, 24–28 Aug 2009, ACM, pp 43–52

  36. Masri W (2010) Fault localization based on information flow coverage. Softw Test Verif Reliab 20(2):121–147

    Article  Google Scholar 

Download references

Acknowledgements

This study was supported by the 13th Five-Year National Science and Technology Major Project of China (Grant No. 2017YFC0702204), the National Natural Science Foundation of China (Grant No. 61672191) and Harbin science and technology innovation talents research project (Grant No. 2016RAQXJ013).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Ke-chao Wang, Tian-tian Wang & Xiao-hong Su

  2. School of Software, Harbin University, Harbin, China

    Ke-chao Wang

Authors
  1. Ke-chao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tian-tian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao-hong Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tian-tian Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Kc., Wang, Tt. & Su, Xh. Test case selection using multi-criteria optimization for effective fault localization. Computing 100, 787–808 (2018). https://doi.org/10.1007/s00607-018-0610-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-018-0610-0

Keywords

Mathematics Subject Classification

Search

Navigation