Computing

pp 1–22 | Cite as

Test case selection using multi-criteria optimization for effective fault localization

Article
  • 30 Downloads

Abstract

As spectra-based fault localization techniques report suspicious statements by analyzing the coverage of test cases, the effectiveness of the results is highly dependent on the composition of test suites. This paper proposes an approach for selecting a subset of the passed test suite when a failure revealed by a failed test case. The goal is to obtain a more effective fault localization using a minimal number of test cases than using the originally given large number of test cases. A novelty is that a prioritization criterion and a selection criterion are defined. Different from previous studies, the failed trace is fully considered. The prioritization criterion partitions statements in the failed trace into more suspicious and less suspicious, and then ranks passed test cases by their ability in distinguishing the more suspicious statements from the less suspicious ones. The selection criterion selects the minimal passed test suite which can maximize the number of coverage equivalent classes in the failed trace, so as to distinguish the suspicious statements and meanwhile reduce the size of the test suite. Another novelty is that our approach turns the test case selection into a multi-criteria optimization to make the prioritization and the selection criteria complement each other. This approach was evaluated with 5 fault localization techniques, 8 subject programs and 35,392 test cases. The results show that the fault localization effectiveness can be significantly improved with less than 5% passed test cases. Our approach has advantages over the statement- based and vector-based test suite reduction approaches in both fault localization effectiveness and test suite reduction rate.

Keywords

Fault localization Test case selection Test case prioritization Multi-criteria optimization 

Mathematics Subject Classification

68N99 

Notes

Acknowledgements

This study was supported by the 13th Five-Year National Science and Technology Major Project of China (Grant No. 2017YFC0702204), the National Natural Science Foundation of China (Grant No. 61672191) and Harbin science and technology innovation talents research project (Grant No. 2016RAQXJ013).

References

  1. 1.
    Kochhar PS, Xia X, Lo D, Li S (2016) Practitioners’ expectations on automated fault localization. In: International symposium on software testing and analysis, pp 165–176Google Scholar
  2. 2.
    Wong WE, Gao R, Li Y, Rui A, Wotawa F (2016) A survey on software fault localization. IEEE Trans Softw Eng 42(8):707–740CrossRefGoogle Scholar
  3. 3.
    Keller B (2017) Evaluating and improving fault localization. In: International conference on software engineering, pp 609–620Google Scholar
  4. 4.
    Naish L, Lee HJ, Ramamohanarao K (2011) A model for spectra-based software diagnosis. ACM Trans Softw Eng Methodol 20(3):11CrossRefGoogle Scholar
  5. 5.
    Xie X, Liu Z, Song S, Chen Z, Xuan J, Xu B (2016) Revisit of automatic debugging via human focus-tracking analysis. In: vol 40, no 6, pp 808–819Google Scholar
  6. 6.
    Tang CM, Chan WK, Yu YT, Zhang Z (2017) Accuracy graphs of spectrum-based fault localization formulas. IEEE Trans Reliab 66(99):1–22Google Scholar
  7. 7.
    Xie X, Kuo FC, Chen TY, Yoo S, Harman M (2013) Provably optimal and human-competitive results in SBSE for spectrum based fault localisation. Springer, BerlinCrossRefGoogle Scholar
  8. 8.
    Jones JA, Harrold MJ, Stasko JT (2002) Visualization of test information to assist fault localization. In: Proceedings of the 22rd international conference on software engineering, ICSE , 19–25 May 2002, Orlando, Florida, USA, pp 467–477Google Scholar
  9. 9.
    Abreu R, Zoeteweij P, van Gemund AJC (2007) On the accuracy of spectrum-based fault localization. In: Proceedings testing: academic and industrial conference practice and research techniques-MUTATION TAICPART-MUTATION 2007, pp 89–98Google Scholar
  10. 10.
    Wong WE, Qi Y, Zhao L, Cai KY (2007) Effective fault localization using code coverage. In: Proceedings of the 31st annual international computer software and applications conference COMPSAC 2007, vol 1, pp 449–456Google Scholar
  11. 11.
    Bandyopadhyay A, Ghosh S (2011) Proximity based weighting of test cases to improve spectrum based fault localization. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering (ASE), pp 420–423Google Scholar
  12. 12.
    Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation. In: Sarkar V, Hall MW (eds) Proceedings of the ACM SIGPLAN 2005 conference on programming language design and implementation, Chicago, IL, USA, 12–15 June 2005, pp 15–26Google Scholar
  13. 13.
    Harrold MJ, Gupta R, Soffa ML (1993) A new methodology for controlling the size of test suite. ACM Trans Softw Eng Methodol 2:270–285CrossRefGoogle Scholar
  14. 14.
    Wong WE, Debroy V (2010) Software fault localization. Encycl Softw Eng 1:1147–1156CrossRefGoogle Scholar
  15. 15.
    Renieris M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: 18th IEEE international conference on automated software engineering (ASE 2003), 6–10 Oct 2003, Montreal, Canada, pp 30–39Google Scholar
  16. 16.
    Liu C, Zhang X, Han J (2008) A systematic study of failure proximity. IEEE Trans Softw Eng 34(6):826–843CrossRefGoogle Scholar
  17. 17.
    Abreu R, Zoeteweij P, van Gemund AJC (2006) An evaluation of similarity coefficients for software fault localization. In: 12th IEEE Pacific rim international symposium on dependable computing (PRDC), 18–20 Dec 2006, University of California, Riverside, USA, IEEE Computer Society, pp 39–46Google Scholar
  18. 18.
    Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization. In: Wermelinger M, Gall H (eds) Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering, 2005, Lisbon, Portugal, 5–9 Sept 2005, ACM, pp 286–295Google Scholar
  19. 19.
    Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering, ASE ’05, ACM, New York, NY, USA, pp 273–282Google Scholar
  20. 20.
    Yu Y, Jones JA, Harrold MJ (2008) An empirical study of the effects of test-suite reduction on fault localization. In: Proceedings ACM/IEEE 30th international conference on software engineering ICSE ’08, pp 201–210Google Scholar
  21. 21.
    Jiang B, Zhang Z, Chan WK, Tse TH, Chen TY (2012) How well does test case prioritization integrate with statistical fault localization? Inf Softw Technol 54(7):739–758CrossRefGoogle Scholar
  22. 22.
    Jiang B, Chan WK, Tse TH (2011) On practical adequate test suites for integrated test case prioritization and fault localization. In: Proceedings of the 11th international conference on quality software (QSIC), pp 21–30Google Scholar
  23. 23.
    Jiang B, Zhang Z, Chan WK, Tse TH (2009) Adaptive random test case prioritization. In: ASE, pp 233–244Google Scholar
  24. 24.
    Hao D, Xie T, Zhang L, Wang X, Sun J, Mei H (2010) Test input reduction for result inspection to facilitate fault localization. Autom Softw Eng 17(1):5–31CrossRefGoogle Scholar
  25. 25.
    González-Sanchez A, Piel É, Abreu R, Groß HG, van Gemund AJC (2011) Prioritizing tests for software fault diagnosis. Softw Pract Exp 41(10):1105–1129Google Scholar
  26. 26.
    Hutchins M, Foster H, Goradia T, Ostrand T (1994) Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In: Proceedings of the 16th international conference on software engineering ICSE-16, pp 191–200Google Scholar
  27. 27.
    Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, MahwahMATHGoogle Scholar
  28. 28.
    Hao D, Pan Y, Zhang L, Zhao W, Mei H, Sun J (2005) A similarity-aware approach to testing based fault localization. In: 20th IEEE/ACM international conference on automated software engineering (ASE 2005), 7–11 Nov 2005, Long Beach, CA, USA, pp 291–294Google Scholar
  29. 29.
    Wong WE, Horgan JR, Mathur AP, Pasquini A (1999) Test set size minimization and fault detection effectiveness: a case study in a space application. J Syst Softw 48(2):79–89CrossRefGoogle Scholar
  30. 30.
    Artzi S, Dolby J, Tip F, Pistoia M (2010) Directed test generation for effective fault localization. In: Proceedings of the nineteenth international symposium on software testing and analysis, ISSTA 2010, Trento, Italy, 12–16 July 2010, pp 49–60Google Scholar
  31. 31.
    Masri W, Assi RA (2010) Cleansing test suites from coincidental correctness to enhance fault-localization. In: Third international conference on software testing, verification and validation (ICST), pp 165–174Google Scholar
  32. 32.
    Hao D, Zhang L, Zhong H, Mei H, Sun J (2005) Eliminating harmful redundancy for testing-based fault localization using test suite reduction: an experimental study. In: Proceedings of the 21st IEEE international conference on software maintenance, ICSM’05, pp 683–686Google Scholar
  33. 33.
    Baudry B, Fleurey F, Traon YL (2006) Improving test suites for efficient fault localization. In: ICSE, pp 82–91Google Scholar
  34. 34.
    Wang X, Jiang S, Gao P, Ju X, Wang R, Zhang Y (2017) Cost-effective testing based fault localization with distance based test-suite reduction. Sci China 60(9):92–112Google Scholar
  35. 35.
    Zhang Z, Chan WK, Tse TH, Jiang B, Wang X (2009) Capturing propagation of infected program states. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT international symposium on foundations of software engineering, 2009, Amsterdam, The Netherlands, 24–28 Aug 2009, ACM, pp 43–52Google Scholar
  36. 36.
    Masri W (2010) Fault localization based on information flow coverage. Softw Test Verif Reliab 20(2):121–147CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Austria, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer Science and TechnologyHarbin Institute of TechnologyHarbinChina
  2. 2.School of SoftwareHarbin UniversityHarbinChina

Personalised recommendations