Empirical Software Engineering

, Volume 20, Issue 4, pp 928–967 | Cite as

Fault density, fault types, and spectra-based fault localization

  • Nicholas DiGiuseppeEmail author
  • James A. Jones


This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a phenomenon revealed in earlier studies of coverage-based fault localization that causes faults to obstruct, or interfere, with other faults’ ability to be localized. Previously, it had been asserted that a fault-localization technique’s effectiveness was negatively correlated to the quantity of faults in the program. To investigate these beliefs, we conducted an experiment on six programs consisting of more than 72,000 multiple-fault versions. Our data suggests that the impact of multiple faults exerts a significant, but slight influence on fault-localization effectiveness. In addition, faults were categorized according to four existing fault-taxonomies and found no correlation between fault type and fault-localization interference. In general, even in the presence of many faults, at least one fault was found by fault localization with similar effectiveness. Additionally, our data exhibits that fault-localization interference is prevalent and exerts a meaningful influence that may cause a fault’s localizability to vary greatly. Because almost all real-world software contains multiple faults, these results affect the practical use and understanding of statistical fault-localization techniques.


Debugging Fault localization Fault behavior 



This material is based upon work supported by the National Science Foundation, through Award CCF-1116943 and through Graduate Research Fellowship under Grant No. DGE-0808392.


  1. Abreu R, Zoeteweij P, Van Gemund AJ (2007) On the accuracy of spectrum-based fault localization. In: Testing: academic and industrial conference practice and research techniques-MUTATION, 2007. TAICPART-MUTATION 2007, pp 89–98. IEEEGoogle Scholar
  2. Ali S, Andrews JH, Dhandapani T, Wang W (2009) Evaluating the accuracy of fault localization techniques. In: Proceedings of the 2009 IEEE/ACM international conference on automated software engineering, pp 76–87. IEEE Computer SocietyGoogle Scholar
  3. Arumuga Nainar P, Liblit B (2010) Adaptive bug isolation. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering, vol 1, pp 255–264. ACMGoogle Scholar
  4. Clark S, Cobb J, Kapfhammer GM, Jones JA, Harrold MJ (2011) Localizing sql faults in database applications. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering (ASE), pp 213–222Google Scholar
  5. Cleve H, Zeller A (2005) Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering, pp 342–351. ACMGoogle Scholar
  6. Debroy V, Wong WE (2009) Insights on fault interference for programs with multiple bugs. In: 20th international symposium on software reliability engineering, 2009. ISSRE’09, pp 165–174. IEEEGoogle Scholar
  7. Denmat T, Ducassé M, Ridoux O (2005) Data mining and cross-checking of execution traces: a re-interpretation of jones, harrold and stasko test information. Tech. rep.Google Scholar
  8. Dickinson W, Leon D, Podgurski A (2001) Finding failures by cluster analysis of execution profiles. In: Proceedings of the international conference on software engineering.
  9. Dickinson W, Leon D, Podgurski A (2001) Pursuing failure: the distribution of program failures in a profile space. In: Proceedings of the international symposium on foundations of software engineering .
  10. DiGiuseppe N, Jones JA (2011a) Fault interaction and its repercussions. In: 2011 27th IEEE international conference on software maintenance (ICSM), pp 3–12. IEEEGoogle Scholar
  11. DiGiuseppe N, Jones JA (2011b) On the influence of multiple faults on coverage-based fault localization. In: Proceedings of the 2011 international symposium on software testing and analysis, pp 210– 220. ACMGoogle Scholar
  12. DiGiuseppe N, Jones JA (2012a) Concept-based failure clustering. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, p 29. ACMGoogle Scholar
  13. DiGiuseppe N, Jones JA (2012b) Software behavior and failure clustering: an empirical study of fault causality. In: 2012 IEEE fifth international conference on software testing, verification and validation (ICST), pp 191–200. IEEEGoogle Scholar
  14. Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir Softw Eng 10(4):405–435CrossRefGoogle Scholar
  15. Hayes JH (1994) Testing of object-oriented programming systems (oops): a fault-based approach. In: Object-oriented methodologies and systems, pp 205–220. SpringerGoogle Scholar
  16. Hayes JH, Chemannoor IR, Holbrook EA (2011) Improved code defect detection with fault links. Softw Test Verification and Reliab 21(4):299–325CrossRefGoogle Scholar
  17. Jones JA (2008) Semi-automatic fault localization. Georgia Institute of Technology, Ph.D. thesisGoogle Scholar
  18. Jones JA, Bowring JF, Harrold MJ (2007) Debugging in parallel. In: Proceedings of the 2007 international symposium on software testing and analysis, pp 16–26. ACMGoogle Scholar
  19. Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering, pp 273–282. ACMGoogle Scholar
  20. Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proceedings of the 24th international conference on Software engineering, pp 467–477. ACMGoogle Scholar
  21. Kung DC, Gao J, Kung CH (1998) Testing object-oriented software. Tech. rep.Google Scholar
  22. Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation, pp 15–26. ACMGoogle Scholar
  23. Liu C, Han J (2006) Failure proximity: a fault localization-based approach. In: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, pp 46–56. ACMGoogle Scholar
  24. Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization, pp 286–295. ACMGoogle Scholar
  25. Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol (TOSEM) 5(2):99–118CrossRefGoogle Scholar
  26. Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers? In: Proceedings of the 2011 international symposium on software testing and analysis, pp 199–209. ACMGoogle Scholar
  27. Podgurski A, Leon D, Francis P, Masri W, Minch M, Sun J, Wang B (2003) Automated support for classifying software failure reports. In: Proceedings 25th international conference on software engineering, 2003, pp 465–475. IEEEGoogle Scholar
  28. Renieres M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: Proceedings. 18th IEEE international conference on automated software engineering, 2003, pp 30–39. IEEEGoogle Scholar
  29. Santelices R, Jones JA, Yu Y, Harrold MJ (2009) Lightweight fault-localization using multiple coverage types. In: Proceedings of the 31st international conference on software engineering, ICSE ’09, pp 56–66Google Scholar
  30. Smith M, Robson D (1992) A framework for testing object-oriented programs. J Object-Oriented Program 5(3):45–53Google Scholar
  31. Srivastav M, Singh Y, Gupta C, Chauhan DS (2010) Complexity estimation approach for debugging in parallel. In: 2010 2nd international conference on computer research and development, pp 223–227. IEEEGoogle Scholar
  32. Vessey I (1985) Expertise in debugging computer programs: a process analysis. Int J Man-Machine Stud 23(5):459–494CrossRefGoogle Scholar
  33. Voas J (1992) Pie: a dynamic failure-based technique. IEEE transactions on software engineering 18(8):717–727. doi: 10.1109/32.153381 Google Scholar
  34. Wong WE, Debroy V (2009) A survey of software fault localization. University of Texas at Dallas. Tech. Rep. UTDCS-45-09Google Scholar
  35. Yu Y, Jones JA, Harrold MJ (2008) An empirical study of the effects of test-suite reduction on fault localization. In: Proceedings of the 30th international conference on software engineering, pp 201–210. ACMGoogle Scholar
  36. Zeller A (2002) Isolating cause-effect chains from computer programs. In: Proceedings of the 10th ACM SIGSOFT symposium on foundations of software engineering, pp 1–10. ACMGoogle Scholar
  37. Zeller A (2009) In: why programs fail: a guide to systematic debugging. Morgan KaufmannGoogle Scholar
  38. Zheng AX, Jordan MI, Liblit B, Naik M, Aiken A (2006) Statistical debugging: simultaneous identification of multiple bugs. In: Proceedings of the 23rd international conference on machine learning, pp 1105–1112. ACMGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.University of CaliforniaIrvineUSA

Personalised recommendations