Skip to main content
SpringerLink
Log in

ConsilientSFL: using preferential voting system to generate combinatorial ranking metrics for spectrum-based fault localization

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Spectrum-based fault localization (SFL) techniques have shown considerable effectiveness in localizing software faults. They leverage a ranking metric to automatically assign suspiciousness scores to certain entities in a given faulty program. However, for some programs, the current SFL ranking metrics lose effectiveness. In this paper, we introduce ConsilientSFL that is served to synthesize a new ranking metric for a given program, based on a customized combination of a set of given ranking metrics. ConsilientSFL can be significant since it demonstrates the usage of voting systems into a software engineering task. First, several mutated, faulty versions are generated for a program. Then, the mutated versions are executed with the test data. Next, the effectiveness of each existing ranking metric is computed for each mutated version. After that, for each mutated version, the computed existing metrics are ranked using a preferential voting system. Consequently, several top metrics are chosen based on their ranks across all mutated versions. Finally, the chosen ranking metrics are normalized and synthesized, yielding a new ranking metric. To evaluate ConsilientSFL, we have conducted experiments on 27 subject programs from Code4Bench and Siemens benchmarks. In the experiments, we found that ConsilientSFL outperformed every single ranking metric. In particular, for all programs on average, we have found performance measures recall, precision, f-measure, and percentage of code inspection, to be nearly 7, 9, 12, and 5 percentages larger than using single metrics, respectively. The impact of this work is twofold. First, it can mitigate the issue with the choice and usage of a proper ranking metric for the faulty program at hand. Second, it can help debuggers find more faults with less time and effort, yielding higher quality software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Thung F, Wang S, Lo D, Jiang L (2012) An empirical study of bugs in machine learning systems. In: Proc. 23rd International Symposium on Software Reliability Engineering, pp 271–280

    Google Scholar 

  2. Zhang M, Li Y, Li X, Chen L, Zhang Y, Zhang L, Khurshid S (2019) An empirical study of boosting spectrum-based fault localization via pagerank. IEEE Trans Softw Eng

  3. Wong WE, Gao R, Li Y, Abreu R, Wotawa F (2016) A survey on software fault localization. IEEE Trans Softw Eng 42(8):707–740

    Article  Google Scholar 

  4. Leitao-Junior PS, Freitas DM, Vergilio SR, Camilo-Junior CG, Harrison R (2020) Search-based fault localisation: a systematic mapping study. Inf Softw Technol 123:106295

    Article  Google Scholar 

  5. Mao X, Lei Y, Dai Z, Qi Y, Wang C (2014) Slice-based statistical fault localization. J Syst Softw 89:51–62

    Article  Google Scholar 

  6. Wang Y, Patil H, Pereira C, Lueck G, Gupta R, Neamtiu I (2014) Drdebug: Deterministic replay based cyclic debugging with dynamic slicing. In: Proc. of Annual IEEE/ACM International Symposium on Code Generation and Optimization, pp 98:98–98:108

    Google Scholar 

  7. Xuan J, Monperrus M (2014) Test case purification for improving fault localization. In: Proc. 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp 52–63

    Chapter  Google Scholar 

  8. Liu C, Fei L, Yan X, Han J, Midkiff SP (2006) Statistical debugging: a hypothesis testing-based approach. IEEE Trans Softw Eng 32(10):831–848

    Article  Google Scholar 

  9. Nath A, Domingos P (2016) Learning tractable probabilistic models for fault localization. In: Proc. 13th AAAI Conference on Artificial Intelligence, pp 1294–1301

    Google Scholar 

  10. Xuan J, Monperrus M (2014) Learning to combine multiple ranking metrics for fault localization. In: Proc. IEEE International Conference on Software Maintenance and Evolution, pp 191–200

    Google Scholar 

  11. Zhang S, Zhang C (2014) Software bug localization with markov logic. In: Companion Proc. 36th International Conference on Software Engineering, pp 424–427

    Chapter  Google Scholar 

  12. Tu J, Xie X, Chen TY, Baowen X (2019) On the analysis of spectrum based fault localization using hitting sets. J Syst Softw 147:106–123

    Article  Google Scholar 

  13. Dallmeier V, Lindig C, Zeller A (2005) Lightweight bug localization with ample. In: Proc. 6th International Symposium on Automated Analysis-driven Debugging, pp 99–104

    Chapter  Google Scholar 

  14. Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proc. 24th International Conference on Software Engineering, pp 467–477

    Google Scholar 

  15. Yoo S, Xie X, Kuo FC, Chen TY, Harman M (2014) No pot of gold at the end of program spectrum rainbow: greatest risk evaluation formula does not exist. Tech. Rep. Research note RN/14/14. University College London

  16. Pearson S, Campos J, Just R, Fraser G, Abreu R, Ernst MD, Pang D, Keller B (2017) Evaluating and improving fault localization. In: Proc. 39th International Conference on Software Engineering, pp 609–620

    Google Scholar 

  17. Khalilian A, Baraani-Dastjerdi A, Zamani B (2019) APRSuite: a suite of components and use cases based on categorical decomposition of automatic program repair techniques and tools. Journal of Computer Languages 100927

  18. Zakari A, Lee SP, Abreu R, Ahmed BH, Rasheed RA (2020) Multiple fault localization of software programs: A systematic literature review. Information and Software Technology 124:106312

    Article  Google Scholar 

  19. Assiri FY, Bieman JM (2017) Fault localization for automated program repair: effectiveness, performance, repair correctness. Software Quality Journal 25(1):171–199

    Article  Google Scholar 

  20. Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649–678

    Article  Google Scholar 

  21. Tideman N (2018) "how should votes be cast and counted?." the Oxford handbook of. Public Choice 2:1

    Google Scholar 

  22. Burnett CM, Kogan V (2015) Ballot (and voter) “exhaustion” under instant runoff voting: an examination of four ranked-choice elections. Elect Stud 37:41–49

    Article  Google Scholar 

  23. Kemeny JG (1959) Mathematics without numbers. Daedalus 88(4):577–591 retrieved from https://www.mitpressjournals.org/loi/daed

    Google Scholar 

  24. Schulze M (2011) A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election method. Soc Choice Welf 36(2):267–303

    Article  MathSciNet  MATH  Google Scholar 

  25. Bagheri B, Alipour MR, Vahidi-Asl M (2019) An Approach to Generate Effective Fault Localization Methods for Programs. In: International Conference on Fundamentals of Software Engineering. Springer, Cham, pp 244–259

    Chapter  Google Scholar 

  26. Khalilian A, Baraani-Dastjerdi A, Zamani B (2021) CGenProg: adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults. Expert Syst Appl 169:114503

    Article  Google Scholar 

  27. Jeffrey D (2006) Regression Testing. In: CSc 620. Slide 5–7

    Google Scholar 

  28. Huang R, Zhang Q, Towey D, Sun W, Chen J (2020) Regression test case prioritization by code combinations coverage. J Syst Softw 169:110712

    Article  Google Scholar 

  29. Gough B, Stallman RM (2004) An Introduction to GCC for the GNU compilers gcc and g++. Network Theory Ltd

    Google Scholar 

  30. Gazzola L, Micucci D, Mariani L (2017) Automatic software repair: a survey. IEEE Trans Softw Eng 45(1):34–67

    Article  Google Scholar 

  31. Zou D, Liang J, Xiong Y, Ernst MD, Lu Z (2019) An empirical study of fault localization families and their combinations. IEEE Trans Softw Eng

  32. Chekam TT, Papadakis M, Bissyandé TF, Le Traon Y, Sen K (2020) Selecting fault revealing mutants. Empirical Software Engineering 25(1):434–487

    Article  Google Scholar 

  33. Saha RK, Lyu Y, Yoshida H, Prasad MR (2017) Elixir: Effective objectoriented program repair. In: Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering, pp 648–659

    Google Scholar 

  34. Toplak J (2017) Preferential voting: definition and classification. Lex Localis 15(4):737–761

    Article  Google Scholar 

  35. Blom M, Stuckey PJ, Teague VJ (2019) Toward computing the margin of victory in single transferable vote elections. INFORMS J Comput 31(4):636–653

    Article  MathSciNet  MATH  Google Scholar 

  36. Xie X, Chen TY, Kuo FC, Xu B (2013) A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering and Methodology 22(4):31:1–31:40

    Article  Google Scholar 

  37. Ghosh D, Singh J (2021) Spectrum-based multi-fault localization using chaotic genetic algorithm. Inf Softw Technol 133:106512

    Article  Google Scholar 

  38. Jones JA, Bowring JF, Harrold MJ (2007) Debugging in parallel. In: Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, pp 16–1l

    Google Scholar 

  39. Ali S, Andrews JH, Dhandapani T, Wang W (2009) Evaluating the accuracy of fault localization techniques. In: Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering, pp 76–87

    Google Scholar 

  40. Tien-Duy B, Le T, Ferdian LD (2013) Theory and practice, do they match? a case with spectrum-based fault localization. In: 29th IEEE International Conference on Software Maintenance (ICSM)

    Google Scholar 

  41. Debroy V, Wong WE (2011) On the consensus-based application of fault localization techniques. In: Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops, pp 506–511

    Chapter  Google Scholar 

  42. Yoo S (2012) Evolving human competitive spectra-based fault localisation techniques. In: Search Based Software Engineering, volume 7515 of Lecture Notes in Computer Science. Springer, pp 244–258

    Google Scholar 

  43. Le TB, Oentaryo RJ, Lo D (2015) Information retrieval and spectrum based bug localization: Better together. In: Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp 579–590

    Google Scholar 

  44. Koohzadi M, Charkari NM, Ghaderi F (2020) Unsupervised representation learning based on the deep multi-view ensemble learning. Appl Intell 50:562–581

    Article  Google Scholar 

  45. Lopez-Garcia P, Masegosa AD, Osaba E et al (2019) Ensemble classification for imbalanced data based on feature space partitioning and hybrid metaheuristics. Appl Intell 49:2807–2822

    Article  Google Scholar 

  46. Zhang Y, Zhu R, Chen Z et al (2020) Evaluating and selecting features via lower bounds of feature inner correlations for high-dimensional data. Eur J Oper Res. https://doi.org/10.1016/j.ejor.2020.09.028

  47. Xie X, Kuo FC, Chen TY, Yoo S, Harman M (2013) Provably optimal and human competitive results in sbse for spectrum based fault localization. In: Proc. 5th International Symposium on Search Based Software Engineering - Volume 8084, pp 224–238

    Google Scholar 

  48. Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir Softw Eng 10(4):405–435

    Article  Google Scholar 

  49. Majd A, Vahidi-Asl M, Khalilian A, Baraani-Dastjerdi A, Zamani B (2019) Code4Bench: A multidimensional benchmark of codeforces data for different program analysis techniques. Journal of Computer Languages 53:38–52

    Article  Google Scholar 

  50. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media

    Book  MATH  Google Scholar 

  51. Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers? In: Proc. International Symposium on Software Testing and Analysis, pp 199–209

    Google Scholar 

  52. Naish L, Lee HJ, Ramamohanarao K (2011) A model for spectra-based software diagnosis. ACM Transactions on Software Engineering and Methodology (TOSEM) 20(3):1–32

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to the anonymous reviewers whose comments have led to a significant improvement of the paper.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran

    Amirabbas Majd, Mojtaba Vahidi-Asl & Babak Bagheri

  2. Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran

    Alireza Khalilian

Authors
  1. Amirabbas Majd
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mojtaba Vahidi-Asl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alireza Khalilian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Babak Bagheri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mojtaba Vahidi-Asl.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Majd, A., Vahidi-Asl, M., Khalilian, A. et al. ConsilientSFL: using preferential voting system to generate combinatorial ranking metrics for spectrum-based fault localization. Appl Intell 52, 11068–11088 (2022). https://doi.org/10.1007/s10489-021-02954-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02954-7

Keywords

Search

Navigation