Skip to main content
SpringerLink
Log in

Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

One promising way to improve the accuracy of fault localization based on statistical debugging is to increase diversity among test cases in the underlying test suite. In many practical situations, adding test cases is not a cost-free option because test oracles are developed manually or running test cases is expensive. Hence, we require to have test suites that are both diverse and small to improve debugging. In this paper, we focus on improving fault localization of Simulink models by generating test cases. We identify four test objectives that aim to increase test suite diversity. We use four objectives in a search-based algorithm to generate diversified but small test suites. To further minimize test suite sizes, we develop a prediction model to stop test generation when adding test cases is unlikely to improve fault localization. We evaluate our approach using three industrial subjects. Our results show (1) expanding test suites used for fault localization using any of our four test objectives, even when the expansion is small, can significantly improve the accuracy of fault localization, (2) varying test objectives used to generate the initial test suites for fault localization does not have a significant impact on the fault localization results obtained based on those test suites, and (3) we identify an optimal configuration for prediction models to help stop test generation when it is unlikely to be beneficial. We further show that our optimal prediction model is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Abreu R, Zoeteweij P, Van Gemund AJ (2007) On the accuracy of spectrum-based fault localization. In: Proceedings of testing: academic and industrial conference practice and research techniques-MUTATION. IEEE, pp 89–98

  • Agrawal H, Horgan JR (1990) Dynamic program slicing. ACM SIGPLAN Not 25(6):246–256

    Article  Google Scholar 

  • Aho AV, Sethi R, Ullman JD (2007) Compilers: principles, techniques, and tools. Addison-Wesley, Reading

    MATH  Google Scholar 

  • Allen FE (1970) Control flow analysis. SIGPLAN Notices 5(7):1–19

    Article  Google Scholar 

  • Alshahwan N, Harman M (2014) Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 23rd international symposium on software testing and analysis (ISSTA’14). ACM, pp 181–192

  • Androutsopoulos K, Clark D, Dan H, Hierons RM, Harman M (2014) An analysis of the relationship between conditional entropy and failed error propagation in software testing. In: Proceedings of the 36th international conference on software engineering (ICSE’14). ACM, pp 573–583

  • Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw Test Verif Reliab 24(3):219–250

    Article  Google Scholar 

  • Artzi S, Dolby J, Tip F, Pistoia M (2010) Directed test generation for effective fault localization. In: Proceedings of the 19th international symposium on software testing and analysis (ISSTA’10). ACM, pp 49–60

  • Bagherzadeh M, Hili N, Dingel J (2017) Model-level, platform-independent debugging in the context of the model-driven development of real-time systems. In: Proceedings of the 11th joint meeting on foundations of software engineering (ESEC/FSE’17). ACM, pp 419–430

  • Balasubramanian D, Pasareanu CS, Whalen MW, Karsai G, Lowry M (2011) Polyglot: modeling and analysis for multiple statechart formalisms. In: Proceedings of the 20th international symposium on software testing and analysis (ISSTA’11). ACM, pp 45–55

  • Baudry B, Fleurey F, Le Traon Y (2006) Improving test suites for efficient fault localization. In: Proceedings of the 28th international conference on software engineering (ICSE’06). ACM, pp 82–91

  • Ben Abdessalem R, Nejati S, Briand L, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. In: Proceedings of the 31st international conference on automated software engineering (ASE’16). ACM, pp 63–74

  • Binder RV (2000) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Professional, Reading

    Google Scholar 

  • Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classification and regression trees. CRC Press, Boca Raton

    MATH  Google Scholar 

  • Briand L, Nejati S, Sabetzadeh M, Bianculli D (2016) Testing the untestable: model testing of complex software-intensive systems. In: Companion proceedings of the 38th international conference on software engineering, (ICSE’16), pp 789–792

  • Brillout A, He N, Mazzucchi M, Kroening D, Purandare M, Rümmer P., Weissenbacher G (2009) Mutation-based test case generation for simulink models. In: International symposium on formal methods for components and objects. Springer, pp 208–227

  • Campos J, Abreu R, Fraser G, d’Amorim M (2013) Entropy-based test generation for improved fault localization. In: Proceedings of the 28th international conference on automated software engineering (ASE’13). IEEE, pp 257–267

  • Chen TY, Leung H, Mak I (2004) Adaptive random testing. In: Proceedings of the 9th Asian computing science conference, pp 320–329

  • Clark D, Feldt R, Poulding S, Yoo S (2015) Information transformation: an underpinning theory for software engineering. In: Proceedings of the 37th international conference on software engineering (ICSE’15), vol 2. IEEE, pp 599–602

  • Cleve H, Zeller A (2005) Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering (ICSE’05). ACM, pp 342–351

  • Fagin R, Kumar R, Sivakumar D (2003) Comparing top k lists. SIAM J Discret Math 17(1):134–160

    Article  MathSciNet  MATH  Google Scholar 

  • Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Trans Softw Eng (TSE) 39(2):276–291

    Article  Google Scholar 

  • Gay G, Rajan A, Staats M, Whalen MW, Heimdahl MPE (2016) The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage. ACM Trans Softw Eng Methodol (TOSEM) 25(3):25:1–25:34

    Article  Google Scholar 

  • Gerlitz T, Kowalewski S (2016) Flow sensitive slicing for MATLAB/Simulink models. In: Proceedings of the 13th working IEEE/IFIP conference on software architecture. IEEE, pp 81–90

  • Gold NE, Binkley D, Harman M, Islam S, Krinke J, Yoo S (2017) Generalized observational slicing for tree-represented modelling languages. In: Proceedings of the 11th joint meeting on foundations of software engineering (ESEC/FSE’17), pp 547–558

  • Hamon G (2008) Simulink Design Verifier-Applying automated formal methods to Simulink and StateFlow. In: Proceedings of the 3rd workshop on automated formal methods

  • Hanh LTM, Binh NT (2012) Mutation operators for Simulink models. In: Proceedings of the 4th international conference on knowledge and systems engineering. IEEE, pp 54–59

  • He N, Rümmer P., Kroening D (2011) Test-case generation for embedded Simulink via formal concept analysis. In: Proceedings of the 48th design automation conference. ACM, pp 224–229

  • Jaccard P (1901) Etude comparative de la distribution florale dans une portion des Alpes et du Jura. Impr. Corbaz

  • Jiang B, Zhang Z, Chan WK, Tse T (2009) Adaptive random test case prioritization. In: Proceedings of the 24th international conference on automated software engineering (ASE’09). IEEE, pp 233–244

  • Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th international conference on automated software engineering (ASE’05). ACM, pp 273–282

  • Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proceedings of the 24th international conference on software engineering (ICSE’02). ACM, pp 467–477

  • Jost L (2006) Entropy and diversity. Oikos 113(2):363–375

    Article  Google Scholar 

  • Kendall MG (1938) A new measure of rank correlation. Biometrika 30:81–93

    Article  MATH  Google Scholar 

  • Kendall MG (1948) Rank correlation methods. Griffin

  • Le TDB, Lo D (2013) Will fault localization work for these failures? An automated approach to predict effectiveness of fault localization tools. In: Proceeding of the 29th international conference on software maintenance (ICSM’13). IEEE, pp 310–319

  • Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation. ACM Spec Interest Group Programm Lang Not 40(6):15–26

    Google Scholar 

  • Liu B (2017) Experiments data. https://github.com/Avartar/TCGenForFL/

  • Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization. ACM SIGSOFT Softw Eng Notes 30(5):286–295

    Article  Google Scholar 

  • Liu B, Lucia, Nejati S, Briand L, Bruckmann T (2016a) Localizing multiple faults in Simulink models. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER’16). IEEE, pp 146–156

  • Liu B, Lucia, Nejati S, Briand L, Bruckmann T (2016b) Simulink fault localization: an iterative statistical debugging approach. Softw Test Verif Reliab J 431–459

  • Liu B, Lucia, Nejati S, Briand LC (2017) Improving fault localization for Simulink models using search-based testing and prediction models. In: Proceedings of the 24th international conference on software analysis, evolution and reengineering (SANER’17). IEEE, pp 359–370

  • Lucia, Lo D, Xia X (2014) Fusion fault localizers. In: Proceedings of the 29th international conference on automated software engineering (ASE’14). ACM, pp 127–138

  • Luke S (2009) Essentials of metaheuristics, vol 113. Lulu, Raleigh

  • Mao X, Lei Y, Dai Z, Qi Y, Wang C (2014) Slice-based statistical fault localization. J Syst Softw 89:51–62

    Article  Google Scholar 

  • Masri W, Assi RA (2014) Prevalence of coincidental correctness and mitigation of its impact on fault localization. ACM Trans Softw Eng Methodol (TOSEM) 23 (1):8:1–8:28

    Article  Google Scholar 

  • Mathur AP (2006) Foundations of software testing. Copymat Services

  • MathWorks (2017) Simulink. http://www.mathworks.nl/products/simulink/

  • MathWorks (2017) StateFlow. http://www.mathworks.nl/products/stateflow/

  • Matinnejad R, Nejati S, Briand L, Bruckmann T, Poull C (2015a) Search-based automated testing of continuous controllers: framework, tool support, and case studies. Inf Softw Technol J 57:705–722

    Article  Google Scholar 

  • Matinnejad R, Nejati S, Briand L, Bruckmann T (2015b) Effective test suites for mixed discrete-continuous StateFlow controllers. In: Proceedings of the 10th joint meeting on foundations of software engineering (ESEC/FSE’15). ACM, pp 84–95

  • Matinnejad R, Nejati S, Briand L, Bruckmann T (2016) Automated test suite generation for time-continuous Simulink models. In: Proceedings of the 38th international conference on software engineering (ICSE’16). ACM, pp 595–606

  • Mohalik S, Gadkari AA, Yeolekar A, Shashidhar K, Ramesh S (2014) Automatic test case generation from Simulink/StateFlow models using model checking. Softw Test Verif Reliab J 24(2):155–180

    Article  Google Scholar 

  • Pantelic V, Postma SM, Lawford M, Korobkine A, Mackenzie B, Ong J, Bender M (2015) A toolset for Simulink - improving software engineering practices in development with Simulink. In: Proceedings of the 3rd international conference on model-driven engineering and software development (MODELS’15). SciTePress, pp 50–61

  • Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers?. In: Proceedings of the 20th international symposium on software testing and analysis (ISSTA’11). ACM, pp 199–209

  • Pasareanu CS, Schumann J, Mehlitz P, Lowry M, Karsai G, Nine H, Neema S (2009) Model based analysis and test generation for flight software. In: Proceedings of the 3rd international conference on space mission challenges for information technology. IEEE, pp 83–90

  • Perez A, Abreu R, van Deursen A (2017) A test-suite diagnosability metric for spectrum-based fault localization approaches. In: Proceedings of the 39th international conference on software engineering (ICSE’17). IEEE, pp 654–664

  • Rapos EJ, Cordy JR (2016) Examining the co-evolution relationship between Simulink models and their test cases. In: Proceedings of the 8th international workshop on modeling in software engineering. ACM, pp 34–40

  • Reicherdt R, Glesner S (2012) Slicing MATLAB Simulink models. In: Proceedings of the 34th international conference on software engineering (ICSE’12), pp 551–561

  • Renieris M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: Proceedings of the 18th international conference on automated software engineering (ASE’03). IEEE, pp 30–39

  • Röszler J, Fraser G, Zeller A, Orso A (2012) Isolating failure causes through test case generation. In: Proceedings of the 21st international symposium on software testing and analysis (ISSTA’12). ACM, pp 309–319

  • Santelices R, Jones JA, Yu Y, Harrold MJ (2009) Lightweight fault-localization using multiple coverage types. In: Proceedings of the 31st international conference on software engineering (ICSE’09). IEEE, pp 56–66

  • Satpathy M, Yeolekar A, Peranandam P, Ramesh S (2012) Efficient coverage of parallel and hierarchical StateFlow models for test case generation. Softw Test Verif Reliab J 22(7):457–479

    Article  Google Scholar 

  • Schneider J (2014) Tracking down root causes of defects in Simulink models. In: Proceedings of the 29th international conference on automated software engineering (ASE’14). ACM, pp 599–604

  • Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101

    Article  Google Scholar 

  • Sridhar A, Srinivasulu D (2014) Slicing MATLAB Simulink/StateFlow models. In: Intelligent computing, networking, and informatics. Springer, pp 737–743

  • Thums A, Quante J (2012) Reengineering embedded automotive software. In: Proceedings of the 28th international conference on software maintenance (ICSM’12). IEEE, pp 493–502

  • Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132

    Google Scholar 

  • Varrette S, Bouvry P, Cartiaux H, Georgatos F (2014) Management of an academic HPC cluster: the UL experience. In: Proceedings of the international conference on high performance computing & simulation. IEEE, pp 959–967

  • Wang X, Cheung SC, Chan WK, Zhang Z (2009) Taming coincidental correctness: coverage refinement with context patterns to improve fault localization. In: Proceedings of the 31st international conference on software engineering (ICSE’09). IEEE Computer Society, pp 45–55

  • Windisch A (2009) Search-based testing of complex Simulink models containing StateFlow diagrams. In: Proceeding of the 31st international conference on software engineering-companion. IEEE, pp 395–398

  • Windisch A (2010) Search-based test data generation from StateFlow StateCharts. In: Proceedings of the 12th annual conference on genetic and evolutionary computation. ACM, pp 1349–1356

  • Wong E, Wei T, Qi Y, Zhao L (2008) A crosstab-based statistical method for effective fault localization. In: Proceedings of the 1st international conference on software testing, verification, and validation (ICST’08). IEEE, pp 42–51

  • Wong E, Debroy V, Gao R, Li Y (2014) The DStar method for effective software fault localization. IEEE Trans Reliab 63(1):290–308

    Article  Google Scholar 

  • Xia X, Gong L, Le TDB, Lo D, Jiang L, Zhang H (2016) Diversity maximization speedup for localizing faults in single-fault and multi-fault programs. Autom Softw Eng J 23(1):43–75

    Article  Google Scholar 

  • Yin YF, Zhou YB, Wang YR (2014) Research and improvements on mutation operators for Simulink models. In: Applied mechanics and materials, vol 687. Trans Tech Publ, pp 1389–1393

  • Zander J, Schieferdecker I, Mosterman PJ (2011) Model-based testing for embedded systems. CRC Press, Boca Raton

  • Zhan Y, Clark J (2004) Search based automatic test-data generation at an architectural level. In: Proceedings of the 6th genetic and evolutionary computation conference. Springer, pp 1413–1424

  • Zhan Y, Clark JA (2005) Search-based mutation testing for Simulink models. In: Proceedings of the 7th genetic and evolutionary computation conference. ACM, pp 1061–1068

  • Zhan Y, Clark JA (2008) A search-based framework for automatic testing of Matlab/Simulink models. J Syst Softw 81(2):262–285

    Article  Google Scholar 

  • Zheng X, Julien C, Kim M, Khurshid S (2017) Perceptions on the state of the art in verification and validation in cyber-physical systems. IEEE Syst J 11:2614–2627

    Article  Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 694277).

Author information

Authors and Affiliations

  1. SnT Centre, University of Luxembourg, Luxembourg, Luxembourg

    Bing Liu, Shiva Nejati,  Lucia & Lionel C. Briand

Authors
  1. Bing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiva Nejati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lucia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lionel C. Briand
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiva Nejati.

Additional information

Communicated by: Gabriele Bavota and Andrian Marcus

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, B., Nejati, S., Lucia et al. Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy. Empir Software Eng 24, 444–490 (2019). https://doi.org/10.1007/s10664-018-9611-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9611-z

Keywords

Search

Navigation