Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy

Article
  • 26 Downloads

Abstract

One promising way to improve the accuracy of fault localization based on statistical debugging is to increase diversity among test cases in the underlying test suite. In many practical situations, adding test cases is not a cost-free option because test oracles are developed manually or running test cases is expensive. Hence, we require to have test suites that are both diverse and small to improve debugging. In this paper, we focus on improving fault localization of Simulink models by generating test cases. We identify four test objectives that aim to increase test suite diversity. We use four objectives in a search-based algorithm to generate diversified but small test suites. To further minimize test suite sizes, we develop a prediction model to stop test generation when adding test cases is unlikely to improve fault localization. We evaluate our approach using three industrial subjects. Our results show (1) expanding test suites used for fault localization using any of our four test objectives, even when the expansion is small, can significantly improve the accuracy of fault localization, (2) varying test objectives used to generate the initial test suites for fault localization does not have a significant impact on the fault localization results obtained based on those test suites, and (3) we identify an optimal configuration for prediction models to help stop test generation when it is unlikely to be beneficial. We further show that our optimal prediction model is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.

Keywords

Fault localization Simulink models Search-based testing Test suite diversity Supervised learning 

Notes

Acknowledgements

We gratefully acknowledge funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 694277).

References

  1. Abreu R, Zoeteweij P, Van Gemund AJ (2007) On the accuracy of spectrum-based fault localization. In: Proceedings of testing: academic and industrial conference practice and research techniques-MUTATION. IEEE, pp 89–98Google Scholar
  2. Agrawal H, Horgan JR (1990) Dynamic program slicing. ACM SIGPLAN Not 25(6):246–256CrossRefGoogle Scholar
  3. Aho AV, Sethi R, Ullman JD (2007) Compilers: principles, techniques, and tools. Addison-Wesley, ReadingMATHGoogle Scholar
  4. Allen FE (1970) Control flow analysis. SIGPLAN Notices 5(7):1–19CrossRefGoogle Scholar
  5. Alshahwan N, Harman M (2014) Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 23rd international symposium on software testing and analysis (ISSTA’14). ACM, pp 181–192Google Scholar
  6. Androutsopoulos K, Clark D, Dan H, Hierons RM, Harman M (2014) An analysis of the relationship between conditional entropy and failed error propagation in software testing. In: Proceedings of the 36th international conference on software engineering (ICSE’14). ACM, pp 573–583Google Scholar
  7. Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw Test Verif Reliab 24(3):219–250CrossRefGoogle Scholar
  8. Artzi S, Dolby J, Tip F, Pistoia M (2010) Directed test generation for effective fault localization. In: Proceedings of the 19th international symposium on software testing and analysis (ISSTA’10). ACM, pp 49–60Google Scholar
  9. Bagherzadeh M, Hili N, Dingel J (2017) Model-level, platform-independent debugging in the context of the model-driven development of real-time systems. In: Proceedings of the 11th joint meeting on foundations of software engineering (ESEC/FSE’17). ACM, pp 419–430Google Scholar
  10. Balasubramanian D, Pasareanu CS, Whalen MW, Karsai G, Lowry M (2011) Polyglot: modeling and analysis for multiple statechart formalisms. In: Proceedings of the 20th international symposium on software testing and analysis (ISSTA’11). ACM, pp 45–55Google Scholar
  11. Baudry B, Fleurey F, Le Traon Y (2006) Improving test suites for efficient fault localization. In: Proceedings of the 28th international conference on software engineering (ICSE’06). ACM, pp 82–91Google Scholar
  12. Ben Abdessalem R, Nejati S, Briand L, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. In: Proceedings of the 31st international conference on automated software engineering (ASE’16). ACM, pp 63–74Google Scholar
  13. Binder RV (2000) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Professional, ReadingGoogle Scholar
  14. Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classification and regression trees. CRC Press, Boca RatonMATHGoogle Scholar
  15. Briand L, Nejati S, Sabetzadeh M, Bianculli D (2016) Testing the untestable: model testing of complex software-intensive systems. In: Companion proceedings of the 38th international conference on software engineering, (ICSE’16), pp 789–792Google Scholar
  16. Brillout A, He N, Mazzucchi M, Kroening D, Purandare M, Rümmer P., Weissenbacher G (2009) Mutation-based test case generation for simulink models. In: International symposium on formal methods for components and objects. Springer, pp 208–227Google Scholar
  17. Campos J, Abreu R, Fraser G, d’Amorim M (2013) Entropy-based test generation for improved fault localization. In: Proceedings of the 28th international conference on automated software engineering (ASE’13). IEEE, pp 257–267Google Scholar
  18. Chen TY, Leung H, Mak I (2004) Adaptive random testing. In: Proceedings of the 9th Asian computing science conference, pp 320–329Google Scholar
  19. Clark D, Feldt R, Poulding S, Yoo S (2015) Information transformation: an underpinning theory for software engineering. In: Proceedings of the 37th international conference on software engineering (ICSE’15), vol 2. IEEE, pp 599–602Google Scholar
  20. Cleve H, Zeller A (2005) Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering (ICSE’05). ACM, pp 342–351Google Scholar
  21. Fagin R, Kumar R, Sivakumar D (2003) Comparing top k lists. SIAM J Discret Math 17(1):134–160MathSciNetCrossRefMATHGoogle Scholar
  22. Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Trans Softw Eng (TSE) 39(2):276–291CrossRefGoogle Scholar
  23. Gay G, Rajan A, Staats M, Whalen MW, Heimdahl MPE (2016) The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage. ACM Trans Softw Eng Methodol (TOSEM) 25(3):25:1–25:34CrossRefGoogle Scholar
  24. Gerlitz T, Kowalewski S (2016) Flow sensitive slicing for MATLAB/Simulink models. In: Proceedings of the 13th working IEEE/IFIP conference on software architecture. IEEE, pp 81–90Google Scholar
  25. Gold NE, Binkley D, Harman M, Islam S, Krinke J, Yoo S (2017) Generalized observational slicing for tree-represented modelling languages. In: Proceedings of the 11th joint meeting on foundations of software engineering (ESEC/FSE’17), pp 547–558Google Scholar
  26. Hamon G (2008) Simulink Design Verifier-Applying automated formal methods to Simulink and StateFlow. In: Proceedings of the 3rd workshop on automated formal methodsGoogle Scholar
  27. Hanh LTM, Binh NT (2012) Mutation operators for Simulink models. In: Proceedings of the 4th international conference on knowledge and systems engineering. IEEE, pp 54–59Google Scholar
  28. He N, Rümmer P., Kroening D (2011) Test-case generation for embedded Simulink via formal concept analysis. In: Proceedings of the 48th design automation conference. ACM, pp 224–229Google Scholar
  29. Jaccard P (1901) Etude comparative de la distribution florale dans une portion des Alpes et du Jura. Impr. CorbazGoogle Scholar
  30. Jiang B, Zhang Z, Chan WK, Tse T (2009) Adaptive random test case prioritization. In: Proceedings of the 24th international conference on automated software engineering (ASE’09). IEEE, pp 233–244Google Scholar
  31. Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th international conference on automated software engineering (ASE’05). ACM, pp 273–282Google Scholar
  32. Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proceedings of the 24th international conference on software engineering (ICSE’02). ACM, pp 467–477Google Scholar
  33. Jost L (2006) Entropy and diversity. Oikos 113(2):363–375CrossRefGoogle Scholar
  34. Kendall MG (1938) A new measure of rank correlation. Biometrika 30:81–93CrossRefMATHGoogle Scholar
  35. Kendall MG (1948) Rank correlation methods. GriffinGoogle Scholar
  36. Le TDB, Lo D (2013) Will fault localization work for these failures? An automated approach to predict effectiveness of fault localization tools. In: Proceeding of the 29th international conference on software maintenance (ICSM’13). IEEE, pp 310–319Google Scholar
  37. Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation. ACM Spec Interest Group Programm Lang Not 40(6):15–26Google Scholar
  38. Liu B (2017) Experiments data. https://github.com/Avartar/TCGenForFL/
  39. Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization. ACM SIGSOFT Softw Eng Notes 30(5):286–295CrossRefGoogle Scholar
  40. Liu B, Lucia, Nejati S, Briand L, Bruckmann T (2016a) Localizing multiple faults in Simulink models. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER’16). IEEE, pp 146–156Google Scholar
  41. Liu B, Lucia, Nejati S, Briand L, Bruckmann T (2016b) Simulink fault localization: an iterative statistical debugging approach. Softw Test Verif Reliab J 431–459Google Scholar
  42. Liu B, Lucia, Nejati S, Briand LC (2017) Improving fault localization for Simulink models using search-based testing and prediction models. In: Proceedings of the 24th international conference on software analysis, evolution and reengineering (SANER’17). IEEE, pp 359–370Google Scholar
  43. Lucia, Lo D, Xia X (2014) Fusion fault localizers. In: Proceedings of the 29th international conference on automated software engineering (ASE’14). ACM, pp 127–138Google Scholar
  44. Luke S (2009) Essentials of metaheuristics, vol 113. Lulu, RaleighGoogle Scholar
  45. Mao X, Lei Y, Dai Z, Qi Y, Wang C (2014) Slice-based statistical fault localization. J Syst Softw 89:51–62CrossRefGoogle Scholar
  46. Masri W, Assi RA (2014) Prevalence of coincidental correctness and mitigation of its impact on fault localization. ACM Trans Softw Eng Methodol (TOSEM) 23 (1):8:1–8:28CrossRefGoogle Scholar
  47. Mathur AP (2006) Foundations of software testing. Copymat ServicesGoogle Scholar
  48. MathWorks (2017) Simulink. http://www.mathworks.nl/products/simulink/
  49. MathWorks (2017) StateFlow. http://www.mathworks.nl/products/stateflow/
  50. Matinnejad R, Nejati S, Briand L, Bruckmann T, Poull C (2015a) Search-based automated testing of continuous controllers: framework, tool support, and case studies. Inf Softw Technol J 57:705–722CrossRefGoogle Scholar
  51. Matinnejad R, Nejati S, Briand L, Bruckmann T (2015b) Effective test suites for mixed discrete-continuous StateFlow controllers. In: Proceedings of the 10th joint meeting on foundations of software engineering (ESEC/FSE’15). ACM, pp 84–95Google Scholar
  52. Matinnejad R, Nejati S, Briand L, Bruckmann T (2016) Automated test suite generation for time-continuous Simulink models. In: Proceedings of the 38th international conference on software engineering (ICSE’16). ACM, pp 595–606Google Scholar
  53. Mohalik S, Gadkari AA, Yeolekar A, Shashidhar K, Ramesh S (2014) Automatic test case generation from Simulink/StateFlow models using model checking. Softw Test Verif Reliab J 24(2):155–180CrossRefGoogle Scholar
  54. Pantelic V, Postma SM, Lawford M, Korobkine A, Mackenzie B, Ong J, Bender M (2015) A toolset for Simulink - improving software engineering practices in development with Simulink. In: Proceedings of the 3rd international conference on model-driven engineering and software development (MODELS’15). SciTePress, pp 50–61Google Scholar
  55. Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers?. In: Proceedings of the 20th international symposium on software testing and analysis (ISSTA’11). ACM, pp 199–209Google Scholar
  56. Pasareanu CS, Schumann J, Mehlitz P, Lowry M, Karsai G, Nine H, Neema S (2009) Model based analysis and test generation for flight software. In: Proceedings of the 3rd international conference on space mission challenges for information technology. IEEE, pp 83–90Google Scholar
  57. Perez A, Abreu R, van Deursen A (2017) A test-suite diagnosability metric for spectrum-based fault localization approaches. In: Proceedings of the 39th international conference on software engineering (ICSE’17). IEEE, pp 654–664Google Scholar
  58. Rapos EJ, Cordy JR (2016) Examining the co-evolution relationship between Simulink models and their test cases. In: Proceedings of the 8th international workshop on modeling in software engineering. ACM, pp 34–40Google Scholar
  59. Reicherdt R, Glesner S (2012) Slicing MATLAB Simulink models. In: Proceedings of the 34th international conference on software engineering (ICSE’12), pp 551–561Google Scholar
  60. Renieris M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: Proceedings of the 18th international conference on automated software engineering (ASE’03). IEEE, pp 30–39Google Scholar
  61. Röszler J, Fraser G, Zeller A, Orso A (2012) Isolating failure causes through test case generation. In: Proceedings of the 21st international symposium on software testing and analysis (ISSTA’12). ACM, pp 309–319Google Scholar
  62. Santelices R, Jones JA, Yu Y, Harrold MJ (2009) Lightweight fault-localization using multiple coverage types. In: Proceedings of the 31st international conference on software engineering (ICSE’09). IEEE, pp 56–66Google Scholar
  63. Satpathy M, Yeolekar A, Peranandam P, Ramesh S (2012) Efficient coverage of parallel and hierarchical StateFlow models for test case generation. Softw Test Verif Reliab J 22(7):457–479CrossRefGoogle Scholar
  64. Schneider J (2014) Tracking down root causes of defects in Simulink models. In: Proceedings of the 29th international conference on automated software engineering (ASE’14). ACM, pp 599–604Google Scholar
  65. Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101CrossRefGoogle Scholar
  66. Sridhar A, Srinivasulu D (2014) Slicing MATLAB Simulink/StateFlow models. In: Intelligent computing, networking, and informatics. Springer, pp 737–743Google Scholar
  67. Thums A, Quante J (2012) Reengineering embedded automotive software. In: Proceedings of the 28th international conference on software maintenance (ICSM’12). IEEE, pp 493–502Google Scholar
  68. Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132Google Scholar
  69. Varrette S, Bouvry P, Cartiaux H, Georgatos F (2014) Management of an academic HPC cluster: the UL experience. In: Proceedings of the international conference on high performance computing & simulation. IEEE, pp 959–967Google Scholar
  70. Wang X, Cheung SC, Chan WK, Zhang Z (2009) Taming coincidental correctness: coverage refinement with context patterns to improve fault localization. In: Proceedings of the 31st international conference on software engineering (ICSE’09). IEEE Computer Society, pp 45–55Google Scholar
  71. Windisch A (2009) Search-based testing of complex Simulink models containing StateFlow diagrams. In: Proceeding of the 31st international conference on software engineering-companion. IEEE, pp 395–398Google Scholar
  72. Windisch A (2010) Search-based test data generation from StateFlow StateCharts. In: Proceedings of the 12th annual conference on genetic and evolutionary computation. ACM, pp 1349–1356Google Scholar
  73. Wong E, Wei T, Qi Y, Zhao L (2008) A crosstab-based statistical method for effective fault localization. In: Proceedings of the 1st international conference on software testing, verification, and validation (ICST’08). IEEE, pp 42–51Google Scholar
  74. Wong E, Debroy V, Gao R, Li Y (2014) The DStar method for effective software fault localization. IEEE Trans Reliab 63(1):290–308CrossRefGoogle Scholar
  75. Xia X, Gong L, Le TDB, Lo D, Jiang L, Zhang H (2016) Diversity maximization speedup for localizing faults in single-fault and multi-fault programs. Autom Softw Eng J 23(1):43–75CrossRefGoogle Scholar
  76. Yin YF, Zhou YB, Wang YR (2014) Research and improvements on mutation operators for Simulink models. In: Applied mechanics and materials, vol 687. Trans Tech Publ, pp 1389–1393Google Scholar
  77. Zander J, Schieferdecker I, Mosterman PJ (2011) Model-based testing for embedded systems. CRC Press, Boca RatonGoogle Scholar
  78. Zhan Y, Clark J (2004) Search based automatic test-data generation at an architectural level. In: Proceedings of the 6th genetic and evolutionary computation conference. Springer, pp 1413–1424Google Scholar
  79. Zhan Y, Clark JA (2005) Search-based mutation testing for Simulink models. In: Proceedings of the 7th genetic and evolutionary computation conference. ACM, pp 1061–1068Google Scholar
  80. Zhan Y, Clark JA (2008) A search-based framework for automatic testing of Matlab/Simulink models. J Syst Softw 81(2):262–285CrossRefGoogle Scholar
  81. Zheng X, Julien C, Kim M, Khurshid S (2017) Perceptions on the state of the art in verification and validation in cyber-physical systems. IEEE Syst J 11:2614–2627CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Bing Liu
    • 1
  • Shiva Nejati
    • 1
  • Lucia
    • 1
  • Lionel C. Briand
    • 1
  1. 1.SnT CentreUniversity of LuxembourgLuxembourgLuxembourg

Personalised recommendations