Mutation-Based Test Case Generation for Simulink Models

  • Angelo Brillout
  • Nannan He
  • Michele Mazzucchi
  • Daniel Kroening
  • Mitra Purandare
  • Philipp Rümmer
  • Georg Weissenbacher
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6286)


The Matlab/Simulink language has become the standard formalism for modeling and implementing control software in areas like avionics, automotive, railway, and process automation. Such software is often safety critical, and bugs have potentially disastrous consequences for people and material involved. We define a verification methodology to assess the correctness of Simulink programs by means of automated test-case generation. In the style of fault- and mutation-based testing, the coverage of a Simulink program by a test suite is defined in terms of the detection of injected faults. Using bounded model checking techniques, we are able to effectively and automatically compute test suites for given fault models. Several optimisations are discussed to make the approach practical for realistic Simulink programs and fault models, and to obtain accurate coverage measures.


Model Check Test Suite Mutation Testing Equivalence Check Conjunctive Normal Form 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Clarke, E., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge (1999)Google Scholar
  2. 2.
    Gadkari, A., Yeolekar, A., Suresh, J., Ramesh, S., Mohalik, S., Shashidar, K.C.: AutoMOTGen: Automatic model oriented test generator for embedded control systems. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 204–208. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Kroening, D., Clarke, E.M., Yorav, K.: Behavioral consistency of C and Verilog programs using bounded model checking. In: Design Automation Conference (DAC), pp. 368–371. ACM, New York (2003)Google Scholar
  4. 4.
    Holzer, A., Schallhart, C., Tautschnig, M., Veith, H.: FShell: Systematic test case generation for dynamic analysis and measurement. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 209–213. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Angeletti, D., Giunchiglia, E., Narizzano, M., Puddu, A., Sabina, S.: Automatic test generation for coverage analysis using CBMC. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) Computer Aided Systems Theory - EUROCAST 2009. LNCS, vol. 5717, pp. 287–294. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Holzer, A., Schallhart, C., Tautschnig, M., Veith, H.: Query-driven program testing. In: Jones, N.D., Müller-Olm, M. (eds.) VMCAI 2009. LNCS, vol. 5403, pp. 151–166. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Ball, T.: A theory of predicate-complete test coverage and generation. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2004. LNCS, vol. 3657, pp. 1–22. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Beyer, D., Chlipala, A.J., Henzinger, T.A., Jhala, R., Majumdar, R.: Generating tests from counterexamples. In: International Conference on Software Engineering (ICSE), pp. 326–335 (2004)Google Scholar
  9. 9.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, TSE (2010)Google Scholar
  10. 10.
    Kupferman, O., Li, W., Seshia, S.A.: A theory of mutations with applications to vacuity, coverage, and fault tolerance. In: Formal Methods in Computer-Aided Design (FMCAD), pp. 1–9. IEEE, Los Alamitos (2008)Google Scholar
  11. 11.
    Ruthruff, J.R., Burnett, M.M., Rothermel, G.: Interactive fault localization techniques in a spreadsheet environment. IEEE Transactions on Software Engineering (TSE) 32, 213–239 (2006)CrossRefGoogle Scholar
  12. 12.
    Schuler, D., Dallmeier, V., Zeller, A.: Efficient mutation testing by checking invariant violations. In: International Symposium on Software Testing and Analysis (ISSTA), pp. 69–80. ACM, New York (2009)Google Scholar
  13. 13.
    Meenakshi, B., Bhatnagar, A., Roy, S.: Tool for translating simulink models into input language of a model checker. In: Liu, Z., He, J. (eds.) ICFEM 2006. LNCS, vol. 4260, pp. 606–620. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Fehnker, A., Krogh, B.H.: Hybrid system verification is not a sinecure: The electronic throttle control case study. In: Wang, F. (ed.) ATVA 2004. LNCS, vol. 3299, pp. 263–277. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  15. 15.
    Joshi, A., Heimdahl, M.P.E.: Model-based safety analysis of Simulink models using SCADE design verifier. In: Winther, R., Gran, B.A., Dahll, G. (eds.) SAFECOMP 2005. LNCS, vol. 3688, pp. 122–135. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  16. 16.
    Ryabtsev, M., Strichman, O.: Translation validation: From simulink to c. In: Bouajjani, A., Maler, O. (eds.) Computer Aided Verification. LNCS, vol. 5643, pp. 696–701. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    The Mathworks: Simulink design verifier user’s guide. version 1.5 (2009),
  18. 18.
    Brillout, A., Kroening, D., Wahl, T.: Mixed abstractions for floating-point arithmetic. In: Formal Methods in Computer-Aided Design (FMCAD), pp. 69–76. IEEE, Los Alamitos (2009)Google Scholar
  19. 19.
    Kuehlmann, A., van Eijk, C.A.J.: Combinational and sequential equivalence checking. In: Logic Synthesis and Verification. Kluwer International Series in Engineering and Computer Science Series, pp. 343–372. Kluwer, Norwell (2002)CrossRefGoogle Scholar
  20. 20.
    Graf, S., Saïdi, H.: Construction of abstract state graphs with PVS. In: Grumberg, O. (ed.) CAV 1997. LNCS, vol. 1254, pp. 72–83. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  21. 21.
    Kroening, D., Clarke, E.: Checking consistency of C and Verilog using predicate abstraction and induction. In: IEEE/ACM International Conference on Computer-Aided Design, pp. 66–72. IEEE, Los Alamitos (2004)Google Scholar
  22. 22.
    Victor, A.C.: Interpretation of IEEE-854 floating-point standard and definition in the HOL system. Technical report, NASA Langley (1995)Google Scholar
  23. 23.
    Harrison, J.: Formal verification of square root algorithms. Formal Methods in System Design (FMSD) 22, 143–153 (2003)CrossRefzbMATHGoogle Scholar
  24. 24.
    Blanchet, B., Cousot, P., Cousot, R., Feret, J., Mauborgne, L., Miné, A., Monniaux, D., Rival, X.: A static analyzer for large safety-critical software. In: Programming Language Design and Implementation (PLDI), pp. 196–207. ACM, New York (2003)Google Scholar
  25. 25.
    Miné, A.: Relational abstract domains for the detection of floating-point run-time errors. In: Schmidt, D. (ed.) ESOP 2004. LNCS, vol. 2986, pp. 3–17. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  26. 26.
    Kroening, D., Strichman, O.: Decision Procedures. Springer, Heidelberg (2008)zbMATHGoogle Scholar
  27. 27.
    Eén, N., Sörensson, N.: An extensible SAT-solver. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, pp. 502–518. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  28. 28.
    Sheeran, M., Singh, S., Stålmarck, G.: Checking safety properties using induction and a SAT-solver. In: Johnson, S.D., Hunt Jr., W.A. (eds.) FMCAD 2000. LNCS, vol. 1954, pp. 108–125. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  29. 29.
    Bryant, R.E.: Graph-based algorithms for Boolean function manipulation. IEEE Transactions on Computers 35, 677–691 (1986)CrossRefzbMATHGoogle Scholar
  30. 30.
    McMillan, K.L.: Interpolation and SAT-based model checking. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 1–13. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  31. 31.
    Chockler, H., Kroening, D., Purandare, M.: Coverage in interpolation-based model checking. In: Design Automation Conference (DAC), ACM, New York (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Angelo Brillout
    • 1
  • Nannan He
    • 2
  • Michele Mazzucchi
    • 1
  • Daniel Kroening
    • 2
  • Mitra Purandare
    • 1
  • Philipp Rümmer
    • 2
  • Georg Weissenbacher
    • 1
    • 2
  1. 1.Computer Systems InstituteETHZurich
  2. 2.Computing LaboratoryOxford UniversityUK

Personalised recommendations