Tightening Test Coverage Metrics: A Case Study in Equivalence Checking Using k-Induction

  • Alastair F. Donaldson
  • Nannan He
  • Daniel Kroening
  • Philipp Rümmer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6957)

Abstract

We present a case study applying the k-induction method to equivalence checking of Simulink designs. In particular, we are interested in the problem of equivalence detection in mutation-based testing: given a design S, determining whether a “mutant” design S′ derived from S by syntactic fault injection is behaviourally equivalent to S. In this situation, efficient equivalence checking techniques are needed to avoid redundant and expensive search for test cases that observe differences between S and S′. We have integrated k-induction into our test case generation framework for Simulink. We show, using a selection of benchmarks, that k-induction can be effective in detecting equivalent mutants, sometimes as a stand-alone technique, and sometimes with some manual assistance. We further discuss how the level of automation of the method can be increased by using static analysis to derive strengthening invariants from the structure of the Simulink models.

Keywords

Equivalence Check Abstract Interpretation Formal Concept Analysis Bound Model Check Random Simulation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Armoni, R., Fix, L., Fraer, R., Huddleston, S., Piterman, N., Vardi, M.Y.: SAT-based induction for temporal safety properties. Electr. Notes Theor. Comput. Sci. 119(2), 3–16 (2005)CrossRefMATHGoogle Scholar
  2. 2.
    Biere, A., Cimatti, A., Clarke, E.M., Strichman, O., Zhu, Y.: Bounded model checking. Advances in Computers 58, 118–149 (2003)Google Scholar
  3. 3.
    Biere, A., Cimatti, A., Clarke, E.M., Zhu, Y.: Symbolic model checking without BDDs. In: Cleaveland, W.R. (ed.) TACAS 1999. LNCS, vol. 1579, pp. 193–207. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  4. 4.
    Bjesse, P., Claessen, K.: SAT-based verification without state space traversal. In: Johnson, S.D., Hunt Jr., W.A. (eds.) FMCAD 2000. LNCS, vol. 1954, pp. 372–389. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  5. 5.
    Brillout, A., He, N., Mazzucchi, M., Kroening, D., Purandare, M., Rümmer, P., Weissenbacher, G.: Mutation-based test case generation for simulink models. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 208–227. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Clarke, E. M., Kröning, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 168–176. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: Principles of Programming Languages (POPL), pp. 238–252. ACM, New York (1977)Google Scholar
  8. 8.
    Déharbe, D., Moreira, A.M.: Using induction and BDDs to model check invariants. In: CHARME. IFIP Conference Proceedings, vol. 105, pp. 203–213. Chapman & Hall, Boca Raton (1997)Google Scholar
  9. 9.
    DeMillo, R., Lipton, R., Sayward, F.: Hints on test data selection: Help for the practicing programmer. Computer 11(4), 34–41 (1978)CrossRefGoogle Scholar
  10. 10.
    Donaldson, A.F., Haller, L., Kroening, D.: Strengthening induction-based race checking with lightweight static analysis. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 169–183. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  11. 11.
    Donaldson, A.F., Haller, L., Kroening, D., Rümmer, P.: Software verification using k-induction. In: Yahav, E. (ed.) SAS 2011. LNCS, vol. 6887, pp. 351–368. Springer, Heidelberg (2011)Google Scholar
  12. 12.
    Donaldson, A.F., Kroening, D., Rümmer, P.: Automatic analysis of scratch-pad memory code for heterogeneous multicore processors. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 280–295. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Donaldson, A.F., Kroening, D., Rümmer, P.: Automatic analysis of DMA races using model checking and k-induction. Formal Methods in System Design (2011)Google Scholar
  14. 14.
    Eén, N., Sörensson, N.: Temporal induction by incremental SAT solving. Electr. Notes Theor. Comput. Sci. 89(4) (2003)Google Scholar
  15. 15.
    van Eijk, C.A.J.: Sequential equivalence checking without state space traversal. In: Proceedings of the Conference on Design, Automation and Test in Europe (DATE), pp. 618–623. IEEE, Los Alamitos (1998)CrossRefGoogle Scholar
  16. 16.
    Franzén, A.: Using satisfiability modulo theories for inductive verification of Lustre programs. Electr. Notes Theor. Comput. Sci. 144(1), 19–33 (2006)CrossRefMATHGoogle Scholar
  17. 17.
    Große, D., Le, H.M., Drechsler, R.: Proving transaction and system-level properties of untimed SystemC TLM designs. In: MEMOCODE, pp. 113–122. IEEE Computer Society, Los Alamitos (2010)Google Scholar
  18. 18.
    Hagen, G., Tinelli, C.: Scaling up the formal verification of Lustre programs with SMT-based techniques. In: FMCAD, pp. 109–117. IEEE, Los Alamitos (2008)Google Scholar
  19. 19.
    He, N., Rümmer, P., Kroening, D.: Test-case generation for embedded Simulink via formal concept analysis. In: Proceedings of DAC (2011)Google Scholar
  20. 20.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, TSE (2010)Google Scholar
  21. 21.
    Kuehlmann, A., van Eijk, C.A.J.: Combinational and sequential equivalence checking. In: Logic Synthesis and Verification. Kluwer International Series in Engineering and Computer Science Series, pp. 343–372. Kluwer, Dordrecht (2002)CrossRefGoogle Scholar
  22. 22.
    Kupferman, O., Li, W., Seshia, S.A.: A theory of mutations with applications to vacuity, coverage, and fault tolerance. In: Formal Methods in Computer-Aided Design (FMCAD), pp. 1–9. IEEE, Los Alamitos (2008)Google Scholar
  23. 23.
    Lillieroth, C.J., Singh, S.: Formal verification of FPGA cores. Nord. J. Comput. 6(3), 299–319 (1999)MATHGoogle Scholar
  24. 24.
    Offutt, J., Voas, J.M.: Subsumption of condition coverage techniques by mutation testing. Tech. Rep. ISSE-TR-96-01, George Mason University (1996)Google Scholar
  25. 25.
    Ruthruff, J.R., Burnett, M.M., Rothermel, G.: Interactive fault localization techniques in a spreadsheet environment. IEEE Transactions on Software Engineering (TSE) 32(4), 213–239 (2006)CrossRefGoogle Scholar
  26. 26.
    Sheeran, M., Singh, S., Stålmarck, G.: Checking safety properties using induction and a SAT-solver. In: Johnson, S.D., Hunt Jr., W.A. (eds.) FMCAD 2000. LNCS, vol. 1954, pp. 108–125. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  27. 27.
    Toom, A., Izerrouken, N., Naks, T., Pantel, M., Kai, O.S.Y.: Towards reliable code generation with an open tool: Evolutions of the Gene-Auto toolset. In: Proceedings, Embedded Real Time Software and Systems, ERTS (2010)Google Scholar
  28. 28.
    Vimjam, V.C., Hsiao, M.S.: Explicit safety property strengthening in SAT-based induction. In: VLSID, pp. 63–68. IEEE, Los Alamitos (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Alastair F. Donaldson
    • 1
  • Nannan He
    • 1
  • Daniel Kroening
    • 1
  • Philipp Rümmer
    • 2
  1. 1.Computer Science DepartmentOxford UniversityUK
  2. 2.Department of Information TechnologyUppsala UniversityUppsalaSweden

Personalised recommendations