Case Study: Automatic Test Case Generation for a Secure Cache Implementation

  • Roderick Bloem
  • Daniel Hein
  • Franz Röck
  • Richard Schumi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9154)

Abstract

While many approaches for automatic test case generation have been proposed over the years, it is often difficult to predict which of them may work well on concrete problems. In this paper, we therefore present a case study in automatic, model-based test case generation: We implemented several graph-based methods that compute test cases with a model checker using trap properties, and evaluate these methods on a Secure Block Device implementation. We compare the number of generated test cases, the required generation time and the achieved code coverage. Our conclusions are twofold: First, automatic test case generation is feasible and beneficial for this case study, and even found a real bug in the implementation. Second, simple coverage methods on the model may already yield test suites of sufficient quality.

Keywords

Automatic test case generation Model-based testing Model checking Trap properties 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)CrossRefGoogle Scholar
  2. 2.
    Ammann, P., Offutt, J., Huang, H.: Coverage criteria for logical expressions. In: 14th International Symposium on Software Reliability Engineering: ISSRE 2003, pp. 99–107. IEEE (2003)Google Scholar
  3. 3.
    Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Softw. Eng. J. 6(6), 387–405 (1991). http://dx.doi.org/10.1049/sej.1991.0040
  4. 4.
    Beyer, D., Chlipala, A.J., Henzinger, T.A., Jhala, R., Majumdar, R.: Generating tests from counterexamples. In: Proceedings of the 26th International Conference on Software Engineering, ICSE 2004, pp. 326–335. IEEE Computer Society, Washington, DC (2004). http://dl.acm.org/citation.cfm?id=998675.999437
  5. 5.
    Bloem, R., Könighofer, R., Röck, F., Tautschnig, M.: Automating test-suite augmentation. In: 2014 14th International Conference on Quality Software, October 2–3, Allen, TX, USA, pp. 67–72 (2014). http://dx.doi.org/10.1109/QSIC.2014.40
  6. 6.
    Bloem, R.P., Greimel, K., Könighofer, R., Röck, F.: Model-based MCDC testing of complex decisions for the java card applet firewall. In: VALID Proceedings, IARIA, Ed., pp. 1–6 (2013)Google Scholar
  7. 7.
    Brooks, R.J., Tobias, A.M.: Choosing the best model: Level of detail, complexity, and model performance. Mathematical and Computer Modelling 24(4), 1–14 (1996)CrossRefMATHGoogle Scholar
  8. 8.
    Chilenski, J.J.: An investigation of three forms of the modified condition decision coverage (MCDC) criterion. Tech. Rep., DTIC Document (2001)Google Scholar
  9. 9.
    Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  10. 10.
    Clarke, E.M., Grumberg, O., Peled, D.: Model checking. MIT Press (2001). http://books.google.de/books?id=Nmc4wEaLXFEC
  11. 11.
    Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: Proceedings of the 21st International Conference on Software Engineering, ICSE 1999, pp. 285–294. ACM, New York (1999). http://doi.acm.org/10.1145/302405.302640
  12. 12.
    de Moura, L., Bjørner, N.S.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  13. 13.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: Help for the practicing programmer. Computer 11(4), 34–41 (1978). http://dx.doi.org/10.1109/C-M.1978.218136
  14. 14.
    Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Larsen, P.G., Wing, J.M. (eds.) FME 1993. LNCS, vol. 670, pp. 268–284. Springer, Heidelberg (1993) CrossRefGoogle Scholar
  15. 15.
    Fraser, G., Gargantini, A.: An evaluation of model checkers for specification based test case generation. In: ICST 2009, Second International Conference on Software Testing Verification and Validation, April 1–4, Denver, Colorado, USA, pp. 41–50 (2009). http://dx.doi.org/10.1109/ICST.2009.33
  16. 16.
    Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: A survey. Softw. Test. Verif. Reliab. 19(3), 215–261 (2009). http://dx.doi.org/10.1002/stvr.v19:3
  17. 17.
    Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. SIGSOFT Softw. Eng. Notes 24(6), 146–162 (1999). http://doi.acm.org/10.1145/318774.318939
  18. 18.
    Gaudel, M.-C.: Testing can be formal, too. In: Mosses, P.D., Nielsen, M. (eds.) CAAP 1995, FASE 1995, and TAPSOFT 1995. LNCS, vol. 915, pp. 82–96. Springer, Heidelberg (1995) CrossRefGoogle Scholar
  19. 19.
    Hong, H.S., Lee, I., Sokolsky, O., Ural, H.: A temporal logic based theory of test coverage and generation. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 327–341. Springer, Heidelberg (2002). http://dl.acm.org/citation.cfm?id=646486.694621
  20. 20.
    Jorgensen, P.C.: Software testing - a craftsman’s approach, 3rd edn. Taylor & Francis (2008)Google Scholar
  21. 21.
    Offutt, A.J., Untch, R.H.: Mutation, : Uniting the orthogonal. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 34–44. Kluwer Academic Publishers (2000)Google Scholar
  22. 22.
    Offutt, J., Liu, S., Abdurazik, A., Ammann, P.: Generating test data from state-based specifications. Software Testing, Verification and Reliability 13, 25–53 (2003)CrossRefGoogle Scholar
  23. 23.
    Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: Wermelinger, M., Gall, H.C. (eds.) Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, September 5–9, pp. 263–272. ACM, Lisbon (2005). http://doi.acm.org/10.1145/1081706.1081750
  24. 24.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012). http://dx.doi.org/10.1002/stvr.456

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Roderick Bloem
    • 1
  • Daniel Hein
    • 1
  • Franz Röck
    • 1
  • Richard Schumi
    • 1
  1. 1.Institute for Applied Information Processing and Communications (IAIK)Graz University of TechnologyGrazAustria

Personalised recommendations