Abstract
While many approaches for automatic test case generation have been proposed over the years, it is often difficult to predict which of them may work well on concrete problems. In this paper, we therefore present a case study in automatic, model-based test case generation: We implemented several graph-based methods that compute test cases with a model checker using trap properties, and evaluate these methods on a Secure Block Device implementation. We compare the number of generated test cases, the required generation time and the achieved code coverage. Our conclusions are twofold: First, automatic test case generation is feasible and beneficial for this case study, and even found a real bug in the implementation. Second, simple coverage methods on the model may already yield test suites of sufficient quality.
This work was supported by the Austrian Research Promotion Agency (FFG) through projects NewP@ss (835917) and TARGET (845633) and by the European Commission through project IMMORTAL(644905).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)
Ammann, P., Offutt, J., Huang, H.: Coverage criteria for logical expressions. In: 14th International Symposium on Software Reliability Engineering: ISSRE 2003, pp. 99–107. IEEE (2003)
Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Softw. Eng. J. 6(6), 387–405 (1991). http://dx.doi.org/10.1049/sej.1991.0040
Beyer, D., Chlipala, A.J., Henzinger, T.A., Jhala, R., Majumdar, R.: Generating tests from counterexamples. In: Proceedings of the 26th International Conference on Software Engineering, ICSE 2004, pp. 326–335. IEEE Computer Society, Washington, DC (2004). http://dl.acm.org/citation.cfm?id=998675.999437
Bloem, R., Könighofer, R., Röck, F., Tautschnig, M.: Automating test-suite augmentation. In: 2014 14th International Conference on Quality Software, October 2–3, Allen, TX, USA, pp. 67–72 (2014). http://dx.doi.org/10.1109/QSIC.2014.40
Bloem, R.P., Greimel, K., Könighofer, R., Röck, F.: Model-based MCDC testing of complex decisions for the java card applet firewall. In: VALID Proceedings, IARIA, Ed., pp. 1–6 (2013)
Brooks, R.J., Tobias, A.M.: Choosing the best model: Level of detail, complexity, and model performance. Mathematical and Computer Modelling 24(4), 1–14 (1996)
Chilenski, J.J.: An investigation of three forms of the modified condition decision coverage (MCDC) criterion. Tech. Rep., DTIC Document (2001)
Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002)
Clarke, E.M., Grumberg, O., Peled, D.: Model checking. MIT Press (2001). http://books.google.de/books?id=Nmc4wEaLXFEC
Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: Proceedings of the 21st International Conference on Software Engineering, ICSE 1999, pp. 285–294. ACM, New York (1999). http://doi.acm.org/10.1145/302405.302640
de Moura, L., Bjørner, N.S.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: Help for the practicing programmer. Computer 11(4), 34–41 (1978). http://dx.doi.org/10.1109/C-M.1978.218136
Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Larsen, P.G., Wing, J.M. (eds.) FME 1993. LNCS, vol. 670, pp. 268–284. Springer, Heidelberg (1993)
Fraser, G., Gargantini, A.: An evaluation of model checkers for specification based test case generation. In: ICST 2009, Second International Conference on Software Testing Verification and Validation, April 1–4, Denver, Colorado, USA, pp. 41–50 (2009). http://dx.doi.org/10.1109/ICST.2009.33
Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: A survey. Softw. Test. Verif. Reliab. 19(3), 215–261 (2009). http://dx.doi.org/10.1002/stvr.v19:3
Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. SIGSOFT Softw. Eng. Notes 24(6), 146–162 (1999). http://doi.acm.org/10.1145/318774.318939
Gaudel, M.-C.: Testing can be formal, too. In: Mosses, P.D., Nielsen, M. (eds.) CAAP 1995, FASE 1995, and TAPSOFT 1995. LNCS, vol. 915, pp. 82–96. Springer, Heidelberg (1995)
Hong, H.S., Lee, I., Sokolsky, O., Ural, H.: A temporal logic based theory of test coverage and generation. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 327–341. Springer, Heidelberg (2002). http://dl.acm.org/citation.cfm?id=646486.694621
Jorgensen, P.C.: Software testing - a craftsman’s approach, 3rd edn. Taylor & Francis (2008)
Offutt, A.J., Untch, R.H.: Mutation, : Uniting the orthogonal. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 34–44. Kluwer Academic Publishers (2000)
Offutt, J., Liu, S., Abdurazik, A., Ammann, P.: Generating test data from state-based specifications. Software Testing, Verification and Reliability 13, 25–53 (2003)
Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: Wermelinger, M., Gall, H.C. (eds.) Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, September 5–9, pp. 263–272. ACM, Lisbon (2005). http://doi.acm.org/10.1145/1081706.1081750
Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012). http://dx.doi.org/10.1002/stvr.456
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Bloem, R., Hein, D., Röck, F., Schumi, R. (2015). Case Study: Automatic Test Case Generation for a Secure Cache Implementation. In: Blanchette, J., Kosmatov, N. (eds) Tests and Proofs. TAP 2015. Lecture Notes in Computer Science(), vol 9154. Springer, Cham. https://doi.org/10.1007/978-3-319-21215-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-21215-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-21214-2
Online ISBN: 978-3-319-21215-9
eBook Packages: Computer ScienceComputer Science (R0)