Skip to main content
SpringerLink
Log in

On theorem prover-based testing

  • Original Article
  • Published:
Formal Aspects of Computing

Abstract

HOL-TestGen is a specification and test case generation environment extending the interactive theorem prover Isabelle/HOL. As such, Testgen allows for an integrated workflow supporting interactive theorem proving, test case generation, and test data generation. The HOL-TestGen method is two-staged: first, the original formula is partitioned into test cases by transformation into a normal form called test theorem. Second, the test cases are analyzed for ground instances (the test data) satisfying the constraints of the test cases. Particular emphasis is put on the control of explicit test-hypotheses which can be proven over concrete programs. Due to the generality of the underlying framework, our system can be used for black-box unit, sequence, reactive sequence and white-box test scenarios. Although based on particularly clean theoretical foundations, the system can be applied for substantial case-studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aehlig K, Haftmann F, Nipkow T (2008) A compiled implementation of normalization by evaluation. In: Mohamed, OA, Muñoz C, Tahar S (eds) Theorem proving in higher order logics. Lecture notes in computer science, vol 5170, Heidelberg, August 2008. Springer, Berlin, pp 39–54

  2. Andrews PB (2002) Introduction to mathematical logic and type theory: to truth through proof, 2nd edn. Kluwer, Dordrecht

    Book  Google Scholar 

  3. Anand S, Pasareanu CS, Visser W (2007) JPF-SE: a symbolic execution extension to Java PathFinder. In: Grumberg O, Huth M (eds) TACAS. Lecture notes in computer science, vol 4424. Springer-Verlag, Heidelberg, pp 134–138

  4. Brucker AD, Brügger L, Kearney P, Wolff B (2010) Verified firewall policy transformations for test-case generation. In: Third international conference on software testing, verification, and validation (ICST), pp 345–354

  5. Brucker AD, Brügger L, Kearney P, Wolff B (2011) An approach to modular and testable security models of real-world health-care applications. In: ACM symposium on access control models and technologies (SACMAT). ACM Press, pp 133–142

  6. Brucker AD, Brügger L, Wolff B (2008) Model-based firewall conformance testing. In: Suzuki K, Higashino T (eds) Testcom/FATES 2008. Lecture notes in computer science, vol 5047. Springer, New York, pp 103–118

  7. Bernot G, Gaudel MC, Marre B (1991) Software testing based on formal specifications: a theory and a tool. Softw Eng J 6(6): 387–405

    Article  Google Scholar 

  8. Boyapati C, Khurshid S, Marinov D (2002) Korat: automated testing based on Java predicates. In: ISSTA, pp 123–133

  9. Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. In: Software engineering and formal methods (SEFM), Los Alamitos, CA, USA. IEEE Computer Society, pp 230–239

  10. Bentakouk L, Poizat P, Zaïdi F (2009) A formal framework for service orchestration testing based on symbolic transition systems. In: Néñez M, Baker P, Merayo MG (eds) TestCom/FATES. Lecture notes in computer science, vol 5826. Springer, Heidelberg, pp 16–32

    Google Scholar 

  11. Bjørner N, Tillmann N, Voronkov A (2009) Path feasibility analysis for string-manipulating programs. In: Kowalewski S, Philippou A (eds) TACAS. Lecture notes in computer science, vol 5505. Springer, Heidelberg, pp 307–321

    Google Scholar 

  12. Brucker AD, Wolff B (2004) Symbolic test case generation for primitive recursive functions. In: Grabowski J, Nielsen B (eds) Formal approaches to testing of software. Lecture notes in computer science, vol 3395. Springer, pp 16–32

  13. Brucker AD, Wolff B (2005) Interactive testing using HOL–TESTGEN. In: Grieskamp W, Weise C (eds) Formal approaches to testing of software. Lecture notes in computer science, vol 3997. Springer-Verlag

  14. Brucker AD, Wolff B (2007) Test-sequence generation with HOL-TESTGEN—with an application to firewall testing. In: Meyer B, Gurevich Y (eds) TAP 2007: tests and proofs. Lecture notes in computer science, vol 4454. Springer, pp 149–168

  15. Brucker AD, Wolff B (2008) An extensible encoding of object-oriented data models in HOL. J Autom Reason 41: 219–249

    Article  MATH  Google Scholar 

  16. Brucker AD, Wolff B (2009) HOL–TESTGEN: an interactive test-case generation framework. In: Chechik M, Wirsing M (eds) Fundamental approaches to software engineering FASE09. Lecture notes in computer science, vol 5503. Springer, pp 417–420

  17. Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the the fifth ACM SIGPLAN international conference on functional programming, New York, USA. ACM Press, pp 268–279

  18. Church A (1940) A formulation of the simple theory of types. J Symb Logic 5(2): 56–68

    Article  MathSciNet  Google Scholar 

  19. Dahl O-J, Dijkstra EW, Hoare CAR (1972) Structured programming. In: A.P.I.C. studies in data processing, vol 8. Academic Press, London

    Google Scholar 

  20. Dick J, Faivre A (1993) Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock JCP, Larsen PG (eds) Formal methods Europe 93: industrial-strength formal methods. Lecture notes in computer science, vol 670. Springer, Heidelberg, pp 268–284

    Chapter  Google Scholar 

  21. D’Agostino M, Gabbay D, Hähnle R, Posegga J (eds) (1996) Handbook of Tableau methods. Kluwer, Dordrecht

  22. de Halleux J, Tillmann N (2008) Parameterized unit testing with Pex. In: Beckert B, Hähnle R (eds) TAP. Lecture notes in computer science, vol 4966. Springer, Heidelberg, pp 171–181

    Google Scholar 

  23. Foster KA (1980) Error sensitive test cases analysis (estca). IEEE Trans Softw Eng 6(3): 258–264

    Article  MATH  Google Scholar 

  24. Gaudel MC (1995) Testing can be formal, too. In: Mosses PD, Nielsen M, Schwartzbach MI (eds) TAPPSOFT’95: theory and practice of software development. Lecture notes in computer science, vol 915. Springer, Heidelberg, pp 82–96

  25. Gaudel M-C, Denise A, Gouraud S-D, Lassaigne R, Oudinet J, Peyronnet S (2008) Coverage-biased random exploration of models. Electron Notes Theor Comput Sci 220(1): 3–14

    Article  Google Scholar 

  26. Gallaher MP, Kropp BM (2002) The economic impacts of inadequate infrastructure for software testing. Technical Report Planning Report 02-03, National Institute of Standards & Technology, May 2002

  27. Grieskamp W, Kicillof N, MacDonald D, Nandan A, Stobie K, Wurden FL (2008) Model-based quality assurance of windows protocol documentation. In: Software testing, verification, and validation (ICST), vol 0, Los Alamitos, CA, USA. IEEE Computer Society, pp 502–506

  28. Grieskamp W, Tillmann N, Veanes M (2004) Instrumenting scenarios in a model-driven development environment. Inf Softw Technol 46(15): 1027–1036

    Article  Google Scholar 

  29. Huima A (2007) Implementing conformiq qtronic. In: Petrenko A, Veanes M, Tretmans J, Grieskamp W (eds) TestCom/FATES. Lecture notes in computer science, vol 4581. Springer, Heidelberg, pp 1–12

  30. Jard C, Jéron T (2005) TGV: theory, principles and algorithms. Softw Tools Technol Transf 7(4): 297–315

    Article  Google Scholar 

  31. Jaffuel E, Legeard B (2007) Leirios test generator: automated test generation from b models. In: Julliand J, Kouchnarenko O (eds) Lecture notes in computer science, vol 4355. Springer, Heidelberg, pp 277–280

  32. Klein G (2009) Operating system verification—an overview. Sādhanā 34(1): 27–69

    MATH  Google Scholar 

  33. Lange C, McLaughlin S, Rabe F (2008) Flyspeck in a semantic Wiki. In: Lange C, Schaffert S, Skaf-Molli H, Völkel M (eds) SemWiki. CEUR workshop proceedings, vol 360. CEUR-WS.org

  34. Marre B, Blanc B (2005) Test selection strategies for lustre descriptions in GATeL. Electron Notes Theor Comput Sci 111: 93–111

    Article  Google Scholar 

  35. Moggi E (1991) Notions of computation and monads. Inf Comput 93(1): 55–92

    Article  MathSciNet  MATH  Google Scholar 

  36. Myers GJ, Sandler C (2004) The art of software testing. Wiley, New York

    Google Scholar 

  37. Nipkow T (1998) Winskel is (almost) right: towards a mechanized semantics textbook. Form Asp Comp 10(2): 171–186

    Article  MATH  Google Scholar 

  38. Nipkow T, Paulson LC, Wenzel M (2002) Isabelle/HOL—a proof assistant for higher-order logic. Lecture notes in computer science, vol 2283. Springer, Heidelberg

  39. Paulson LC (1999) A generic tableau prover and its integration with isabelle. J Univ Comput Sci 5(3): 73–87

    MathSciNet  MATH  Google Scholar 

  40. Roscoe AW (1998) Theory and practice of concurrency. Prentice Hall

  41. Tretmans GJ, Brinksma H (2003) Torx: automated model-based testing. In: Hartman A, Dussa-Ziegler K (eds) First European conference on model-driven software engineering, Nuremberg, Germany, pp 31–43

  42. Tillmann N, de Halleux J (2008) Pex—white box test generation for .NET. In: Beckert B, Hähnle R (eds) TAP. Lecture notes in computer science, vol 4966. Springer, Heidelberg, pp 134–153

    Google Scholar 

  43. Tej H, Wolff B (1997) A corrected failure divergence model for CSP in Isabelle/HOL. In: Fitzgerald JS, Jones CB, Lucas P (eds) Formal Methods Europe (FME) Lecture notes in computer science, vol 1313. Springer, Heidelberg, pp 318–337

    Google Scholar 

  44. Veanes M, Campbell C, Grieskamp W, Schulte W, Tillmann N, Nachmanson L (2008) Model-based testing of object-oriented reactive systems with spec explorer. In: Hierons RM, Bowen JP, Harman M (eds) Formal methods and testing. Lecture notes in computer science, vol 4949. Springer, pp 39–76

    Chapter  Google Scholar 

  45. Visser W, Havelund K, Brat GP, Park S, Lerda F (2003) Model checking programs. Autom Softw Eng 10(2): 203–232

    Article  Google Scholar 

  46. von Oheimb D (2001) Analyzing Java in Isabelle/HOL: formalization, type safety and Hoare logic. PhD thesis, Technische Universität München

  47. Visser W, Păsăreanu CS, Khurshid S (2004) Test input generation with Java PathFinder. SIGSOFT Softw Eng Notes 29(4): 97–107

    Article  Google Scholar 

  48. Wadler P (1995) Monads for functional programming. In: Jeuring J, Meijer E (eds) Advanced functional programming .Lecture notes in computer science, vol 925. Springer, Heidelberg, pp 24–52

    Chapter  Google Scholar 

  49. Wenzel MM (2002) Isabelle/Isar—a versatile environment for human-readable formal proof documents. PhD thesis, TU München, München, February 2002

  50. Winskel G (1993) The formal semantics of programming languages. MIT Press, Cambridge

    MATH  Google Scholar 

  51. Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4): 366–427

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SAP Research, Vincenz-Priessnitz-Str. 1, 76131, Karlsruhe, Germany

    Achim D. Brucker

  2. Université Paris-Sud 11, LRI-Batiment 650, 91405, Orsay Cedex, France

    Burkhart Wolff

Authors
  1. Achim D. Brucker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Burkhart Wolff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Burkhart Wolff.

Additional information

by Jim Woodcock

This work was partially supported by the Digiteo Foundation.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Brucker, A.D., Wolff, B. On theorem prover-based testing. Form Asp Comp 25, 683–721 (2013). https://doi.org/10.1007/s00165-012-0222-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00165-012-0222-y

Keywords

Search

Navigation