Formal Aspects of Computing

, Volume 25, Issue 5, pp 683–721 | Cite as

On theorem prover-based testing

  • Achim D. Brucker
  • Burkhart WolffEmail author
Original Article


HOL-TestGen is a specification and test case generation environment extending the interactive theorem prover Isabelle/HOL. As such, Testgen allows for an integrated workflow supporting interactive theorem proving, test case generation, and test data generation. The HOL-TestGen method is two-staged: first, the original formula is partitioned into test cases by transformation into a normal form called test theorem. Second, the test cases are analyzed for ground instances (the test data) satisfying the constraints of the test cases. Particular emphasis is put on the control of explicit test-hypotheses which can be proven over concrete programs. Due to the generality of the underlying framework, our system can be used for black-box unit, sequence, reactive sequence and white-box test scenarios. Although based on particularly clean theoretical foundations, the system can be applied for substantial case-studies.


Test case generation Domain partitioning Test sequence Theorem proving HOL-TestGen 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. AHN08.
    Aehlig K, Haftmann F, Nipkow T (2008) A compiled implementation of normalization by evaluation. In: Mohamed, OA, Muñoz C, Tahar S (eds) Theorem proving in higher order logics. Lecture notes in computer science, vol 5170, Heidelberg, August 2008. Springer, Berlin, pp 39–54Google Scholar
  2. And02.
    Andrews PB (2002) Introduction to mathematical logic and type theory: to truth through proof, 2nd edn. Kluwer, DordrechtCrossRefGoogle Scholar
  3. APV07.
    Anand S, Pasareanu CS, Visser W (2007) JPF-SE: a symbolic execution extension to Java PathFinder. In: Grumberg O, Huth M (eds) TACAS. Lecture notes in computer science, vol 4424. Springer-Verlag, Heidelberg, pp 134–138Google Scholar
  4. BBKW10.
    Brucker AD, Brügger L, Kearney P, Wolff B (2010) Verified firewall policy transformations for test-case generation. In: Third international conference on software testing, verification, and validation (ICST), pp 345–354Google Scholar
  5. BBKW11.
    Brucker AD, Brügger L, Kearney P, Wolff B (2011) An approach to modular and testable security models of real-world health-care applications. In: ACM symposium on access control models and technologies (SACMAT). ACM Press, pp 133–142Google Scholar
  6. BBW08.
    Brucker AD, Brügger L, Wolff B (2008) Model-based firewall conformance testing. In: Suzuki K, Higashino T (eds) Testcom/FATES 2008. Lecture notes in computer science, vol 5047. Springer, New York, pp 103–118Google Scholar
  7. BGM91.
    Bernot G, Gaudel MC, Marre B (1991) Software testing based on formal specifications: a theory and a tool. Softw Eng J 6(6): 387–405CrossRefGoogle Scholar
  8. BKM02.
    Boyapati C, Khurshid S, Marinov D (2002) Korat: automated testing based on Java predicates. In: ISSTA, pp 123–133Google Scholar
  9. BN04.
    Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. In: Software engineering and formal methods (SEFM), Los Alamitos, CA, USA. IEEE Computer Society, pp 230–239Google Scholar
  10. BPZ09.
    Bentakouk L, Poizat P, Zaïdi F (2009) A formal framework for service orchestration testing based on symbolic transition systems. In: Néñez M, Baker P, Merayo MG (eds) TestCom/FATES. Lecture notes in computer science, vol 5826. Springer, Heidelberg, pp 16–32Google Scholar
  11. BTV09.
    Bjørner N, Tillmann N, Voronkov A (2009) Path feasibility analysis for string-manipulating programs. In: Kowalewski S, Philippou A (eds) TACAS. Lecture notes in computer science, vol 5505. Springer, Heidelberg, pp 307–321Google Scholar
  12. BW04.
    Brucker AD, Wolff B (2004) Symbolic test case generation for primitive recursive functions. In: Grabowski J, Nielsen B (eds) Formal approaches to testing of software. Lecture notes in computer science, vol 3395. Springer, pp 16–32Google Scholar
  13. BW05.
    Brucker AD, Wolff B (2005) Interactive testing using HOL–TESTGEN. In: Grieskamp W, Weise C (eds) Formal approaches to testing of software. Lecture notes in computer science, vol 3997. Springer-VerlagGoogle Scholar
  14. BW07.
    Brucker AD, Wolff B (2007) Test-sequence generation with HOL-TESTGEN—with an application to firewall testing. In: Meyer B, Gurevich Y (eds) TAP 2007: tests and proofs. Lecture notes in computer science, vol 4454. Springer, pp 149–168Google Scholar
  15. BW08.
    Brucker AD, Wolff B (2008) An extensible encoding of object-oriented data models in HOL. J Autom Reason 41: 219–249zbMATHCrossRefGoogle Scholar
  16. BW09.
    Brucker AD, Wolff B (2009) HOL–TESTGEN: an interactive test-case generation framework. In: Chechik M, Wirsing M (eds) Fundamental approaches to software engineering FASE09. Lecture notes in computer science, vol 5503. Springer, pp 417–420Google Scholar
  17. CH00.
    Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the the fifth ACM SIGPLAN international conference on functional programming, New York, USA. ACM Press, pp 268–279Google Scholar
  18. Chu40.
    Church A (1940) A formulation of the simple theory of types. J Symb Logic 5(2): 56–68MathSciNetCrossRefGoogle Scholar
  19. DDH72.
    Dahl O-J, Dijkstra EW, Hoare CAR (1972) Structured programming. In: A.P.I.C. studies in data processing, vol 8. Academic Press, LondonGoogle Scholar
  20. DF93.
    Dick J, Faivre A (1993) Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock JCP, Larsen PG (eds) Formal methods Europe 93: industrial-strength formal methods. Lecture notes in computer science, vol 670. Springer, Heidelberg, pp 268–284CrossRefGoogle Scholar
  21. DGHP96.
    D’Agostino M, Gabbay D, Hähnle R, Posegga J (eds) (1996) Handbook of Tableau methods. Kluwer, DordrechtGoogle Scholar
  22. dHT08.
    de Halleux J, Tillmann N (2008) Parameterized unit testing with Pex. In: Beckert B, Hähnle R (eds) TAP. Lecture notes in computer science, vol 4966. Springer, Heidelberg, pp 171–181Google Scholar
  23. Fos80.
    Foster KA (1980) Error sensitive test cases analysis (estca). IEEE Trans Softw Eng 6(3): 258–264zbMATHCrossRefGoogle Scholar
  24. Gau95.
    Gaudel MC (1995) Testing can be formal, too. In: Mosses PD, Nielsen M, Schwartzbach MI (eds) TAPPSOFT’95: theory and practice of software development. Lecture notes in computer science, vol 915. Springer, Heidelberg, pp 82–96Google Scholar
  25. GDG+08.
    Gaudel M-C, Denise A, Gouraud S-D, Lassaigne R, Oudinet J, Peyronnet S (2008) Coverage-biased random exploration of models. Electron Notes Theor Comput Sci 220(1): 3–14CrossRefGoogle Scholar
  26. GK02.
    Gallaher MP, Kropp BM (2002) The economic impacts of inadequate infrastructure for software testing. Technical Report Planning Report 02-03, National Institute of Standards & Technology, May 2002Google Scholar
  27. GKM+08.
    Grieskamp W, Kicillof N, MacDonald D, Nandan A, Stobie K, Wurden FL (2008) Model-based quality assurance of windows protocol documentation. In: Software testing, verification, and validation (ICST), vol 0, Los Alamitos, CA, USA. IEEE Computer Society, pp 502–506Google Scholar
  28. GTV04.
    Grieskamp W, Tillmann N, Veanes M (2004) Instrumenting scenarios in a model-driven development environment. Inf Softw Technol 46(15): 1027–1036CrossRefGoogle Scholar
  29. Hui07.
    Huima A (2007) Implementing conformiq qtronic. In: Petrenko A, Veanes M, Tretmans J, Grieskamp W (eds) TestCom/FATES. Lecture notes in computer science, vol 4581. Springer, Heidelberg, pp 1–12Google Scholar
  30. JJ05.
    Jard C, Jéron T (2005) TGV: theory, principles and algorithms. Softw Tools Technol Transf 7(4): 297–315CrossRefGoogle Scholar
  31. JL07.
    Jaffuel E, Legeard B (2007) Leirios test generator: automated test generation from b models. In: Julliand J, Kouchnarenko O (eds) Lecture notes in computer science, vol 4355. Springer, Heidelberg, pp 277–280Google Scholar
  32. Kle09.
    Klein G (2009) Operating system verification—an overview. Sādhanā 34(1): 27–69zbMATHGoogle Scholar
  33. LMR08.
    Lange C, McLaughlin S, Rabe F (2008) Flyspeck in a semantic Wiki. In: Lange C, Schaffert S, Skaf-Molli H, Völkel M (eds) SemWiki. CEUR workshop proceedings, vol 360. CEUR-WS.orgGoogle Scholar
  34. MB05.
    Marre B, Blanc B (2005) Test selection strategies for lustre descriptions in GATeL. Electron Notes Theor Comput Sci 111: 93–111CrossRefGoogle Scholar
  35. Mog91.
    Moggi E (1991) Notions of computation and monads. Inf Comput 93(1): 55–92MathSciNetzbMATHCrossRefGoogle Scholar
  36. MS04.
    Myers GJ, Sandler C (2004) The art of software testing. Wiley, New YorkGoogle Scholar
  37. Nip98.
    Nipkow T (1998) Winskel is (almost) right: towards a mechanized semantics textbook. Form Asp Comp 10(2): 171–186zbMATHCrossRefGoogle Scholar
  38. NPW02.
    Nipkow T, Paulson LC, Wenzel M (2002) Isabelle/HOL—a proof assistant for higher-order logic. Lecture notes in computer science, vol 2283. Springer, HeidelbergGoogle Scholar
  39. Pau99.
    Paulson LC (1999) A generic tableau prover and its integration with isabelle. J Univ Comput Sci 5(3): 73–87MathSciNetzbMATHGoogle Scholar
  40. Ros98.
    Roscoe AW (1998) Theory and practice of concurrency. Prentice HallGoogle Scholar
  41. TB03.
    Tretmans GJ, Brinksma H (2003) Torx: automated model-based testing. In: Hartman A, Dussa-Ziegler K (eds) First European conference on model-driven software engineering, Nuremberg, Germany, pp 31–43Google Scholar
  42. TdH08.
    Tillmann N, de Halleux J (2008) Pex—white box test generation for .NET. In: Beckert B, Hähnle R (eds) TAP. Lecture notes in computer science, vol 4966. Springer, Heidelberg, pp 134–153Google Scholar
  43. TW97.
    Tej H, Wolff B (1997) A corrected failure divergence model for CSP in Isabelle/HOL. In: Fitzgerald JS, Jones CB, Lucas P (eds) Formal Methods Europe (FME) Lecture notes in computer science, vol 1313. Springer, Heidelberg, pp 318–337Google Scholar
  44. VCG+08.
    Veanes M, Campbell C, Grieskamp W, Schulte W, Tillmann N, Nachmanson L (2008) Model-based testing of object-oriented reactive systems with spec explorer. In: Hierons RM, Bowen JP, Harman M (eds) Formal methods and testing. Lecture notes in computer science, vol 4949. Springer, pp 39–76CrossRefGoogle Scholar
  45. VHB+03.
    Visser W, Havelund K, Brat GP, Park S, Lerda F (2003) Model checking programs. Autom Softw Eng 10(2): 203–232CrossRefGoogle Scholar
  46. vO01.
    von Oheimb D (2001) Analyzing Java in Isabelle/HOL: formalization, type safety and Hoare logic. PhD thesis, Technische Universität MünchenGoogle Scholar
  47. VPK04.
    Visser W, Păsăreanu CS, Khurshid S (2004) Test input generation with Java PathFinder. SIGSOFT Softw Eng Notes 29(4): 97–107CrossRefGoogle Scholar
  48. Wad95.
    Wadler P (1995) Monads for functional programming. In: Jeuring J, Meijer E (eds) Advanced functional programming .Lecture notes in computer science, vol 925. Springer, Heidelberg, pp 24–52CrossRefGoogle Scholar
  49. Wen02.
    Wenzel MM (2002) Isabelle/Isar—a versatile environment for human-readable formal proof documents. PhD thesis, TU München, München, February 2002Google Scholar
  50. Win93.
    Winskel G (1993) The formal semantics of programming languages. MIT Press, CambridgezbMATHGoogle Scholar
  51. ZHM97.
    Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4): 366–427CrossRefGoogle Scholar

Copyright information

© British Computer Society 2012

Authors and Affiliations

  1. 1.SAP ResearchKarlsruheGermany
  2. 2.Université Paris-Sud 11Orsay CedexFrance

Personalised recommendations