Controllable Combinatorial Coverage in Grammar-Based Testing

  • Ralf Lämmel
  • Wolfram Schulte
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3964)


Given a grammar (or other sorts of meta-data), one can trivially derive combinatorially exhaustive test-data sets up to a specified depth. Without further efforts, such test-data sets would be huge at the least and explosive most of the time. Fortunately, scenarios of grammar-based testing tend to admit non-explosive approximations of naive combinatorial coverage.

In this paper, we describe the notion of controllable combinatorial coverage and a corresponding algorithm for test-data generation. The approach is based on a suite of control mechanisms to be used for the characterization of test-data sets as well-defined and understandable approximations of full combinatorial coverage.

The approach has been implemented in the C#-based test-data generator Geno, which has been successfully used in projects that required differential testing, stress testing and conformance testing of grammar-driven functionality.


Depth Limit Symbolic Execution Argument Position Conformance Testing Abstract State Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Andoni, A., Daniliuc, D., Khurshid, S., Marinov, D.: Evaluating the Small Scope Hypothesis (unpublished, September 2002), Available at:
  2. 2.
    Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Software Engineering Journal 6(6), 387–405 (1991)CrossRefGoogle Scholar
  3. 3.
    Bird, D.L., Munoz, C.U.: Automatic generation of random self-checking test cases. IBM Systems Journal 22(3), 229–245 (1983)CrossRefGoogle Scholar
  4. 4.
    Bouge, L., Choquet, N., Fribourg, L., Gaudel, M.-C.: Test sets generation from algebraic specifications using logic programming. Journal of Systems and Software 6(4), 343–360 (1986)CrossRefGoogle Scholar
  5. 5.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on java predicates. In: Proc. International Symposium on Software testing and analysis, pp. 123–133. ACM Press, New York (2002)Google Scholar
  6. 6.
    Burgess, C.: The Automated Generation of Test Cases for Compilers. Software Testing, Verification and Reliability 4(2), 81–99 (1994)CrossRefGoogle Scholar
  7. 7.
    Burgess, C.J., Saidi, M.: The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology 38(2), 111–119 (1996)CrossRefGoogle Scholar
  8. 8.
    Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP 2000: Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, pp. 268–279. ACM Press, New York (2000)CrossRefGoogle Scholar
  9. 9.
    Cohen, D., Dalal, S., Fredman, M., Patton, G.: The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering 23(7), 437–443 (1997)CrossRefGoogle Scholar
  10. 10.
    Foundations of Software Engineering, Microsoft Research. AsmL—Abstract State Machine Language (2005),
  11. 11.
    Gurevich, Y., Wallace, C.: Specification and Verification of the Windows Card Runtime Environment Using Abstract State Machines. Technical report, Microsoft Research (February 1999) MSR-TR-99-07Google Scholar
  12. 12.
    Harm, J., Lämmel, R.: Two-dimensional Approximation Coverage. Informatica 24(3), 355–369 (2000)MATHGoogle Scholar
  13. 13.
    Jackson, D., Damon, C.A.: Elements of Style: Analyzing a Software Design Feature with a Counterexample Detector. IEEE Transactions on Software Engineering 22(7), 484–495 (1996)CrossRefGoogle Scholar
  14. 14.
    King, J.C.: Symbolic execution and program testing. Communications of the ACM 19(7), 385–394 (1976)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Knuth, D.: Semantics of context-free languages. Mathematical Systems Theory 2, 127–145 (1968), Corrections in 5, 95–96 (1971)Google Scholar
  16. 16.
    Lämmel, R.: Grammar Testing. In: Hussmann, H. (ed.) FASE 2001. LNCS, vol. 2029, pp. 201–216. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  17. 17.
    Lämmel, R., Harm, J.: Test case characterisation by regular path expressions. In: Brinksma, E., Tretmans, J. (eds.) Proc. Formal Approaches to Testing of Software (FATES 2001), August 2001. Notes Series NS-01-4, pp. 109–124. BRICS (2001)Google Scholar
  18. 18.
    Lei, Y., Tai, K.-C.: In-parameter-order: A test generation strategy for pairwise testing. In: HASE, pp. 254–261. IEEE Computer Society, Los Alamitos (1998)Google Scholar
  19. 19.
    Maurer, P.: Generating test data with enhanced context-free grammars. IEEE Software 7(4), 50–56 (1990)CrossRefGoogle Scholar
  20. 20.
    McKeeman, W.: Differential testing for software. Digital Technical Journal of Digital Equipment Corporation 10(1), 100–107 (1998)Google Scholar
  21. 21.
    Purdom, P.: A sentence generator for testing parsers. BIT 12(3), 366–375 (1972)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Schlimmer, J., et al.: Web Services Policy Framework (September 2004), Available at:
  23. 23.
    Sirer, E.G., Bershad, B.N.: Using production grammars in software testing. In: USENIX (ed.) Proceedings of the 2nd Conference on Domain-Specific Languages (DSL 1999), Austin, Texas, USA, October 3–5, pp. 1–13. USENIX, Berkeley (1999)CrossRefGoogle Scholar
  24. 24.
    Slutz, D.: Massive Stochastic Testing for SQL. Technical Report MSR-TR-98-21, Microsoft Research, Redmond (1998); A shorter form of the paper appeared in the Proc. of the 24th VLDB Conference, New York, USA (1998)Google Scholar
  25. 25.
    Stepney, S.: High Integrity Compilation: A Case Study. Prentice Hall, Englewood Cliffs (1993)MATHGoogle Scholar
  26. 26.
    Tai, K., Lei, Y.: A Test Generation Strategy for Pairwise Testing. IEEE Transactions on Software Engineering 28(1), 109–111 (2002)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Tillmann, N., Schulte, W., Grieskamp, W.: Parameterized Unit Tests. Technical report, Microsoft Research, MSR-TR-2005-64 (2005); also appeared in FSE/ESEC 2005Google Scholar
  28. 28.
    W3C. XML Path Language (XPath) Version 1.0 (November 1999),
  29. 29.
    W3C. XML Schema (2000–2003),
  30. 30.
    Zaytsev, V.V.: Combinatorial test set generation: Concepts, implementation, case study. Master’s thesis, Universiteit Twente, Enschede, The Netherlands (June 2004)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2006

Authors and Affiliations

  • Ralf Lämmel
    • 1
  • Wolfram Schulte
    • 2
  1. 1.Microsoft Corp., Webdata/XMLRedmondUSA
  2. 2.Microsoft Research, FSE LabRedmondUSA

Personalised recommendations