Skip to main content
SpringerLink
Log in

Analysing the effectiveness of rule-coverage as a reduction criterion for test suites of grammar-based software

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The term grammar-based software describes software whose input can be specified by a context-free grammar. This grammar may occur explicitly in the software, in the form of an input specification to a parser generator, or implicitly, in the form of a hand-written parser. Grammar-based software includes not only programming language compilers, but also tools for program analysis, reverse engineering, software metrics and documentation generation. Hence, ensuring their completeness and correctness is a vital prerequisite for their use. In this paper we propose a strategy for the construction of test suites for grammar based software, and illustrate this strategy using the ISO C + +  grammar. We use the concept of grammar-rule coverage as a pivot for the reduction of an implementation-based test suite, and demonstrate a significant decrease in the size of this suite. The effectiveness of this reduced test suite is compared to the original test suite with respect to code coverage and more importantly, fault detection. This work greatly expands upon previous work in this area and utilises large scale mutation testing to compare the effectiveness of grammar-rule coverage to that of statement coverage as a reduction criterion for test suites of grammar-based software. This work finds that when grammar rule coverage is used as the sole criterion for reducing test suites of grammar based software, the fault detection capability of that reduced test suite is greatly diminished when compared to other coverage criteria such as statement coverage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Acostachioaie D (2000) DOC++: Open Source - Open Science - Open systems. Circles Electronic Magazine (36)

  • Andrews JH, Briand LC, Labiche Y, Namin AS (2006) Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans Softw Eng 32(8):608–624

    Article  Google Scholar 

  • Bazzichi F, Spadafora I (1982) An automatic generator for compiler testing. IEEE Trans Softw Eng 8(4):343–353

    Article  Google Scholar 

  • Celentano A, Crespi-Reghizzi S, Vigna PD, Ghezzi C, Granata G, Savoretti F (1980) Compiler testing using a sentence generator. Softw Pract Exp 10(11):897–918

    Article  Google Scholar 

  • Garey MR, Johnson DS (1979) Computers and intractability: a guide to the theory of NP-completeness. W.H. Freeman, San Francisco

    MATH  Google Scholar 

  • Gibbs TH, Malloy BA, Power JF (2003a) Decorating tokens to facilitate recognition of ambiguous language constructs. Softw Pract Exp 33(1):19–39

    Article  MATH  Google Scholar 

  • Gibbs TH, Malloy BA, Power JF (2003b) Progression toward conformance of C + +  language compilers. Dr Dobbs J 28(11):54–60

    Google Scholar 

  • Harm J, Lämmel R (2000) Two-dimensional approximation coverage. Informatica 24(3):355–369

    MATH  Google Scholar 

  • Harrold MJ, Gupta R, Soffa ML (1993) A methodology for controlling the size of a test Suite. ACM Trans Softw Eng Methodol 2(3):270–285

    Article  Google Scholar 

  • Heimdahl MPE, George D (2004) Test suite reduction for model based tests: effects on test quality and implications for testing. In: 19th IEEE international conference on automated software engineering. IEEE, Linz, pp 176–185

    Chapter  Google Scholar 

  • Hennessy M, Power JF (2005a) An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In: 20th IEEE international conference on automated software engineering. IEEE, Long Beach, pp 104–113

    Chapter  Google Scholar 

  • Hennessy M, Power JF (2005b) Generation strategies for test suites of grammar-based software. Technical Report NUIM-CS-TR-2005-02, Department of Computer Science, National Univesity of Ireland, Maynooth

  • Hennessy M, Power JF, Malloy BA (2003) gccXfront : Exploiting gcc as a Front-end for Program Comprehension via XML/XSL. In: 11th international workshop on program comprehension, Portland, OR, 10–11 May 2003

  • ISO/IEC JTC 1 (1998) International standard: programming languages - C + + . No. 14882:1998(E), 1st edn. American National Standards Institute, New York

    Google Scholar 

  • ISO/IEC JTC 1 (2003) International standard: programming languages - C + + . No. 14882:2003(E), 2nd edn. American National Standards Institute, New York

    Google Scholar 

  • Jones JA, Harrold MJ (2003) Test suite reduction and prioritization for modified condition/decision coverage. IEEE Trans Softw Eng Methodol 29(3):195–210

    Article  Google Scholar 

  • Klint P, Lämmel R, Verhoef C (2005) Toward an engineering discipline for grammarware. ACM Trans Softw Eng Methodol 14(3):331–380

    Article  Google Scholar 

  • Koppler R (1997) A systematic approach to fuzzy parsing. Softw Pract Exp 27(6):637–649

    Article  Google Scholar 

  • Lämmel R (2001) Grammar testing. In: Fundamental approaches to software engineering. LNCS, vol 2029. Springer, Berlin Heidelberg New York, pp 201–216

    Chapter  Google Scholar 

  • Ma Y-S, Offutt J, Kwon YR (2005) MuJava : an automated class mutation system. J Softw Test Verif Reliab 15(2):97–133

    Article  Google Scholar 

  • Malloy BA, Linde SA, Duffy EB, Power JF (2002) Testing C + +  compilers for ISO language conformance. Dr Dobbs J 27(6):71–78

    Google Scholar 

  • Malloy BA, Power JF (2001) An interpretation of Purdom’s algorithm for automatic generation of test-cases. In: 1st annual international conference on computer and information science, Orlando, FL, 3–5 October 2001

  • Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99–118

    Article  Google Scholar 

  • Offutt AJ, Alexander RT, Wu Y, Xiao Q, Hutchinson C (2001) A fault model for subtype inheritance and polymorphism. In: 12th international symposium on software reliability engineering, Hong Kong, 27–30 November 2001, pp 84–95

  • Perennial (2007) Perennial test suite for ISO C + + . http://www.peren.com/pages/cppvs.htm. Accessed March 2008

  • Plum Hall (2008) Plum Hall test suites. http://www.plumhall.com/. Accessed March 2008

  • Power JF, Malloy BA (2002) Program annotation in XML: a parser-based approach. In: Working conference on reverse engineering, Richmond, 28 October–1 November 2002, pp 190–198

  • Power JF, Malloy BA (2004) A metrics suite for grammar-based software. Softw Maint Evol Res Pract 16(6):405–426

    Article  Google Scholar 

  • Purdom P (1972) A sentence generator for testing parsers. BIT 12(3):366–375

    Article  MATH  MathSciNet  Google Scholar 

  • Roper M (1994) Software testing. McGraw-Hill, New York

    Google Scholar 

  • Rothermel G, Harrold MJ, Ostrin J, Hong C (1998) An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: International conference on software maintenance, Bethesda, 16–19 November 1998, pp 34–43

  • Sim SE, Holt RC, Easterbrook S (2002) On using a benchmark to evaluate C + +  extractors. In: 10th international workshop on program comprehension, Paris, 27–29 June 2002, pp 114–126

  • Spinczyk O, Gal A, Schröder-Preikschat W (2002) AspectC++: an aspect-oriented extension to C + + . In: 40th international conference on technology of object-oriented languages and systems, Sydney, February 2002, pp 53–60

  • Wong WE, Horgan JR, London S, Mathur AP (1998) Effect of test-set minimization on fault detection effectiveness. Softw Pract Exp 28(4):347–369

    Article  Google Scholar 

Download references

Acknowledgements

This work has been funded under the Embark scheme administered by the Irish Research Council for Science, Engineering and Technology.

Author information

Authors and Affiliations

  1. Computer Science Department, National University of Ireland, Maynooth, Co. Kildare, Ireland

    Mark Hennessy & James F. Power

Authors
  1. Mark Hennessy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James F. Power
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark Hennessy.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hennessy, M., Power, J.F. Analysing the effectiveness of rule-coverage as a reduction criterion for test suites of grammar-based software. Empir Software Eng 13, 343–368 (2008). https://doi.org/10.1007/s10664-008-9067-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-008-9067-7

Keywords

Search

Navigation