Skip to main content
SpringerLink
Log in

An evaluation of combination strategies for test case selection

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ammann P, Offutt AJ (1994) Using formal methods to derive test frames in category-partition testing. Proceedings of the Ninth Annual Conference on Computer Assurance (COMPASS’94), Gaithersburg MD, IEEE Computer Society Press, pp 69–80

  • Anderson T, Avizienis A, Carter W, Costes A, Christian F, Koga Y, Kopetz H, Lala J, Laprie J, Meyer, J Randell B, Robinson A (1994) Dependability: basic concepts and terminology. Technical report, IFIP. WG 10.4

  • Archibald R (1992) Managing high-technology programs and projects. John Wiley and Sons, Inc.

  • Bache R (1997) The effect of fault size on testing. The Journal of Software Testing, Verification, and Reliability 7: 139–152

    Article  Google Scholar 

  • Basili VR, Rombach HD (1988) The TAME project: Towards improvement-oriented software environments. IEEE Transactions on Software Engineering SE-14(6): 758–773

    Article  Google Scholar 

  • Basili V, Selby R (1987) Comparing the effectiveness of software testing strategies. IEEE Transactions on Software Engineering SE-13(12): 1278–1296

    Google Scholar 

  • Basili V, Shull F, Lanubile F (1999) Using experiments to build a body of knowledge. Perspectives of System Informatics, Third International Andrei Ershov Memorial Conference (PSI 99). Akademgorodok, Novosibirsk, Russia, 6-9 July 1999, Proceedings, pp 265–282

  • Briand L, Pfahl D (1999) Using simulation for assessing the real impact of test coverage on defect coverage. Proceedings of the International Conference on Software Maintenance (ICSM99), Oxford, The UK, 30th of Aug - 3rd Sept 1999, pp 475–482

  • Brownlie R, Prowse J, Phadke M (1992) Robust testing of AT&T PMX/StarMAIL using OATS. AT&T technical Journal 71(3): 41–47

    Google Scholar 

  • Burr K, Young W (1998) Combinatorial test techniques: Table-based automation, test generation and code coverage. Proceedings of the International Conference on Software Testing, Analysis, and Review (STAR’98). San Diego, CA, USA, pp 26–28

  • Cohen D, Dalal S, Kajla A, Patton G (1994) The automatic efficient test generator (AETG) system. Proceedings of Fifth International Symposium on Software Reliability Engineering (ISSRE’94), Los Alamitos, California, USA, 6-9 November 1994, IEEE Computer Society, pp 303–309

  • Cohen D, Dalal S, Parelius J, Patton G (1996) The combinatorial design approach to automatic test generation. IEEE Software 13(5): 83–89

    Article  Google Scholar 

  • Cohen D, Dalal S, Fredman M, Patton G (1997) The AETG System: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering 23(7): 437–444

    Article  Google Scholar 

  • Cohen M, Gibbons P, Mugridge W, Colburn C (2003) Constructing test cases for interaction testing, Proceedings of the 25th International Conference on Software Engineering, (ICSE’03), Portland, Oregon, USA, 3-10 May 2003, IEEE Computer Society, pp 38–48

  • Dalal S, Mallows C (1998) Factor-covering designs for testing software. Technometrics 50(3): 234–243

    Article  Google Scholar 

  • Dunietz I, Ehrlich W, Szablak B, Mallows C, Iannino A (1997) Applying design of experiments to software testing. Proceedings of 19th International Conference on Software Engineering (ICSE’97). Boston, MA, USA 1997, ACM, pp 205–215

  • Frankl P, Hamlet R, Littlewood B, Stringini L (1998) Evaluating testing methods by delivered reliability. IEEE Transactions on Software Engineering 24: 586–601

    Article  Google Scholar 

  • Grindal M, Lindström B, Offutt AJ, Andler SF (2003) An evaluation of combination strategies for test case selection. Technical Report. Technical Report HS-IDA-TR-03-001. Department of Computer Science, University of Skövde

  • Grindal M, Offutt AJ, Andler SF (2005) Combination testing strategies: A survey. Software Testing, Verification, and Reliability 15(3): 167–199

    Article  Google Scholar 

  • Harman M, Hierons R, Holocombe M, Jones B, Reid S, Roper M, Woodward, M (1999) Towards a maturity model for empirical studies of software testing. Proceedings of the 5th Workshop on Empirical Studies of Software Maintenance (WESS’99). Keble College, Oxford, UK

  • Kamsties E, Lott C (1995a) An empirical evaluation of three defect detection techniques. Technical Report ISERN 95-02. Dept of Computer Science, University of Kaiserslauten

  • Kamsties E, Lott C (1995b) An empirical evaluation of three defect detection techniques. Proceedings of the 5th European Software Engineering Conference (ESEC95). Sitges, Barcelona, Spain, 25-28 September 1995

  • Kropp N, Koopman P, Siewiorek D (1998) Automated robustness testing of off-the-shelf software components. Proceedings of FTCS’98: Fault Tolerant Computing Symposium, 23–25 June 1998 Munich, Germany. IEEE, pp 230–239

  • Lei Y, Tai K (1998) In-parameter-order: A test generation strategy for pair-wise testing. Proceedings of the third IEEE High Assurance Systems Engineering Symposium, IEEE, pp 254–261

  • Lei Y, Tai K (2001) A test generation strategy for pairwise testing. Technical Report TR-2001-03. Department of Computer Science, North Carolina State University, Raleigh

  • Lindström B, Grindal M, Offutt J (2004) Using an existing suite of test objects: Experience from a testing experiment. Workshop on Empirical Research in Software Testing, ACM SIGFOST Software Engineering Notes. Boston, MA, USA

  • Lott C (n.d.) A repeatable software experiment. URL:www.chris-lott.org/work/exp/

  • Lott C, Rombach H (1996) Repeatable software engineering experiments for comparing defect-detection techniques. Journal of Empirical Software Engineering 1(3): 241–277

    Article  Google Scholar 

  • Mandl R (1985) Orthogonal latin squares: An application of experiment design to compiler testing. Communications of the ACM 28(10): 1054–1058

    Article  Google Scholar 

  • Miller J, Roper M, Wood M, Brooks A (1995) Towards a benchmark for the evaluation of software testing techniques. Information and Software Technology 37(1): 5–13

    Article  Google Scholar 

  • Musa J (1993) Operational profiles in software-reliability engineering. IEEE Software 10: 14–32

    Article  Google Scholar 

  • Myers G (1979) The Art of Software Testing, John Wiley and Sons

  • Ntafos SC (1984) On required element testing. IEEE Transactions on Software Engineering SE-10: 795–803

    Article  Google Scholar 

  • Offutt AJ, Hayes JH (1996) A semantic model of program faults. Proceedings of the 1996 International Symposium on Software Testing, and Analysis. ACM Press, San Diego, CA, pp 195–200

  • Offutt AJ, Xiong Y, Liu S (1999) Criteria for generating specification-based tests. Fifth IEEE International Conference on Engineering of Complex Systems (ICECCS’99). Las Vegas NV, IEEE, pp 119–129

  • Ostrand T, Balcer M (1988) The category-partition method for specifying and generating functional tests. Communications of the ACM 31(6): 676–686

    Article  Google Scholar 

  • Piwowarski P, Ohba M, Caruso, J (1993) Coverge measure experience during function test. Proceedings of 14th International Conference on Software Engineering (ICSE’93). Los Alamitos, CA, USA 1993, ACM, pp 287–301

  • Reid S (1997) An empirical analysis of equivalence partitioning, boundary value analysis and random testing. Proceedings of the 4th International Software Metrics Symposium (METRICS’97), Albaquerque, New Mexico, USA, 5–7 Nov 1997, IEEE, pp 64–73

  • Shiba T, Tsuchiya T, Kikuno T ( 2004) Using artificial life techniques to generate test cases for combinatorial testing. Proceedings of 28th Annual International Computer Software and Applications Conference (COMPSAC’04) 2004. Hong Kong, China, IEEE Computer Society, 28–30 September 2004, pp 72–77

  • So S, Cha S, Shimeall T, Kwon Y (2002) An empirical evaluation of six methods to detect faults in software. Software Testing, Verification and Reliability 12(3): 155–171

    Article  Google Scholar 

  • Wallace DR, Kuhn D (2001) Failure modes in medical device software: An analysis of 15 years of recall data. International Journal of Reliability, Quality, and Safety Engineering 8(4): 351–371

    Article  Google Scholar 

  • Williams A (2000) Determination of test configurations for pair-wise interaction coverage. Proceedings of the 13th International Conference on the Testing of Communicating Systems (TestCom 2000). Ottawa, Canada, August 2000, pp 59–74

  • Williams A, Probert R (1996) A practical strategy for testing pair-wise coverage of network interfaces. Proceedings of the 7th International Symposium on Software Reliability Engineering (ISSREt’96). White Plains, New York, USA, Oct 30 – Nov 2 1996, pp 246–254

  • Wood M, Roper M, Brooks A, Miller J (1997) Comparing and combining software defect detection techniques: A replicated study. Proceedings of the Sixth European Software Engineerng Conference (ESEC/FSE 97). Lecture Notes in Computer Science Nr. 1013, Springer Verlag, pp 262–277

  • Woodward MR, Al-Khanjari ZA (2000) Testability, fault size and the domain-to-range ratio: An eternal triangle. Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA ’00). ACM Press, Portland, Oregon, USA, pp 168–172

  • Yin H, Lebne-Dengel Z, Malaiya Y (1997) Automatic test generation using checkpoint encoding and antirandom testing. Technical Report CS-97-116. Colorado State University

  • Zweben S, Heym W (1992) Systematic Testing of Data Abstractions Based on Software Specifications. Journal of Software Testing, Verification, and Reliability 1(4): 39–55

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Humanities and Informatics, University of Skövde, Skövde, Sweden

    Mats Grindal, Birgitta Lindström & Sten F. Andler

  2. Department of Information and Software Engineering, George Mason University, Fairfax, 22030, USA

    Jeff Offutt

Authors
  1. Mats Grindal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Birgitta Lindström
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeff Offutt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sten F. Andler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mats Grindal.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Grindal, M., Lindström, B., Offutt, J. et al. An evaluation of combination strategies for test case selection. Empir Software Eng 11, 583–611 (2006). https://doi.org/10.1007/s10664-006-9024-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-006-9024-2

Keywords

Search

Navigation