Skip to main content

What Is a Test Case? Revisiting the Software Test Case Concept

  • Conference paper
Software Process Improvement (EuroSPI 2009)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 42))

Included in the following conference series:

Abstract

Since the 1980s the term "Test Case" (TC) has been recognized as a building block for describing testing items, widely used as a work unit, metric and documentation entity. In light of the centrality of the TC concept in testing processes, the questions this paper attempts to answer are: What are the uses of TC in software testing? Is there a general, commonly agreed-upon definition of a TC? If not, what are the implications of this situation?

This article reviews and explores the history, use and definitions of TCs, showing that while extensively used in research and practice, there is no one formal agreed upon definition of a TC. In this paper we point at undesirable implications of this situation, suggest four criteria for a ’good’ TC definition, and discuss the benefits accrued from such a definition. We conclude by urging the academic and professional community to formalize a TC definition for the benefits of the industry and its customers, and strongly believe that this review paves the way to articulating a formal TC definition. Such a definition, when widely accepted, will clarify some of the ambiguity currently associated with TC interpretation, hence with software testing assessment which relies on TCs as metrics. Furthermore, a formal definition can advance automation of TC generation and management.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Tassey, G.: The Economic Impacts of Inadequate Infrastructure for Software Testing. National Institute of Standards and Technology (2002)

    Google Scholar 

  2. Basili, V., Caldiera, G., McGarry, F., Pajerski, R., Page, G., Waligora, S.: The software engineering laboratory: an operational software experience factory. In: Proceedings of the 14th international conference on Software engineering. ACM, New York (1992)

    Google Scholar 

  3. Geras, A.M., Smith, M.R., Miller, J.: A survey of software testing practices in alberta. Canadian Journal of Electrical and Computer Engineering 29(3), 183–191 (2004)

    Article  Google Scholar 

  4. Grindal, M., Offutt, J., Mellin, J.: On the Testing Maturity of Software Producing Organizations. In: Proceedings of the Testing: Academic and Industrial Conference-Practice and Research Techniques TAIC PART (2006)

    Google Scholar 

  5. Ng, S.P., Murnane, T., Reed, K., Grant, D., Chen, T.Y.: A Preliminary Survey on Software Testing Practices in Australia. In: Software Engineering Conference (2004)

    Google Scholar 

  6. Illes, T., Herrmann, A., Paech, B., Rockert, J.: Criteria for Software Testing Tool Evaluation. A Task Oriented View. In: Proceedings of the 3rd World Congress for Software Quality (2005)

    Google Scholar 

  7. Almog, D.: Verification Points for Better Testing Efficiency. In: StarEastSQE (2007)

    Google Scholar 

  8. Bach, J.: Exploratory Testing Explained, http://www.satisfice.com/articles/et-article.pdf

  9. Kaner, C.: The Ongoing Revolution in Software Testing. In: Software Test & Performance Conference (2004)

    Google Scholar 

  10. Gelperin, D., Hetzel, B.: The Growth of Software Testing. Communications of the ACM 31(6), 687–695 (1988)

    Article  Google Scholar 

  11. Ostrand, T.J., Balcer, M.J.: The Category-Partition Method for Specifying and Generating Functional Tests. Commun. ACM 31(6), 676–686 (1988)

    Article  Google Scholar 

  12. Weyuker, E.J.: The Cost of Data Flow Testing: An Empirical Study. IEEE Transactions on Software Engineering 16(2), 121–128 (1990)

    Article  MathSciNet  Google Scholar 

  13. Harrold, M.J., Rajiv, G., Mary Lou, S.: A Methodology for Controlling the Size of a Test Suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993)

    Article  Google Scholar 

  14. Rosenberg, L., Hammer, T.F., Huffman, L.L.: Requirements, Testing and Metrics. In: 15th Annual Pacific Northwest Software Quality Conference (1998)

    Google Scholar 

  15. Iberle, K.: Divide and Conquer: Making Sense of Test Planning. In: The International Conference on Software Testing, Analysis and Review, STARWEST (1999)

    Google Scholar 

  16. Aichernig, B.K.: Test-Case Calculation through Abstraction. In: International Symposium of Formal Methods. Springer, Heidelberg (2001)

    Google Scholar 

  17. Jorgensen, P.: Software Testing: A Craftsman’s Approach. CRC Press, Boca Raton (2002)

    Book  MATH  Google Scholar 

  18. Kaner, C.: What Is a Good Test Case? In: Star East (2003)

    Google Scholar 

  19. Grindal, M., Offutt, J., Andler, S.F.: Combination Testing Strategies: a Survey. Software Testing Verification and Reliability 15(3), 167 (2005)

    Article  Google Scholar 

  20. Grindal, M., Lindstrom, B., Offutt, J., Andler, S.F.: An Evaluation of Combination Strategies for Test Case Selection. Empirical Software Engineering 11(4), 583–611 (2006)

    Article  Google Scholar 

  21. Ambrosio, A., Mattiello-Francisco, F., Santiago, V., Silva, W., Martins, E.: Designing Fault Injection Experiments Using State-Based Model to Test a Space Software. In: Bondavalli, A., Brasileiro, F., Rajsbaum, S. (eds.) LADC 2007. LNCS, vol. 4746, pp. 170–178. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  22. Ambrosio, A.M., Martins, E., Vijaykumar, N.L., de Carvalho, S.V.: Systematic Generation of Test and Fault Cases for Space Application Validation. In: DASIA: Data Systems in Aerospace, European Space Agency (2005)

    Google Scholar 

  23. Desai, H.D.: Test Case Management System (TCMS). In: IEEE Conference Global Telecommunications GLOBECOM: ’Communications: The Global Bridge’ (1994)

    Google Scholar 

  24. Craig, R.D., Jaskiel, S.P.: Systematic Software Testing. Artech House (2002)

    Google Scholar 

  25. Jorgensen: Testing with Hostile Data Streams. ACM Sigsoft Software Engineering Notes 28(2), 1 (2003)

    Article  Google Scholar 

  26. Nagappan, N.: A Software Testing and Reliability Early Warning (STREW) Metric Suite, Thesis: Computer Science, North Carolina University (2005)

    Google Scholar 

  27. Sherriff, M., Boehm, B.W., Williams, L., Nagappan, N.: An Empirical Process for Building and Validating Software Engineering Parametric Models. North Carolina State Univeristy CSC-TR-2005-45, October, 19 (2005)

    Google Scholar 

  28. Hoffman, D.: The Darker Side of Metrics. In: Conference of the Association of Software Testing, CAST (2006)

    Google Scholar 

  29. Binder, R.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley Professional, Reading (2000)

    Google Scholar 

  30. Talby, D., Hazzan, O., Dubinsky, Y., Keren, A.: Agile Software Testing in a Large-Scale Project. IEEE Software, 30–37 (2006)

    Google Scholar 

  31. Beck, K.: Test-driven Development: By Example. Addison-Wesley Professional, Reading (2003)

    Google Scholar 

  32. Utting, M., Legeard, B., Pretschner, A.: A Taxonomy of Model-based Testing. Dept. of Computer Science, University of Waikato Hamilton, New Zealand (2006)

    Google Scholar 

  33. Bohnet, R., Meszaros, G.: Test-Driven Porting. In: Proceedings of the Agile Development Conference (2005)

    Google Scholar 

  34. Lewis, G.A., Morris, E., Simanta, S., Wrage, L.: Common Misconceptions about Service-Oriented Architecture. In: Proceedings of the Sixth International IEEE Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems (2007)

    Google Scholar 

  35. Hiebert, D., Klaedtke, R.A., Lowery, E., Nartovich, A., Raut, N., Sandberg, M.J.: Building SOA-based Solutions for IBM System i Platform. IBM (2007)

    Google Scholar 

  36. Karam, M., Safa, H., Artail, H.: An Abstract Workflow-Based Framework for Testing Composed Web Services. In: IEEE/ACS International Conference on Computer Systems and Applications, AICCSA (2007)

    Google Scholar 

  37. Rehman, M.J., Jabeen, F., Bertolino, A., Polini, A.: Testing Software Components for Integration: A Survey of Issues and Techniques. Software Testing, Verification & Reliability 17(2), 95–133 (2007)

    Article  Google Scholar 

  38. Weyuker, E.J.: Testing Component-Based Software: A Cautionary Tale. IEEE Software 15(5), 54–59 (1998)

    Article  Google Scholar 

  39. Cai, K.Y., Zhao, L., Hu, H., Jiang, C.H.: On the Test Case Definition for GUI Testing. In: Fifth International Conference on Quality Software, QSIC (2005)

    Google Scholar 

  40. Boujarwah, A.S., Saleh, K.: Compiler Test Case Generation Methods: A Survey and Assessment. Information and Software Technology 39(9), 617–625 (1997)

    Article  Google Scholar 

  41. Calam, J.R., Ioustinova, N., Pol, J.: Towards Automatic Generation of Parameterized Test Cases from Abstractions. Technical Report SEN-E0602, Centrum voor Wiskunde en Informatica (2006)

    Google Scholar 

  42. Byers, D., Engstrom, M., Kamkar, M.: The Design of a Test Case Definition Language. Automated and Algorithmic Debugging, 69–78 (1997)

    Google Scholar 

  43. Nebut, C., Fleurey, F., Le Traon, Y., Jezequel, J.M.: Automatic Test Generation: A Use Case Driven Approach. IEEE Transactions on Software Engineering, 140–155 (2006)

    Google Scholar 

  44. Prasanna, M., Sivanandam, S.N., Venkatesan, R., Sundarrajan, R.: A Survey on Automatic Test Case Generation. Academic Open Internet Journal 15 (2005)

    Google Scholar 

  45. Coulter, A.C.: Graybox Software Testing Methodology: Embedded Software Testing Technique. In: Proceedings of the18th Digital Avionics Systems Conference (1999)

    Google Scholar 

  46. Craig, R.: Measurement and Metrics for Test Managers. In: STAR East. SQE (2007)

    Google Scholar 

  47. Kaner, C.: Measurement Issues and Software Testing (2001)

    Google Scholar 

  48. Landis, L., Waligora, S., McGarry, F.: Recommended Approach to Software Development. Software Engineering Laboratory Series, pp. 81–305. NASA (1992)

    Google Scholar 

  49. Bassin, K., Biyani, S., Santhanam, P.: Metrics to Evaluate Vendor-Developed Software Based on Test Case Execution Results. IBM Systems Journal 41(1), 13–30 (2002)

    Article  Google Scholar 

  50. Jorgensen, M., Shepperd, M.: A Systematic Review of Software Development Cost Estimation Studies. IEEE Transactions on Software Engineering 33(1), 33–53 (2007)

    Article  Google Scholar 

  51. Binkley, D.: Semantics Guided Regression Test Cost Reduction. IEEE Transactions on Software Engineering 23(8), 498–516 (1997)

    Article  Google Scholar 

  52. Leung, H.K.N., White, L.: Insights into Regression Testing [software testing]. In: Conference on Software Maintenance (1989)

    Google Scholar 

  53. Nageswaran, S.: Test Effort Estimation Using Use Case Points. In: 14th International Internet & Software Quality Week (2001)

    Google Scholar 

  54. Aranha, E., Borba, P.: An Estimation Model for Test Execution Effort. In: International Symposium on Empirical Software Engineering and Measurement, ESEM 2007 (2007)

    Google Scholar 

  55. Abran, A., Bourque, P., Dupuis, R., Moore, J.W.: Guide to the Software Engineering Body of Knowledge - SWEBOK. In: Alain, A., et al. (eds.). IEEE Press, Los Alamitos (2004)

    Google Scholar 

  56. Stocks, P.A., Carrington, D.A.: Test Templates: A Specification-Based Testing Framework. In: Proceedings of the 15th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos (1993)

    Google Scholar 

  57. Beizer, B.: Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Inc., Chichester (1995)

    Google Scholar 

  58. Offutt, J., Abdurazik, A.: Generating tests from UML specifications. In: Proc. Second International Conference on the Unified Modeling Language (1999)

    Google Scholar 

  59. Taylor, C.M.: EPDAV – A Model for Test Case Definition. In: Conference of the Association of Software Testing (2006)

    Google Scholar 

  60. Bruno, M., Canfora, G., Di Penta, M., Esposito, G., Mazza, V.: Using Test Cases as Contract to Ensure Service Compliance Across Releases. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 87–100. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  61. Kaner, C., Falk, J.L., Nguyen, H.Q.: Testing Computer Software. John Wiley & Sons, Inc., New York (1999)

    MATH  Google Scholar 

  62. Maletic, J.I., Soliman, K.S., Moreno, M.A., Mercer, W.M.: Identification of Test Cases from Business Requirements of Software Systems. In: American Conference on Information Systems AMCIS (1999)

    Google Scholar 

  63. Baudry, B., Fleurey, F., Jezequel, J.M., Le Traon, Y.: Genes and Bacteria for Automatic Test Cases Optimization in the .NET Environment. In: Proceedings of the13th International Symposium on Software Reliability Engineering, ISSRE (2002)

    Google Scholar 

  64. Menter, F.: CFD Best Practice Guidelines for CFD Code Validation for Reactor- Safety Applications. CFX, Germany (2002)

    Google Scholar 

  65. Mikhailova, A., Doche, M., Butler, M.: Contracts for Scenario-Based Testing of Object-Oriented Programs (2002)

    Google Scholar 

  66. Bruno, M., Canfora, G., Di Penta, M., Esposito, G., Mazza, V.: Using Test Cases as Contract to Ensure Service Compliance Across Releases. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 87–100. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Almog, D., Heart, T. (2009). What Is a Test Case? Revisiting the Software Test Case Concept. In: O’Connor, R.V., Baddoo, N., Cuadrago Gallego, J., Rejas Muslera, R., Smolander, K., Messnarz, R. (eds) Software Process Improvement. EuroSPI 2009. Communications in Computer and Information Science, vol 42. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04133-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-04133-4_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-04132-7

  • Online ISBN: 978-3-642-04133-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics