Skip to main content
SpringerLink
Log in

Information gain of black-box testing

  • Original Article
  • Published:
Formal Aspects of Computing

Abstract

For model-based black-box testing, test cases are often selected from the syntactic appearance of the specification of the system under test, according to a pre-given test data adequacy criterion. We introduce a novel approach that is semantics-based, independent of the syntactic appearance of the system specification. Basically, we model the system under test as a random variable, whose sample space consists of all possible behavior sets (with respect to the specification) over the known interface of the black-box. The entropy of the system is measured as the (Shannon) entropy of the random variable. In our criterion, the coverage of a test set is measured as the expected amount of entropy decrease (i.e. the expected amount of information gained) once the test set is run. Since our criterion is syntactic independent, we study the notion of information-optimal software testing where, within a given constraint, a test set is selected to gain the most information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ammann P, Offutt J (2008) Introduction to Software Testing. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  2. Ammann P, Offutt J, Huang H (2003) Coverage criteria for logical expressions. In: ISSRE ’03: Proceedings of the 14th International Symposium on Software Reliability Engineering. IEEE Computer Society, USA, pp 99–107

  3. Andrés C, Llana L, Rodríguez I (2009) Formally transforming user-model testing problems into implementer-model testing problems and viceversa. J Log Algebraic Program 78(6): 425–453

    Article  MATH  Google Scholar 

  4. Beizer B (1995) Black-box testing: techniques for functional testing of software and systems. Wiley, New York

    Google Scholar 

  5. Broy M, Jonsson B, Katoen J-P, Leucker M, Pretschner A (2005) Model-based testing of reactive systems: advanced lectures. In: Lecture notes in computer science, vol 3472. Springer, Berlin

  6. Clarke EM, Grumberg O, Peled DA (1999) Model checking. The MIT Press, Cambridge

    Google Scholar 

  7. Cover TM, Thomas JA (2006) Elements of information theory, 2nd edn. Wiley, New York

    MATH  Google Scholar 

  8. Duran JW, Ntafos S (1984) An evaluation of random testing. IEEE Trans Softw Eng 10(4): 438–444

    Article  Google Scholar 

  9. Gaudel MC (1995) Testing can be formal, too. In: TAPSOFT ’95: Proceedings of the 6th international joint conference CAAP/FASE on theory and practice of software development, 1995. Lecture notes in computer science, vol 915. Springer, Berlin, pp 82–96

  10. Goodenough JB, Gerhart SL (1975) Toward a theory of test data selection. In: Proceedings of the international conference on reliable software. ACM, New York, pp 493–510

  11. Harel D (1987) Statecharts: a visual formalism for complex systems. Sci Comput Program 8(3): 231–274

    Article  MathSciNet  MATH  Google Scholar 

  12. Harel D, Pnueli A (1985) On the development of reactive systems. In: Apt KR (ed) Logics and models of concurrent systems, vol 13. NATO ASI Series. Springer, New York, pp 477–498

    Google Scholar 

  13. Heitmeyer C (2002) Software cost reduction. Encyclopedia of software engineering, 2nd edn. Wiley, New York

    Google Scholar 

  14. Holzmann GJ (1997) The model checker SPIN. IEEE Trans Softw Eng 23(5): 279–295

    Article  MathSciNet  Google Scholar 

  15. Hopcroft JE, Motwani R, Ullman JD (2007) Introduction to automata theory, languages, and computation, 3rd edn. Addison-Wesley, Reading

    Google Scholar 

  16. Ladyman J (2007) Physics and computation: the status of landauer’s principle. In: CiE ’07: Proceedings of the 3rd conference on computability in Europe, 2007. Lecture notes in computer science, vol 4497. Springer, Berlin, pp 446–454

  17. Landauer R (1961) Irreversibility and heat generation in the computing process. IBM J Res Dev 5: 183–191

    Article  MathSciNet  MATH  Google Scholar 

  18. Last M, Friedman M, Kandel A (2003) The data mining approach to automated software testing. In: KDD ’03: Proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 388–396

  19. Lynch NA, Tuttle MR (1989) An introduction to input/output automata. CWI-Q 2(3): 219–246

    MathSciNet  MATH  Google Scholar 

  20. Myers GJ, Sandler C (2004) The art of software testing. Wiley, New York

    Google Scholar 

  21. Nachmanson L, Veanes M, Schulte W, Tillmann N, Grieskamp W (2004) Optimal strategies for testing nondeterministic systems. SIGSOFT Softw Eng Notes 29(4): 55–64

    Article  Google Scholar 

  22. Peled D, Vardi MY, Yannakakis M (2001) Black box checking. J Autom Lang Comb 7(2): 225–246

    MathSciNet  Google Scholar 

  23. Petrenko A, Yevtushenko N, Huo J (2003) Testing transition systems with input and output testers. In: TestCom ’03, 2003. Lecture notes in computer science, vol 2644. Springer, Berlin, pp 129–145

  24. Pnueli A (1977) The temporal logic of programs. In: SFCS ’77: Proceedings of the 18th annual symposium on foundations of computer science. IEEE Computer Society, USA, pp 46–57

  25. Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27: 379–423, 623656

    MathSciNet  MATH  Google Scholar 

  26. Tan L, Sokolsky O, Lee I (2004) Specification-based testing with linear temporal logic. In: IRI’04: proceedings of IEEE internation conference on information reuse and integration. IEEE Computer Society, USA, pp 483–498

  27. Tretmans J (2008) Model based testing with labelled transition systems. In: Formal methods and testing, 2008. Lecture notes in computer science, vol 4949. Springer, Berlin, pp 1–38

  28. Tretmans J, Brinksma E (2003) Torx: automated model-based testing. In: First European conference on model-driven software engineering. N/A, pp 31–43

  29. van der Bijl M, Peureux F (2005) I/O-automata based testing. In: Model-based testing of reactive systems, Chapt. 7. Lecture notes in computer science, vol 3472, Springer, Berlin, pp 173–200

  30. Weyuker EJ, Jeng B (1991) Analyzing partition testing strategies. IEEE Trans Softw Eng 17(7): 703–711

    Article  Google Scholar 

  31. Xie G (2005) Fundamental studies on automatic verification of component-based systems—a decompositional and hybrid approach. PhD thesis, Washington State University

  32. Xie G, Dang Z (2006) Testing systems of concurrent black-boxes—an automata-theoretic and decompositional approach. In: FATES’05: Proceedings of the 5th international workshop on formal approaches to software testing, 2006. Lecture notes in computer science, vol 3997. Springer, Berlin, pp 170–186

  33. Yang L, Dang Z, Fischer TR (2009) Optimal software testing—a cooling down process. In: FCS’09: Proceedings of the international conference on foundations of computer science. CSREA Press, pp 162–168

  34. Zhu H, Hall PAV (1992) Test data adequacy measurement. Softw Eng J 8(1): 21–29

    Google Scholar 

  35. Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4): 366–427

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA, 99164, USA

    Linmin Yang, Zhe Dang & Thomas R. Fischer

Authors
  1. Linmin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhe Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas R. Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linmin Yang.

Additional information

Jim Woodcock

A 7-page abstract of this paper [YDF05] is in the conference proceedings of FCS’09.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yang, L., Dang, Z. & Fischer, T.R. Information gain of black-box testing. Form Asp Comp 23, 513–539 (2011). https://doi.org/10.1007/s00165-011-0175-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00165-011-0175-6

Keywords

Search

Navigation