A Note on an Anomaly in Black-Box Testing

  • Antti Huima
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3997)


Testing should not reduce confidence in the system under test – unless defects are found. We show that for a general class of finite-state systems this intuition is incorrect. We base our argument on the view of risk as a probability. We calculate the risk of having an invalid implementation, based on a concrete, believable fault model, and show that executing correct test runs can actually decrease confidence in the system under test. This anomaly is important as it explains some of the difficulty in establishing mathematical links between fault models and testing efficiency. The presented anomaly itself is claimed to be independent of the particular structure of systems. We provide critique of the result, and discuss the potential limits of the presented anomaly as well as ways to remedy it.


Fault Model Markov Chain Model Correct Implementation European Telecommunication Standard Institute Nondeterministic Choice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alur, R., Courcoubetis, C., Yannakakis, M.: Distinguishing tests for nondeterministic and probabilistic machines. In: Proc. 27th ACM Symposium on Theory of Computing, pp. 363–372 (1995)Google Scholar
  2. 2.
    European Telecommunications Standards Institute (ETSI). The testing and test control notation version 3; part 1: TTCN-3 core language. ETSI ES 201 873-1, V2.2.1 (February 2003)Google Scholar
  3. 3.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines—a survey. Proceedings of the IEEE 84(8), 1090–1123 (1996)CrossRefGoogle Scholar
  4. 4.
    Luo, G., von Bochmann, G., Petrenko, A.: Test selection based on communicating nondeterministic finite state machines using a generalized wp-method. IEEE Transactions on Software Engineering SE-20(2), 149–162 (1994)Google Scholar
  5. 5.
    Mas-Colell, A., Whinston, M.D., Green, J.R.: Microeconomic Theory. Oxford University Press, Oxford (1995)MATHGoogle Scholar
  6. 6.
    Milton, J.S., Arnold, J.C.: Introduction to probability and statistics. McGraw–Hill, New York (1995)Google Scholar
  7. 7.
    Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology 1(1), 5–20 (1992)CrossRefGoogle Scholar
  8. 8.
    Offutt, A.J., Lee, S.: An empirical evaluation of weak mutation. IEEE Transactions on Software Engineering 20(5), 337–344 (1994)CrossRefGoogle Scholar
  9. 9.
    Peterson, J.L.: Petri Net Theory and the Modeling of Systems. Prentice Hall PTR, Englewood Cliffs (1981)MATHGoogle Scholar
  10. 10.
    Petrenko, A.: Fault model-driven test derivation from finite state models: Annotated bibliography. In: Cassez, F., Jard, C., Rozoy, B., Dermot, M. (eds.) MOVEP 2000. LNCS, vol. 2067, pp. 196–205. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  11. 11.
    Petrenko, A., Yevtushenko, N., Bochman, G.V.: Fault models for testing in context. In: FORTE 1996, pp. 163–178 (1996)Google Scholar
  12. 12.
    Petrenko, A., Yevtushenko, N., Huo, J.L.: Testing transition systems with input and output tester. In: Hogrefe, D., Wiles, A. (eds.) TestCom 2003. LNCS, vol. 2644, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  13. 13.
    Pyhälä, T., Heljanko, K.: Specification coverage aided test selection. In: Lilius, J., Balarin, F., Machado, R.J. (eds.) Proceeding of the 3rd International Conference on Application of Concurrency to System Design (ACSD 2003), pp. 187–195. IEEE Computer Society, Guimaraes, Portugal (2003)CrossRefGoogle Scholar
  14. 14.
    Rimen, M., Ohlsson, J., Torin, J.: On microprocessor error behavior modeling. In: 24th IEEE International Symposium on Fault-Tolerant Computing (1994)Google Scholar
  15. 15.
    Tretmans, J.: A formal approach to conformance testing. In: Proc. 6th International Workshop on Protocols Test Systems, IFIP Transactions, vol. C-19, pp. 257–276 (1994)Google Scholar
  16. 16.
    van der Bijl, M., Rensink, A., Tretmans, J.: Compositional testing with IOCO. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 86–100. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  17. 17.
    Whittaker, J., Thomason, M.: A markov chain model for statistical software testing. IEEE Transactions on Software Engineering 20(10), 812–824 (1994)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Antti Huima
    • 1
  1. 1.Conformiq Software Ltd.Finland

Personalised recommendations