A Note on an Anomaly in Black-Box Testing
Testing should not reduce confidence in the system under test – unless defects are found. We show that for a general class of finite-state systems this intuition is incorrect. We base our argument on the view of risk as a probability. We calculate the risk of having an invalid implementation, based on a concrete, believable fault model, and show that executing correct test runs can actually decrease confidence in the system under test. This anomaly is important as it explains some of the difficulty in establishing mathematical links between fault models and testing efficiency. The presented anomaly itself is claimed to be independent of the particular structure of systems. We provide critique of the result, and discuss the potential limits of the presented anomaly as well as ways to remedy it.
KeywordsFault Model Markov Chain Model Correct Implementation European Telecommunication Standard Institute Nondeterministic Choice
Unable to display preview. Download preview PDF.
- 1.Alur, R., Courcoubetis, C., Yannakakis, M.: Distinguishing tests for nondeterministic and probabilistic machines. In: Proc. 27th ACM Symposium on Theory of Computing, pp. 363–372 (1995)Google Scholar
- 2.European Telecommunications Standards Institute (ETSI). The testing and test control notation version 3; part 1: TTCN-3 core language. ETSI ES 201 873-1, V2.2.1 (February 2003)Google Scholar
- 4.Luo, G., von Bochmann, G., Petrenko, A.: Test selection based on communicating nondeterministic finite state machines using a generalized wp-method. IEEE Transactions on Software Engineering SE-20(2), 149–162 (1994)Google Scholar
- 6.Milton, J.S., Arnold, J.C.: Introduction to probability and statistics. McGraw–Hill, New York (1995)Google Scholar
- 11.Petrenko, A., Yevtushenko, N., Bochman, G.V.: Fault models for testing in context. In: FORTE 1996, pp. 163–178 (1996)Google Scholar
- 13.Pyhälä, T., Heljanko, K.: Specification coverage aided test selection. In: Lilius, J., Balarin, F., Machado, R.J. (eds.) Proceeding of the 3rd International Conference on Application of Concurrency to System Design (ACSD 2003), pp. 187–195. IEEE Computer Society, Guimaraes, Portugal (2003)CrossRefGoogle Scholar
- 14.Rimen, M., Ohlsson, J., Torin, J.: On microprocessor error behavior modeling. In: 24th IEEE International Symposium on Fault-Tolerant Computing (1994)Google Scholar
- 15.Tretmans, J.: A formal approach to conformance testing. In: Proc. 6th International Workshop on Protocols Test Systems, IFIP Transactions, vol. C-19, pp. 257–276 (1994)Google Scholar