Abstract
We replicated a controlled experiment first run in the early 1980's to evaluate the effectiveness and efficiency of 50 student subjects who used three defect-detection techniques to observe failures and isolate faults in small C programs. The three techniques were code reading by stepwise abstraction, functional (black-box) testing, and structural (white-box) testing. Two internal replications showed that our relatively inexperienced subjects were similarly effective at observing failures and isolating faults with all three techniques. However, our subjects were most efficient at both tasks when they used functional testing. Some significant differences among the techniques in their effectiveness at isolating faults of different types were seen. These results suggest that inexperienced subjects can apply a formal verification technique (code reading) as effectively as an execution-based validation technique, but they are most efficient when using functional testing.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
References
Victor R. Basili and Richard W. Selby. Comparing the effectiveness of software testing techniques. IEEE Transactions on Software Engineering, 13(12):1278–1296, December 1987.
Victor R. Basili, Richard W. Selby, and David H. Hutchens. Experimentation in software engineering. IEEE Transactions on Software Engineering, SE-12(7):733–743, July 1986.
G. E. P. Box, W. G. Hunter, and J. S. Hunter. Statistics for Experimenters. John Wiley & Sons, New York, 1978.
Ruven E. Brooks. Studying programmer behavior experimentally: The problems of proper methodology. Communications of the ACM, 23(4):207–213, April 1980.
Bill Curtis. Measurement and experimentation in software engineering. Proceedings of the IEEE, 68(9): 1144–1157, September 1980.
J. Daly, A. Brooks, J. Miller, M. Roper, and M. Wood. Verification of results in software maintenance through external replication. In Hausi A. Müller and Mari Georges, editors, Proceedings of the International Conference on Software Maintenance, pages 50–57. IEEE Computer Society Press, September 1994.
W. H. Hetzel. An Experimental Analysis of Program Verification Methods. PhD thesis, University of North Carolina at Chapel Hill, 1976.
William E. Howden. An evaluation of the effectiveness of symbolic testing. Software-Practice and Experience, 8(4):381–398, July/August 1978.
William E. Howden. Functional program testing. IEEE Transactions on Software Engineering, SE-6:162–169, March 1980.
Watts H. Humphrey. A Discipline for Software Engineering. Addison-Wesley, 1995.
Institute of Electrical and Electronics Engineers. Standard Glossary of Software Engineering Terminology, 1983.
Charles M. Judd, Eliot R. Smith, and Louise H. Kidder. Research Methods in Social Relations. Holt, Rinehart and Winston, 6th edition, 1991.
Erik Kamsties and Christopher M. Lott. An empirical evaluation of three defect-detection techniques. Technical Report ISERN 95-02, Department of Computer Science, University of Kaiserslautern, 67653 Kaiserslautern, Germany, May 1995.
Richard C. Linger, Harlan D. Mills, and Bernard I. Witt. Structured Programming: Theory and Practice. Addison-Wesley Publishing Company, 1979.
Brian Marick. The Craft of Software Testing. Prentice Hall, 1994.
Glenford J. Myers. A controlled experiment in program testing and code walkthroughs / inspections. Communications of the ACM, 21(9):760–768, September 1978.
Glenford J. Myers. The Art of Software Testing. John Wiley & Sons, New York, 1979.
H. M. Parsons. What happened at Hawthorne? Science, 183(4128):922–932,March 1974.
Karl R. Popper. Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge and Keegan Paul, London, 5th edition, 1992.
Adam A. Porter and Lawrence G. Votta. An experiment to assess different defect detection methods for software requirements inspections. In Proceedings of the 16 th International Conference on Software Engineering, pages 103–112, 1994.
H. Dieter Rombach, Victor R. Basili, and Richard W. Selby, editors. Experimental Software Engineering Issues: A critical assessment and future directions. Lecture Notes in Computer Science Nr. 706, Springer-Verlag, 1993.
Richard W. Selby. Evaluations of Software Technologies: Testing, CLEANROOM, and Metrics. PhD thesis, Department of Computer Science, University of Maryland, College Park, MD 20742, May 1985.
Richard W. Selby. Combining software testing strategies: An empirical evaluation. In Proceedings of the Workshop on Software Testing, 15–17 July, Banff, Canada, pages 82–90. IEEE Computer Society Press, 1986.
Richard W. Selby, Victor R. Basili, Jerry Page, and Frank E. McGarry. Evaluating software testing strategies. In Proceedings of the 9 th Annual Software Engineering Workshop, pages 42–53. NASA Goddard Space Flight Center, Greenbelt MD 20771,1984.
Sylvia B. Sheppard, Bill Curtis, Phil Milliman, and Tom Love. Modern coding practices and programmer performance. IEEE Computer, 12:41–49, December 1979.
Elaine J. Weyuker, On testing non-testable programs. Computer Journal, 25(4):465–470, November 1982.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1995 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kamsties, E., Lott, C.M. (1995). An empirical evaluation of three defect-detection techniques. In: Schäfer, W., Botella, P. (eds) Software Engineering — ESEC '95. ESEC 1995. Lecture Notes in Computer Science, vol 989. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-60406-5_25
Download citation
DOI: https://doi.org/10.1007/3-540-60406-5_25
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-60406-8
Online ISBN: 978-3-540-45552-3
eBook Packages: Springer Book Archive