Positive test bias in software testing among professionals: A review

  • Laura Marie Leventhal
  • Barbee M. Teasley
  • Diane S. Rohlman
  • Keith Instone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 753)


Fundamental but virtually unexplored issues in human-computer interaction involve the roles of biases in software engineering tasks. In studies of naturalistic testing tasks, as well as ones which follow common laboratory models in this area, we have found ample evidence that testers have positive test bias. This bias is manifest as a tendency to execute about four times as many positive tests, designed to show that “the program works,” as tests which challenge the program. In our prior work, we have found that the expertise of the subjects, the completeness of the software specifications, and the presence/absence of program errors may reduce positive test bias. Skilled computer scientists invent specifications to test in the absence of actual specifications, but still exhibit positive test bias.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Beizer, B. (1990). Software Testing Techniques, 2nd ed., New York: Van Nostrand Reinhold.Google Scholar
  2. Griggs, R.A. and Newstead, S.E. (1982) The role of problem structure in a deductive reasoning task. Journal of Experimental Psychology, Memory & Cognition. 8. 297–307.Google Scholar
  3. Gorman, M. E., Stafford, A., & Gorman, M. E. (1987). Disconfirmation and dual hypotheses on a more difficult version of Wason's 2-4-6 task. The Quarterly Journal of Experimental Psychology. 39A. 1–28.Google Scholar
  4. IEEE (1983). IEEE Standard Glossary of Software Engineering Terminology. ANSI Std 729-1983. New York.Google Scholar
  5. Katz, I.R. & Anderson, J.R. (1988) Debugging: An analysis of bug-location strategies. Human-Computer Interaction. 3. 351–399.Google Scholar
  6. Kern, L. H., Mirels, H. L. & Hinshaw, V. G. (1983) Scientists' understanding of propositional logic: An experimental investigation. Social Studies of Science. 13. 131–146.Google Scholar
  7. Klayman, J. & Ha, Y.-W. (1987) Confirmation, disconfirmation and information in hypothesis testing. Psychological Review. 94. 211–228.Google Scholar
  8. Mahoney, M. J. & DeMonbreun, B. G. (1978) Psychology of the scientist: An analysis of problem-solving bias. Cognitive Therapy and Research. 1. 229–238.Google Scholar
  9. Mitroff, I. (1974) Norms and counter-norms in a select group of Apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review. 39. 579–595.Google Scholar
  10. Myers, G. J. (1979). The Art of Software Testing. New York: Wiley.Google Scholar
  11. Mynatt, B. T. (1990). Software Engineering with Student Project Guidance. New York: Prentice-Hall.Google Scholar
  12. Mynatt, C.R., Mynatt, B.T., Leventhal, L.M., and Rohlman, D.S. (1992) Software Testing as Hypothesis Testing: A Study of Positive Test Strategy. 2nd International Conference on Thinking. July 27–31, 1992. Plymouth, U.K.Google Scholar
  13. Omar, A. A. & Mohammed, F. A. (1991). A survey of software functional testing methods. ACM SIGSOFT Software Engineering Notes. 16. 75–92.Google Scholar
  14. Pennington, N. and Grabowski, B. (1990) The tasks of programming. In Hoc, J.-M., T.R.G. Green, R. Samurcay & D.J. Gilmore (Eds.) Psychology of Programming. San Diego, CA: Academic Press LTD. 45–62.Google Scholar
  15. Pressman, R. S. (1992). Software Engineering: A Practitioner's Approach (3rd ed.). New York: MaGraw-Hill, Inc.Google Scholar
  16. Rohlman, D. S. (1992). The influence of problem representation on hypothesis testing. Unpublished manuscript.Google Scholar
  17. Rohlman, D. S. Mynatt, C. R., Mynatt, B. T., & Leventhal, L. M. (1992, April) The Influence of Expertise on the Four-Card Selection Task in the Domain of Software Testing. Presented at the Annual Meeting of the Midwestern Psychological Association, Chicago, IL.Google Scholar
  18. Teasley, B.M., Leventhal, L.M., Mynatt C.R., & Rohlman, D. S. (1993) Why Software Testing Is Sometimes Ineffective: Two Applied Studies of Positive Test Strategy. Submitted to: Journal of Applied Psychology.Google Scholar
  19. Tversky, A. and Kahneman. D. (1974) Judgement under uncertainty: Heuristics and biases. Science. 185. 1124–1131.Google Scholar
  20. Tweney, R. D. (1985) Faraday's discovery of induction: A cognitive approach. In D. Gooding, & R. James (Eds.), Faraday Rediscovered. London: MacMillan. 159–206.Google Scholar
  21. Tweney, R. D., Doherty, M. E., & Mynatt, C. R. (Eds.) (1981) On Scientific Thinking. New York: Columbia University Press.Google Scholar
  22. Tweney, R. D., Doherty, M. E., Worner, W., Pliske, D., Mynatt, C.R., Gross, K., & Arkelin, D. (1980) Strategies of rule discovery in an inference task. Quarterly Journal of Experimental Psychology. 32. 109–123.Google Scholar
  23. Vessey, I. (1985) Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies. 23. 459–494.Google Scholar
  24. Vessey, I. (1986) Expertise in debugging computer programs: An analysis of the content of verbal protocols. IEEE Transactions on Systems, Man, and Cybernetics. 16. 621–637.Google Scholar
  25. Walker, B. J. and Harper, D. R. (1990) Decision-making under system failure conditions. Final report to Air Force Office of Scientific Research, Contract Number F49620-88-C-0053. (Unpublished manuscript).Google Scholar
  26. Wason, P. C. (1968) Reasoning about a rule. Quarterly Journal of Experimental Psychology. 23. 273–281.Google Scholar
  27. Zweben, S. H., Stringfellow, C., & Barton, R. (1989) Exploratory studies of the software testing methods used by novice programmers. In Third Software Engineering Institute Conference on Software Engineering Education. New York: Springer-Verlag.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1993

Authors and Affiliations

  • Laura Marie Leventhal
    • 1
  • Barbee M. Teasley
    • 2
  • Diane S. Rohlman
    • 2
  • Keith Instone
    • 1
  1. 1.Computer Science DepartmentBowling Green State UniversityOhioUSA
  2. 2.Oregon Health Sciences UniversityPortlandUSA

Personalised recommendations