A note on risks in software development with imperfect testing

Abstract

This study analytically assesses the statistical risk of releasing defective software that cannot be exhaustively tested, and of needlessly testing defect-free software. Specifically, it quantifies the probability of committing Type I (α) and Type II errors (β) in software development when one may release software that still is faulty or do needless testing since the test methods themselves may not be perfect. The study uses Truncated-Poisson and geometric distributed path lengths and Bernoulli-type inspection errors to link α and β to software design features, the development philosophies employed, and certain aspects that include code quality, cyclomatic complexity and the average length of basis paths. For risk reduction this study finds quantitative justification for raising test coverage, perfecting the test methods, the adoption of recent innovations and programming methods such as component-based design, SOA and XP as ways to raise the likelihood that the product developed will be fault free. Results are relatively robust with respect to the probability distributions assumed.