Empirical Software Engineering

, 13:499

On the effectiveness of early life cycle defect prediction with Bayesian Nets

  • Norman Fenton
  • Martin Neil
  • William Marsh
  • Peter Hearty
  • Łukasz Radliński
  • Paul Krause
Article

Abstract

Standard practice in building models in software engineering normally involves three steps: collecting domain knowledge (previous results, expert knowledge); building a skeleton of the model based on step 1 including as yet unknown parameters; estimating the model parameters using historical data. Our experience shows that it is extremely difficult to obtain reliable data of the required granularity, or of the required volume with which we could later generalize our conclusions. Therefore, in searching for a method for building a model we cannot consider methods requiring large volumes of data. This paper discusses an experiment to develop a causal model (Bayesian net) for predicting the number of residual defects that are likely to be found during independent testing or operational usage. The approach supports (1) and (2), does not require (3), yet still makes accurate defect predictions (an R2 of 0.93 between predicted and actual defects). Since our method does not require detailed domain knowledge it can be applied very early in the process life cycle. The model incorporates a set of quantitative and qualitative factors describing a project and its development process, which are inputs to the model. The model variables, as well as the relationships between them, were identified as part of a major collaborative project. A dataset, elicited from 31 completed software projects in the consumer electronics industry, was gathered using a questionnaire distributed to managers of recent projects. We used this dataset to validate the model by analyzing several popular evaluation measures (R2, measures based on the relative error and Pred). The validation results also confirm the need for using the qualitative factors in the model. The dataset may be of interest to other researchers evaluating models with similar aims. Based on some typical scenarios we demonstrate how the model can be used for better decision support in operational environments. We also performed sensitivity analysis in which we identified the most influential variables on the number of residual defects. This showed that the project size, scale of distributed communication and the project complexity cause the most of variation in number of defects in our model. We make both the dataset and causal model available for research use.

Keywords

Software defect prediction Qualitative factors Quantitative data Bayesian network Decision support Sensitivity analysis 

References

  1. AgenaRisk (2007) Bayesian Network Software Tool. www.agenarisk.com
  2. Boehm B, Clark B, Horowitz E, Westland C, Madachy R, Selby R (1995) Cost models for future software life cycle process: COCOMO 2.0. Ann Softw Eng 1(1):57–94CrossRefGoogle Scholar
  3. Boetticher G, Menzies T, Ostrand T (2008) PROMISE Repository of Empirical Software Engineering Data http://promisedata.org/ repository, West Virginia University, Department of Computer Science
  4. Cangussu JW, DeCarlo RA, Mathur AP (2003) Using sensitivity analysis to validate a state variable model of the software test process. IEEE Trans Softw Eng 29(5):430–443CrossRefGoogle Scholar
  5. Chulani S, Boehm B (1999) Modelling Software Defect Introduction and Removal: COQUALMO (COnstructive QUAlity MOdel). Technical Report USC-CSE-99-510, University of Southern California, Center for Software EngineeringGoogle Scholar
  6. Chulani S, Boehm B, Steece B (1999) Bayesian analysis of empirical software engineering cost models. IEEE Trans Softw Eng 25(4):573–583CrossRefGoogle Scholar
  7. Compton T, Withrow C (1990) Prediction and Control of Ada Software Defects. J Syst Softw 12:199–207CrossRefGoogle Scholar
  8. eXdecide (2005) Quantified Risk Assessment and Decision Support for Agile Software Projects. EPSRC project EP/C005406/1. www.dcs.qmul.ac.uk/~norman/radarweb/core_pages/projects.html
  9. Fenton NE, Pfleeger SL (1998) Software Metrics: A Rigorous and Practical Approach (2nd Edition). PWS Publishing, BostonGoogle Scholar
  10. Fenton NE, Neil M (1999) A critique of software defect prediction models. IEEE Trans Software Eng 25(5):675–689CrossRefGoogle Scholar
  11. Fenton NE, Krause P, Neil M (2002a) Probabilistic modelling for software quality control. J Appl Non-Class Log 12(2):173–188MATHCrossRefMathSciNetGoogle Scholar
  12. Fenton NE, Krause P, Neil M (2002b) Software measurement: uncertainty and causal modelling. IEEE Software. 10(4):116–122CrossRefGoogle Scholar
  13. Fenton NE, Marsh W, Neil M, Cates P, Forey S, Tailor M (2004) Making Resource Decisions for Software Projects. Proceedings of 26th International Conference on Software Engineering (ICSE 2004), Edinburgh, United Kingdom, 397–406Google Scholar
  14. Fenton NE, Neil M, Caballero JG (2007a) Using ranked nodes to model qualitative judgments in Bayesian networks. IEEE Trans Knowl Data Eng 19(10):1420–1432CrossRefGoogle Scholar
  15. Fenton NE, Neil M, Marsh W, Hearty P, Marquez D, Krause P, Mishra R (2007b) Predicting software defects in varying development lifecycles using Bayesian nets. Inf Softw Technol 49(1):32–43CrossRefGoogle Scholar
  16. Fenton N, Neil M, Marsh W, Hearty P, Radliński Ł, Krause P (2007c) Project Data Incorporating Qualitative Factors for Improved Software Defect Prediction. Proceedings of the 3rd International Workshop on Predictor Models in Software Engineering. International Conference on Software Engineering. IEEE Computer Society, Washington, DC: 2Google Scholar
  17. Gaffney JR (1984) Estimating the Number of Faults in Code. IEEE Trans Softw Eng 10(4):141–152MathSciNetCrossRefGoogle Scholar
  18. Henry S, Kafura D (1984) The evaluation of software system’s structure using quantitative software metrics. Softw Pract Exp 14(6):561–573CrossRefGoogle Scholar
  19. ISBSG (2007) Repository Data Release 10. International Software Benchmarking Standards Group. www.isbsg.org
  20. Jensen FV (1996) An introduction to Bayesian networks. UCL Press, LondonGoogle Scholar
  21. Jones C (1986) Programmer productivity. McGraw Hill, New YorkGoogle Scholar
  22. Jones C (1999) Software sizing. IEE Review 45(4):165–167CrossRefGoogle Scholar
  23. Kitchenham BA, Pickard LM, MacDonell SG, Shepperd MJ (2001) What accuracy statistics really measure. IEE Proc Softw 148(3):81–85CrossRefGoogle Scholar
  24. Lipow M (1982) Number of Faults per Line of Code. IEEE Trans Softw Eng 8(4):437–439CrossRefGoogle Scholar
  25. Menzies T, Greenwald J, Frank A (2007) Data mining static code attributes to learn defect predictors. IEEE Trans Softw Eng 32(11):1–12Google Scholar
  26. MODIST (2003) Models of Uncertainty and Risk for Distributed Software Development. EC Information Society Technologies Project IST-2000-28749. www.modist.org
  27. Musilek P, Pedrycz W, Nan Sun, Succi G (2002) On the Sensitivity of COCOMO II Software Cost Model. Proc of the 8th IEEE Symposium on Software Metrics: 13–20Google Scholar
  28. Neapolitan RE (2004) Learning Bayesian networks. Pearson Prentice Hall, Upper Saddle RiverGoogle Scholar
  29. Neil M, Krause P, Fenton NE (2003) Software Quality Prediction Using Bayesian Networks Software Engineering with Computational Intelligence (Ed T.M. Khoshgoftaar). Kluwer, Chapter 6Google Scholar
  30. Ostrand TJ, Weyuker EJ, Bell RM (2005) Predicting the location and number of faults in large software systems. IEEE Trans Softw Eng 31(4):340–355CrossRefGoogle Scholar
  31. Radliński Ł, Fenton N, Neil M, Marquez D (2007) Improved Decision-Making for Software Managers Using Bayesian Networks, Proc. of 11th IASTED Int. Conf. Software Engineering and Applications (SEA), Cambridge, MA: 13–19Google Scholar
  32. Saltelli A (2000) What is Sensitivity Analysis. In: Saltelli A, Chan K, Scott EM (eds) Sensitivity Analysis. John Wiley & Sons, pp. 4–13Google Scholar
  33. SIMLAB (2004) Simulation Environment for Uncertainty and Sensitivity Analysis Version 2.2. Joint Research Centre of the European Commission. http://simlab.jrc.cec.eu.int/
  34. Stensrud E, Foss T, Kitchenham B, Myrtveit I (2002) An Empirical Validation of the Relationship Between the Magnitude of Relative Error and Project Size. Proc. of 8th IEEE Symposium on Software Metrics: 3–12Google Scholar
  35. Wagner S (2007a) An Approach to Global Sensitivity Analysis: FAST on COCOMO. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement: 440–442Google Scholar
  36. Wagner S (2007b) Global Sensitivity Analysis of Predictor Models in Software Engineering. Proceedings of the 3rd International Workshop on Predictor Models in Software Engineering. International Conference on Software Engineering. IEEE Computer Society, Washington, DC: 3Google Scholar
  37. Winkler RL (2003) An introduction to Bayesian inference and decision,, 2nd edn. Probabilistic Publishing, GainesvilleGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Norman Fenton
    • 1
  • Martin Neil
    • 1
  • William Marsh
    • 1
  • Peter Hearty
    • 1
  • Łukasz Radliński
    • 1
    • 2
  • Paul Krause
    • 3
  1. 1.Department of Computer Science, Queen MaryUniversity of LondonLondonUK
  2. 2.Institute of Information Technology in ManagementUniversity of SzczecinSzczecinPoland
  3. 3.Department of ComputingUniversity of SurreyGuildfordUK

Personalised recommendations