Forecasting software reliability

  • Bev Littlewood
Part of the Lecture Notes in Computer Science book series (LNCS, volume 341)


Computer software fails because of the presence of intellectual faults, ranging from simple coding faults to fundamental design faults. In principle, such faults can be permanently removed when they are detected by failure of the software. Then the software will exhibit reliability growth. The problem considered here is the one of forecasting this growth: it includes the estimation of the current reliability of the program from the previous failure data. We begin with a brief description of the software failure process: a non-stationary stochastic process. Several of the best-known software reliability growth models are described, and examples given of their performance on real software failure data. They show marked disagreement and thus reveal a need for methods of comparing and evaluating software reliability forecasts. Several simple techniques for conducting this evaluation are described and illustrated using several different models on real data sets. Finally, it is shown how in certain circumstances it is possible to improve the predictive accuracy of software reliability models by a re-calibration technique.


Prediction System Software Reliability Predictive Density Reliability Growth Software Reliability Growth Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Z Jelinski and P B Moranda, ‘Software reliability research', in Statistical Computer Performance Evaluation (W Freiberger, ed. New York: Academic Press, 1972, pp. 465–484.Google Scholar
  2. [2]
    M Shooman, ‘Operational testing and software reliability during program development', in Record. 1973 IEEE Symp. Computer Software Reliability (New York, NY, 1973, April 30–May 2) pp. 51–57.Google Scholar
  3. [3]
    B Littlewood and J L Verrall, ‘A Bayesian reliability growth model for computer software', J Royal Statist. Soc., C (Applied Statistics), 2, 1973, pp. 332–346.Google Scholar
  4. [4]
    J D Musa, A theory of software reliability and its application', IEEE Trans. Software Engineering, Vol SE-1, 1975 Sept, pp. 312–327.Google Scholar
  5. [5]
    C J Dale, 'software Reliability Evaluation Methods', British Aerospace Dynamics Group, ST-26750, 1982.Google Scholar
  6. [6]
    P A Keiller, B Littlewood, D R Miller and A Sofer, ‘Comparison of software reliability predictions', Digest FTCS 13 (13th International Symposium on Fault-Tolerant Computing) pp. 128–134, 1983.Google Scholar
  7. [7]
    B Littlewood, ‘How to measure software reliability and how not to', IEEE Trans. Reliability, Vol R-28, 1979, June pp. 103–110.Google Scholar
  8. [8]
    J C Laprie, ‘Dependability evaluation of software systems in operation’ IEEE Trans. Software Engineering, 1984, December.Google Scholar
  9. [9]
    E N Adams, ‘Optimizing preventive service of software products', IBM Journal of Research and Development, Vol 28, No 1, 1984.Google Scholar
  10. [10]
    H Ascher and H Feingold, Repairable Systems Reliability, Lecture Notes in Statistics, No 7, Marcel Dekker, New York, 1984.Google Scholar
  11. [11]
    J Aitchison and I R Dunsmore, Statistical Prediction Analysis, Cambridge University Press, Cambridge, 1975.Google Scholar
  12. [12]
    P M Nagel and J A Skrivan, 'software reliability: repetitive run experimentation and modelling', BCS-40399, Boeing Computer Services Company, Seattle, Washington, 1981, December.Google Scholar
  13. [13]
    E H Forman and N D Singpurwalla, ‘An empirical stopping rule for debugging and testing computer software', J Amer Statist Assoc, Vol 72, 1977, Dec., pp 750–757.Google Scholar
  14. [14]
    B Littlewood and J L Verrall, ‘On the likelihood function of a debugging model for computer software reliability', IEEE Trans Reliability, Vol R-30, 1981 June, pp 145–148.Google Scholar
  15. [15]
    H Joe and N Reid, ‘Estimating the number of faults in a system', J Amer Statist Assoc, Vol 80, 1985 March, pp 222–226.Google Scholar
  16. [16]
    B Littlewood and A Sofer, ‘A Bayesian modification to the Jelinski-Moranda software reliability growth model', CSR Technical Report, 1985.Google Scholar
  17. [17]
    B Littlewood, ‘Stochastic reliability growth: a model for fault-removal in computer programs and hardware designs', IEEE Trans. Reliability, Vol R-30, 4, Oct 1981, pp 313–320.Google Scholar
  18. [18]
    A A Abdel Ghaly, Ph D Thesis, City University, London 1986.Google Scholar
  19. [19]
    P A Keiller, B Littlewood, D R Miller and A Sofer, ‘On the quality of software reliability predictions', Proc. NATO ASI on Electronic Systems Effectiveness and Life Cycle Costing (Norwich, UK, 1982), Springer, 1983, pp 441–460.Google Scholar
  20. [20]
    D R Miller, ‘Exponential order statistic models of software reliability growth', Tech Report, T-496/84, George Washington University, Washington DC, 1984.Google Scholar
  21. [21]
    J T Duane, ‘Learning curve approach to reliability monitoring', IEEE Trans Aerospace, 2, 1964, pp 563–566.Google Scholar
  22. [22]
    L H Crow, ‘Confidence interval procedures for reliability growth analysis', Tech Report No 197, US Army Material Systems Analysis Activity, Aberdeen, Md, 1977.Google Scholar
  23. [23]
    H Akaike, ‘Prediction and Entropy', MRC Technical Summary Report, Mathematics Research Center, University of Wisconsin-Madison, June 1982.Google Scholar
  24. [24]
    M G Kendall and A Stuart, The Advanced Theory of Statistics, Griffin, London, 1961.Google Scholar
  25. [25]
    M Rosenblatt, ‘Remarks on a multivariate transformation', Ann Math Statist, 23, pp 470–472, 1952.Google Scholar
  26. [26]
    A P Dawid, ‘Statistical theory: the prequential approach', J Royal Statist Soc, A (1984), 147, pp 278–292.Google Scholar
  27. [27]
    A P Dawid, ‘Calibration-based empirical probability', Res Report 36, Department of Statistical Science, University College, London, 1984.Google Scholar
  28. [28]
    D R Cox and P A W Lewis, Statistical Analysis of Series of Events, Methuen, London, 1966.Google Scholar
  29. [29]
    A P Dawid, ‘The well-calibrated Bayesian', (with discussion) J Amer Statist Assoc, 77, pp 605–613, 1982.Google Scholar
  30. [30]
    A P Dawid, ‘Probability Forecasting', Encyclopedia of Statistical Sciences, Vol 6 (S Kotz, N L Johnson and C B Read, eds), Wiley-Interscience (to appear).Google Scholar
  31. [31]
    D R Miller, private communication, 1983.Google Scholar
  32. [32]
    H Braun and J M Paine, ‘A comparative study of models for reliability growth', Tech Report No 126, Series 2, Department of Statistics, Princeton University, 1977.Google Scholar
  33. [33]
    B Littlewood and P A Keiller, ‘Adaptive software reliability modelling', Digest FTCS-14 (14th International Conference on Fault-Tolerant Computing), pp 108–113, 1984.Google Scholar
  34. [34]
    I Goudie, private communication, 1984.Google Scholar
  35. [35]
    J D Musa, 'software reliability data', report available from Data and Analysis Center for Software, Rome Air Development Center, Rome, NYGoogle Scholar
  36. [36]
    N Langberg and N D Singpurwalla, ‘A unification of some software reliability models via the Bayesian approach', Tech Report, TM-66571, The George Washington University, Washington DC, 1981.Google Scholar
  37. [37]
    A L Goel and K Okumoto, ‘Time-dependent error-detection rate model for software reliability and other performance measures', IEEE Trans Reliability, Vol R-28, pp 206–211, 1979.Google Scholar
  38. [38]
    P E Amman and J C Knight, ‘Data diversity: an approach to software fault tolerance', Digest FTCS-17 (17th International Symposium on Fault-tolerant Computing), pp 122–126, 1987.Google Scholar
  39. [39]
    P Y Chan, B Littlewood and J Snell, ‘Panametric spline approach to adaptive reliability modelling', CSR Technical Report July 1985, City University, London.Google Scholar
  40. [40]
    S Brocklehurst, ‘On the effectiveness of adaptive software reliability modelling', CSR Technical Report, October 1987, City University, London.Google Scholar

Copyright information

© Springer-Verlag 1988

Authors and Affiliations

  • Bev Littlewood
    • 1
  1. 1.Centre for Software ReliabilityCity UniversityLondon

Personalised recommendations