Abstract
Nowadays many commercial projects use open source applications or components (OSS). A recurring problem is therefore the selection of the most appropriate OSS for a project. A relevant criterion for selection is the reliability of the OSS. In this paper, we propose a method that selects the software reliability growth model (SRGM), which among several alternative models best predicts the reliability of the OSS, in terms of residual defects. Several methods exist for predicting residual defects in software, and a widely used method is SRGM. SRGM has underlying assumptions, which are often violated in practice, but empirical evidence has shown that many models are quite robust despite these assumption violations. However, within the SRGM family, many models are available, and it is often difficult to know which models are better to apply in a given context. We present an empirical method that applies various SRGMs iteratively on OSS defect data and selects the model which best predicts the residual defects of the OSS. We empirically validate the method by applying it to defect data collected from 21 different releases of 7 OSS projects. The results show that the method helps in selecting the best model among several alternative models. The method selects the best model 17 times out of 21. In the remaining 4, it selects the second best model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abdel-Ghaly, A. A., Chan, P. Y., & Littlewood, B. (1986). Evaluation of competing software reliability predictions. IEEE Transaction on Software Engineering, SE-12(12), 950–967.
Basili, V., et al, 1994. The goal question metric method. In Encyclopedia of Software Engineering, Wiley.
Brocklehurst, S., Chan, P. Y., Littlewood, B., & Snell, J. (1990). Recalibrating software reliability models. IEEE Transaction on Software Engineering, SE-16(4), 458–470.
Brown, S. (2004). What exactly are the merits of open-source software? Microsoft and Novell go head to head to talk out the issues—View #2. IEE Review, 50(10), 48–50.
Farber, D. (2004). Six barriers to open source adoption, ZDNet Tech Update, March, 2004 [WWW document].
Fenhzong, Z. et al. (2008). Analyzing and Modeling Open Source Software Bug Report Data. 19th Australian Conference on Software Engineering.
Gaudoin, O., Xie, M., & Yang, B. (2002). A simple goodness-of-fit test for the power-law process, based on the Duane plot. IEEE Transactions on Reliability, 52, 69.
Goel, A. L. (1985). Software reliability models: Assumption, limitations, and applicability. IEEE Transaction on Software Engineering, 12, 1411–1423.
Khoshgoftaar T. M., (1988). “On model selection in software reliability,” in 8th Symposium in Computational Statistics, August 1988, pp. 13–14, (Compstat’88).
Khoshgoftaar T. M., and Woodcock T. G., (1989). A simulation study of the performance of the Akaike information criterion for the selection of software reliability growth models. In Proceedings of the 27th Annual South East Region ACM Conference, April 1989, pp. 419–423.
Khoshgoftaar, T. M., and Woodcock, T. (1991). Software reliability model selection: A case study. In Proceedings of the 2nd International Symposium on Software Reliability Engineering. Austin, TX: IEEE Computer Society Press, 1991, pp. 183–191.
Lee, W., Jung, B. -G., and Baik J., (2008). Early reliability prediction: An method to software reliability assessment in open software adoption stage. In Secure System Integration and Reliability Improvement, 2008. SSIRI ‘08. Second International Conference on, pp. 226–227.
Li, P., et al. (2005). Finding predictors of field defects for open source software systems in commonly available data sources: A case study of openbsd CMU-ISRI-05-12.
Li, X., et al. (2011). Reliability analysis and optimal release-updating for open source software. Information and Software Technology, 53(2011), 929–936.
Lyu, M. R. (1996). Handbook of software reliability engineering. New York: McGraw Hill.
Lyu, M., and Nikora, A., (1992). CASREA—A computer-aided software reliability estimation tool. In Proceedings of the Fifth International Workshop on Computer-Aided Software Engineering, Montreal, CA, pp. 264–275.
Musa, J. D., and Okumoto K., (1983). A logarithmic Poisson execution time model for software reliability measurement. In Conference Proceedings 7th International Conference on Software Engineering, pp. 230–237.
Musa, J. D., Iannino, A., & Okumoto, K. (1987). Software reliability, measurement, prediction and application. New York: McGraw-Hill.
Pham, H. (2003). Software reliability and cost models: Perspectives, comparison and practice. European Journal of Operational Research, 149, 475–489.
Rahmani, C., et al. (2010). A comparative analysis of open source software reliability. Journal of Software, 5, 1384–1394.
Rossi, B. et al. (2010). Modelling failures occurrences of open source software with reliability growth. Journal of Open Source Software: New Horizons, p 268–280.
Schick, G. H., & Wolverton, R. W. (1978). An analysis of competing software reliability models. IEEE Transaction on Software Engineering, 2, 104–120.
Sharma, K., et al. (2010). Selection of optimal software reliability growth models using a distance based method. IEEE Transactions on Reliability, 59(2), 266.
IEEE Std 1633-2008 IEEE Recommended Practice in Software Reliability.
Stringfellow, C. 2000. An integrated method for improving testing effectiveness and efficiency. PhD Dissertation, Colorado State University.
Stringfellow, C., et al. (2002). An empirical method for selecting software reliability growth models. Empirical Software Engineering, Journal, 7, 319–343.
Sukert, A. N. (1979). Empirical validation of three software errors predictions models. IEEE Transaction on Reliability, 28, 199–205.
Syed-Mohamad, S. M., et al. (2008). Reliability growth of open source software using defect analysis. International Conference on Computer Science and Software Engineering, 2, 662.
Ullah, N., & Morisio, M. (2012). An empirical study of reliability growth of open versus closed source software through software reliability growth models. Proceeding of APSEC 2012, 1, 356.
Ullah, N., Morisio, M,. Vetro’, A., (2012). A comparative analysis of software reliability growth models using defects data of closed and open source software. In: 35th Annual IEEE Software Engineering Workshop, Heraclion, Crete, Greece, 12–13 October 2012. pp. 187–192.
Wanga, Wen.-Li., et al. (2006). Architecture-based software reliability modeling. Journal of System and Software, 79(1), 132–146.
Wohlin, Claes., Runeson, Per., Höst, Martin., Ohlsson, Magnus. C., Regnell, Bjöorn., & Wesslén, Anders. (2000). Experimentation in software engineering: An introduction. Norwell, MA: Kluwer Academic Publishers.
Wood, A. (1994). Software-reliability growth model: Primary-failures generate secondary-faults under imperfect debugging. IEEE Transactions on Reliability, 43(3), 408.
Xie, M. (1991). Software reliability modeling. Singapore: World Scientific Publishing.
Zeng, J. et al. (2010). A prototype system of software reliability prediction and estimation. Third International Symposium on Intelligent Information Technology and Security Informatics, pp. 558–561.
Zhou, Y., et al. (2005). Open source software reliability model: An empirical method. ACM SIGSOFT Software Engineering Notes, 30, 1.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ullah, N. A method for predicting open source software residual defects. Software Qual J 23, 55–76 (2015). https://doi.org/10.1007/s11219-014-9229-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-014-9229-3