Automated Software Engineering

, Volume 24, Issue 3, pp 575–602 | Cite as

Meta-learning based selection of software reliability models

  • Rafael Caiuta
  • Aurora Pozo
  • Silvia Regina Vergilio


The correct estimation of the software reliability level is fundamental to reduce efforts in the testing, maintenance and release activities. To help in this task, we find in the literature an increasing number of software reliability models (SRMs). However, none has proved to perform well considering different contexts. Due to this, the selection of the best model for a particular case is an important task. Most existing works on SRM selection need to test different models and decide based on how well the model fits the data and predicts the future events. Moreover, in general, they do not consider search-based models. Considering this fact, this paper introduces a Meta-learning approach for SRM selection. In such approach, some meta-features are used to indicate the best performing model. The approach is independent of the type of models to be selected, and can be used with different data mining algorithms. It includes the following activities: meta-knowledge extraction, meta-learning and classification. The activities meta-knowledge extraction and meta-learning are performed just once and generate a meta-classifier. Therefore, the meta-classifier is used to select the most adequate model for new projects (classification activity). The approach is evaluated in a set of experiments and the results do not show statistical difference between the Meta-learning approach and the choice of the best performing model. Otherwise, the results point out statistical difference between the Meta-learning approach and the choice of the worst performing model with a large stochastic difference according to the Vargha and Delaney Effect Size.


Software reliability models Meta-learning Data mining 



The authors would like to thank CNPq for financial support.


  1. Afzal, W., Torkar, R.: Suitability of genetic programming for software reliability growth modeling. In: International Symposium on Computer Science and its Applications, pp. 114–117 (2008)Google Scholar
  2. Aljahdali, S.H., Sheta, A., Rine, D.: Prediction of software reliability: a comparison between regression and neural network non-parametric models. In: ACS/IEEE International Conference on Computer Systems and Aplications, pp. 470–473 (2001)Google Scholar
  3. Arlot, S., Celisse, A.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010). doi: 10.1214/09-SS054 MathSciNetCrossRefMATHGoogle Scholar
  4. Asad, C.A., Ullah, M.I., Rehman, M.J.: An approach for software reliability model selection. In: 28th Annual International Computer Software and Applications Conference (COMPSAC’04), pp. 534–539. IEEE Computer Society (2004)Google Scholar
  5. Borowska, K., Topczewska, M.: Data preprocessing in the classification of the imbalanced data. Adv. Comput. Sci. Res. 11, 31–46 (2014)Google Scholar
  6. Brazdil, P., Giraud-Carrier, C., Soares, C., Vilalta, R.: Metalearning: Applications to Data Mining (Cognitive Technologies). Springer, Berlin (2009)MATHGoogle Scholar
  7. Caiuta, R., Emmendorfer, L., Pozo, A., Vergilio, S.R.: Selecting software reliability models with a neural network classifier. In: IEEE International Joint Conference on Neural Networks (IJCNN) (2008)Google Scholar
  8. Chatzis, S., Andreou, A.: Maximum entropy discrimination poisson regression for software reliability modeling. Neural Netw. Learn. Syst. PP(99), 1–1 (2015)Google Scholar
  9. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)MATHGoogle Scholar
  10. Costa, E.O., Souza, G.A., Pozo, A., Vergilio, S.R.: Exploring genetic programming and boosting techiques to model software reliability. IEEE Trans. Reliab. 56(3), 422–434 (2007)CrossRefGoogle Scholar
  11. Demsar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetMATHGoogle Scholar
  12. Febrero, F., Calero, C., ngeles Moraga, M.: A systematic mapping study of software reliability modeling. Inf. Softw. Technol. 56(8), 839–849 (2014)CrossRefGoogle Scholar
  13. Haykin, S.: Neural Networks: A Comprehensive Foundation, 2nd edn. Prentice Hall, Upper Saddle River (1998)MATHGoogle Scholar
  14. Kartalopoulos, S.: Understanding Neural Networks and Fuzzy Logic: Basic Concepts and Applications. Wiley, New York (1996)MATHGoogle Scholar
  15. Karunanithi, N., Whitley, D., Malaiya, Y.K.: Prediction of software reliability using connectionist models. IEEE Trans. Softw. Eng. 18(7), 563–574 (1992)CrossRefGoogle Scholar
  16. Khoshgoftaar, T., Woodcock, T.: Software reliability model selection: a case study. In: IEEE International Symposium on Software Reliability Engineering, pp. 183–191 (1991)Google Scholar
  17. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International Conference on Artificial Intelligence (1995)Google Scholar
  18. Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47(260), 583–621 (1952)CrossRefMATHGoogle Scholar
  19. Lakey, P., Neufelder, A.: System and Software Reliability Assurance Notebook. Rome Laboratory, Rome (1997)Google Scholar
  20. Lyu, M.: Handbook of software reliability engineering. IEEE Computer Society Press and McGraw-Hill. (2005). Accessed Jan 2008
  21. Manderna, R.: Software reliability model selection criteria: a literature review. Softw. Eng. Technol. 5(6), 200–204 (2013)Google Scholar
  22. Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)MATHGoogle Scholar
  23. Moranda, P.: Predictions of software reliability during debugging. In: Proceedings of the Annual Reliability Maintainability Symposium (1975)Google Scholar
  24. Moranda, P., Jelinski, Z.: Final report on software reliability study. Tech. rep, McDonnall Douglas Astronautics Company (1972)Google Scholar
  25. Musa, J.: A theory of software reliability and its application. IEEE Trans. Softw. Eng. 1(3), 312–327 (1975)CrossRefGoogle Scholar
  26. Musa, J.: Software Reliability Data. Data & Analysis Center for Software, New York (1980)Google Scholar
  27. Musa, J.D., Ianino, A., Okumoto, K.: Software Reliability: Measurement, Prediction, Application. McGraw-Hill, New York (1990)Google Scholar
  28. Pai, G.: A survey of software reliability models. Tech. rep., DCE, University of Virginia (2002)Google Scholar
  29. Pappa, G., Ochoa, G., Hyde, M., Freitas, A., Woodward, J., Swan, J.: Contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms. Genet. Program. Evol. Mach. 15(1), 3–35 (2014)CrossRefGoogle Scholar
  30. Park, J., Baik, J.: Improving software reliability prediction through multi-criteria based dynamic model selection and combination. J. Syst. Softw. 101, 236–244 (2015)CrossRefGoogle Scholar
  31. Prudencio, R., Ludermir, T.: Meta-learning approaches to selecting time series models. Neurocomput. J. 61, 121–137 (2004)CrossRefGoogle Scholar
  32. Prudencio, R., Ludermir, T., Carvalho, F.: A modal symbolic classifier to select time series models. Pattern Recogn. Lett. 25(8), 911–921 (2004)CrossRefGoogle Scholar
  33. Quinlan, J.R.: C4.5 Programs for Machine Learning. Morgan Kaufmann Publishers, San Francisco (1993)Google Scholar
  34. Santos, P., Ludemir, T., Prudencio, R.: Selection of time series forecasting models based on performance information. In: Proceedings of Fourth International Conference on Hybrid Intelligent Systems (HIS 04) (2004)Google Scholar
  35. Siegal, S., Castellan, N.: Non-Parametric Statistics for the Behavioural Sciences. McGraw Hill, New York (1988)Google Scholar
  36. Sitte, R.: Comparison of software reliability growth predictions: neural networks vs parametric recalibration. IEEE Trans. Softw. Eng. 48(3), 285–291 (1999)Google Scholar
  37. Smith-Miles, K.A.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41, 0300–0360 (2008)CrossRefGoogle Scholar
  38. Souza, G.A., Vergilio, S.R.: Modeling software reliability growth with artificial neural networks. In: IEEE Latin American Test Workshop, pp. 165–170. Buenos Aires (2006)Google Scholar
  39. Sundarkumar, G.G., Ravi, V.: A novel hybrid undersampling method for mining unbalanced datasets in banking and insurance. Eng. Appl. Artif. Intell. 37, 368–377 (2015)CrossRefGoogle Scholar
  40. University, W.: Weka: Machine Lerning Software in Java. University of Waikato. (2007)
  41. Vargha, A., Delaney, H.D.: A critique and improvement of the cl common language effect size statistics of mcgraw and wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)Google Scholar
  42. Vilalta, R., Giraud-Carrier, C., Brazdil, P.: Meta-learning: concepts and techniques. In: Maimon, O., Rokach, L. (eds.) Data mining and knowledge discovery handbook (2005)Google Scholar
  43. Wu, Y., Yang, R.: Software reliability modeling based on svm and virtual sample. In: Reliability and Maintainability Symposium (RAMS), 2013 Proceedings-Annual, pp. 1–6 (2013)Google Scholar
  44. Yamada, S.: Software Reliability Modeling: Fundamentals and Applications. Springer, Berlin (2014)CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Rafael Caiuta
    • 1
  • Aurora Pozo
    • 1
  • Silvia Regina Vergilio
    • 1
  1. 1.Computer Science DepartmentFederal University of Paraná (UFPR)CuritibaBrazil

Personalised recommendations