Skip to main content
SpringerLink
Log in

Determining the Best Prediction Accuracy of Software Maintainability Models Using Auto-WEKA

  • Conference paper
  • First Online:
Advances in Data Science, Cyber Security and IT Applications (ICC 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1098))

Included in the following conference series:

  • 771 Accesses

Abstract

Highly accurate prediction of software maintainability models is a significant requirement to achieve software quality assurance. Development an accurate prediction model may involve on trying several types of machine learning models with different configurations that include tuning parameters and selected features. However, this is a difficult and very time-consuming task to implement. In this paper, we report on the experience of using a new, rapid automated tool to identify the best prediction accuracy of a software maintainability model, namely Auto-WEKA, applied to sets of different models with various configurations. Auto-WEKA is applied to five datasets collected from real-world open-source software systems. The mean magnitude relative error (MMRE) value is used to evaluate the accuracy of predictive models, along with ZeroR model to compare selected model performance with the baseline. The results obtained from this study provide empirical evidence of the ability of Auto-WEKA to identify the best model to predict software maintainability. Auto-WEKA selected the following as the best prediction models: SMOreg in Eclipse JDT Core dataset, RandomSubSpace in Eclipse PDE UI dataset, KStar in Equinox Framework dataset, RandomForet in Lucene and Mylyn datasets. These selected models achieved a low MMRE value and improved performance of the accuracy prediction over the baseline. The proposed tool was found useful in identifying the best model in predicting software maintainability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jung, H.-W., Kim, S.-G., Chung, C.-S.: Measuring software product quality: A survey of ISO/IEC 9126. IEEE Softw. 21(5), 88–92 (2004)

    Article  Google Scholar 

  2. IEEE Std 610.12-1990: IEEE Standard Glossary of Software Engineering Terminology. IEEE (1990)

    Google Scholar 

  3. van Koten, C., Gray, A.R.: An application of Bayesian network for predicting object-oriented software maintainability. Inf. Software Technol. 48(1), 59–67 (2006)

    Article  Google Scholar 

  4. Zhou, Y., Leung, H.: Predicting object-oriented software maintainability using multivariate adaptive regression splines. J. Syst. Software 80(8), 1349–1361 (2007)

    Article  Google Scholar 

  5. Elish, M.O., Elish, K.O.: Application of TreeNet in predicting object-oriented software maintainability: a comparative study. In: 2009 13th European Conference on Software Maintenance and Reengineering, pp. 69–78 (2009)

    Google Scholar 

  6. Malhotra, R., Chug, A.: Application of Group Method of Data Handling model for software maintainability prediction using object oriented systems. Int. J. Syst. Assur. Eng. Manag. 5(2), 165–173 (2014)

    Article  Google Scholar 

  7. Thwin, M.M.T., Quah, T.-S.: Application of neural networks for software quality prediction using object-oriented metrics. J. Syst. Softw. 76(2), 147–156 (2005)

    Article  Google Scholar 

  8. Kumar, L., Rath, S.K.: Software maintainability prediction using hybrid neural network and fuzzy logic approach with parallel computing concept. Int. J. Syst. Assur. Eng. Manag. J. Article 8(2), 1487–1502 (2017)

    Article  Google Scholar 

  9. Alsolai, H.: Predicting software maintainability in object-oriented systems using ensemble techniques. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 716–721. IEEE (2018)

    Google Scholar 

  10. Li, W., Henry, S.: Object-oriented metrics that predict maintainability. J. Syst. Softw. 23(2), 111–122 (1993)

    Article  Google Scholar 

  11. Chidamber, S.R., Kemerer, C.F.: A metrics suite for object oriented design. IEEE Trans. Software Eng. 20(6), 476–493 (1994)

    Article  Google Scholar 

  12. Dahiya, S.S., Chhabra, J.K., Kumar, S.: Use of genetic algorithm for software maintainability metrics’ conditioning. In: 15th International Conference on Advanced Computing and Communications (ADCOM 2007), pp. 87–92: IEEE (2007)

    Google Scholar 

  13. Kotthoff, L., Thornton, C., Hoos, H.H., Hutter, F., Leyton-Brown, K.: Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res. 18(1), 826–830 (2017)

    MathSciNet  MATH  Google Scholar 

  14. Kotthoff, L., Thornton, C., Hutter, F.: User guide for auto-WEKA version 2.3. Department of Computer Science, University of British Columbia, BETA lab, Vancouver, BC, Canada, Technical report, vol. 2 (2017)

    Google Scholar 

  15. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15(1), 3133–3181 (2014)

    MathSciNet  MATH  Google Scholar 

  16. Olson, R.S., La Cava, W., Mustahsan, Z., Varik, A., Moore, J.H.: Data-driven advice for applying machine learning to bioinformatics problems. arXiv preprint arXiv:1708.05070 (2017)

  17. Laradji, I.H., Alshayeb, M., Ghouti, L.: Software defect prediction using ensemble learning on selected features. Inf. Softw. Technol. 58, 388–402 (2015)

    Article  Google Scholar 

  18. Gao, K., Khoshgoftaar, T.M., Wang, H., Seliya, N.: Choosing software metrics for defect prediction: an investigation on feature selection techniques. Softw. Pract. Experience 41(5), 579–606 (2011)

    Article  Google Scholar 

  19. Catal, C., Diri, B.: Investigating the effect of dataset size, metrics sets, and feature selection techniques on software fault prediction problem. Inf. Sci. 179(8), 1040–1058 (2009)

    Article  Google Scholar 

  20. Fu, W., Menzies, T., Shen, X.: Tuning for software analytics: is it really necessary? Inf. Softw. Technol. 76, 135–146 (2016)

    Article  Google Scholar 

  21. Ambros, M.D., Lanza, M., Robbes, R.: An extensive comparison of bug prediction approaches. In: 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), pp. 31–41 (2010)

    Google Scholar 

  22. Yang, J., Qian, H.: Defect prediction on unlabeled datasets by using unsupervised clustering. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 465–472 (2016)

    Google Scholar 

  23. Boucher, A., Badri, M.: Using software metrics thresholds to predict fault-prone classes in object-oriented software. In: 2016 4th International Conference on Applied Computing and Information Technology/3rd International Conference on Computational Science/Intelligence and Applied Informatics/1st International Conference on Big Data, Cloud Computing, Data Science & Engineering (ACIT-CSII-BCD), pp. 169–176 (2016)

    Google Scholar 

  24. Ahmed, M.A., Al-Jamimi, H.A.: Machine learning approaches for predicting software maintainability: a fuzzy-based transparent model. IET Softw. 7(6), 317–326 (2013)

    Article  Google Scholar 

  25. Kumar, L., Rath, S.: Predicting object-oriented software maintainability using hybrid neural network with parallel computing concept. In: Proceedings of the 8th India Software Engineering Conference, pp. 100–109. ACM (2015)

    Google Scholar 

  26. Kumar, L., Naik, D.K., Rath, S.K.: Validating the effectiveness of object-oriented metrics for predicting maintainability. Procedia Computer Science 57, 798–806 (2015)

    Article  Google Scholar 

  27. Kitchenham, B.A., Pickard, L.M., MacDonell, S.G., Shepperd, M.J.: What accuracy statistics really measure [software estimation]. IEE Proc. Softw. 148(3), 81–85 (2001)

    Article  Google Scholar 

  28. Shepperd, M., MacDonell, S.: Evaluating prediction systems in software project estimation. Inf. Softw. Technol. 54(8), 820–827 (2012)

    Article  Google Scholar 

  29. Bi, J., Bennett, K.P.: Regression error characteristic curves. In: Proceedings of the 20th international conference on machine learning (ICML-03), pp. 43–50 (2003)

    Google Scholar 

  30. Mendes, E., Kitchenham, B.: Further comparison of cross-company and within-company effort estimation models for web applications. In: 10th International Symposium on Software Metrics, 2004. Proceedings, pp. 348–357. IEEE (2004)

    Google Scholar 

  31. Aher, S.B., Lobo, L.: Data mining in educational system using Weka. In: IJCA Proceedings on International Conference on Emerging Technology Trends (ICETT), vol. 3, pp. 20–25 (2011)

    Google Scholar 

  32. Venkatesh, A., Jacob, S.G.: Prediction of credit-card defaulters: a comparative study on performance of classifiers. Int. J. Comput. Appl. 145(7) (2016)

    Google Scholar 

  33. Alsolai, H., Roper, M.: Application of ensemble techniques in predicting object-oriented software maintainability. In: Proceedings of the Evaluation and Assessment on Software Engineering, Copenhagen, Denmark (2019)

    Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge Princess Nourah bint Abdulrahman University for their support in pursuing this research. We thank the University of Strathclyde for providing several facilities in the production of this research paper.

Author information

Authors and Affiliations

  1. Computer Science and Information System, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia

    Hadeel Alsolai

  2. Computer and Information Sciences, University of Strathclyde, Glasgow, UK

    Hadeel Alsolai & Marc Roper

Authors
  1. Hadeel Alsolai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marc Roper
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hadeel Alsolai .

Editor information

Editors and Affiliations

  1. Princess Nourah Bint Abdul Rahman University, Riyadh, Saudi Arabia

    Auhood Alfaries

  2. Princess Nourah Bint Abdul Rahman University, Riyadh, Saudi Arabia

    Hanan Mengash

  3. Hasselt University, Hasselt, Belgium

    Ansar Yasar

  4. Acadia University, Wolfville, NS, Canada

    Elhadi Shakshuki

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alsolai, H., Roper, M. (2019). Determining the Best Prediction Accuracy of Software Maintainability Models Using Auto-WEKA. In: Alfaries, A., Mengash, H., Yasar, A., Shakshuki, E. (eds) Advances in Data Science, Cyber Security and IT Applications. ICC 2019. Communications in Computer and Information Science, vol 1098. Springer, Cham. https://doi.org/10.1007/978-3-030-36368-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-36368-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-36367-3

  • Online ISBN: 978-3-030-36368-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation