Accurate and practical software maintainability prediction enables organizations to effectively manage their maintenance resources and guide maintenance-related decision making. This paper presents SMPLearner, an automated learning-based approach to train maintainability predictors by harvesting the actual average maintenance effort computed from the code change history as well as employing a much richer set of 44 four-level hierarchical code metrics collected by static code analysis tools. We systematically evaluated SMPLearner on 150 observations partitioned from releases of eight large scale open source software systems. Our evaluation showed that SMPLearner not only outperformed the traditional 4-metric MI model but also the recent learning-based maintainability predictors constructed based on single Class-level metrics, demonstrating that single Class-level metrics were not sufficient for maintainability prediction.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Understand is a static analysis tool for maintaining, measuring, and analyzing critical or large code bases. It calculates the four MI metrics objectively based on the code structure. It can be accessible at http://www.scitools.com/index.php.
VirtualBox’s version control system is available at https://www.virtualbox.org/browser/vbox/trunk/src/VBox.
A description of Logiscope is available at https://www-304.ibm.com/support/docview.wss?uid=swg24021021.
See Sect. 4.2 for details on how the conversion is done and why it is needed.
Spearman’s rank correlation coefficient rather than Pearson’s method is used to compute all correlation results in this paper due to its robustness and lack of assumptions about normality.
Al-Kilidar, H., Cox, K., Kitchenham, B.: The use and usefulness of the ISO/IEC 9126 quality standard, In IEEE International Symposium on Empirical Software Engineering, pp. 7–17 (2005)
Antonellis, P., Antoniou, D., Kanellopoulos, Y., Makris, C., Theodoridis, E., Tjortjis, C., Tsirakis, N.: A data mining methodology for evaluating maintainability according to ISO/IEC-9126 software engineering Cproduct quality standard. In Special Session on System Quality and Maintainability (SQM) (2007)
Baggen, R., Correia, J.P., Schill, K., Visser, J.: Standardized code quality benchmarking for improvingsoftware maintainability. Software Quality Journal 20(2), 287–307 (2012)
Bhattacharya, P., Iliofotou, M., Neamtiu, I., Faloutsos, M.: Graph-based analysis and prediction for software evolution. ICSE, pp. 419–429 (2012)
Bhattacharya, P., Neamtiu, I.: Assessing programming language impact on development and maintenance: A study on C and C++. ICSE 2011, pp. 171–180 (2011)
Burkett, D. Klein, D.: Two Languages are Better than one (for Syntactic Parsing). In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 877–886 (2008)
Chidamber, S.R., Kemerer, C.F.: A metrics suite for object oriented design. IEEE Transactions on Software Engineering 20(6), 476–493 (1994)
Coleman, D., Ash, D., Lowther, B., Oman, P.: Using Metrics to Evaluate Software System Maintainability. IEEE Computer, pp. 44–49 (1994)
Foss, T., Stensrud, E., Kitchenham, B., Myrtveit, I.: A simulation study of the model evaluation criterion MMRE. IEEE Trans. Softw. Eng. 29(11), 985–995 (2003)
Glass, R.L.: Facts and Fallacies of Software Engineering. Addison Wesley Professional, Boston (2002)
Halstead, M.H.: Elements of Software Science (Operating and Programming Systems Series). Elsevier, New York (1977)
Heitlager, I., Kuipers, T., Visser, J.: A practical model for measuring maintainability. In International Conference on Quality of Information and Communications Technology, pp. 30–39 (2007)
IEEE Std. 610.12-1990: Standard Glossary of Software Engineering Terminology. IEEE Computer Society Press, Los Alamitos (1993)
Kaur, A., Kaur, K.: Statistical comparison of modelling methods for software maintainability prediction. Int. J. Softw. Eng. Knowl. Eng. 23(6), 743–774 (2013)
Kitchenham, B.A., Pickard, L.M., MacDonell, S.G., Shepperd, M.J.: What accuracy statistics really measure. IEE. Prcc Softw 148(3), 81–85 (2001)
Koten, C.V., Gray, A.R.: An application of bayesian network for predicting object-oriented software maintainability. Inform. Softw. Technol. 48(1), 59–67 (2006)
Li, W., Henry, S.: Object-oriented metrics that predict maintainability. J. Syst. Softw. 23(2), 111–122 (1993)
Liso, A.: Software maintainability metrics model: an improvement in the Coleman-Oman model. Crosstalk, 15–17 (2001)
Malhotra, R., Chug, A.: Software maintainability prediction using machine learning algorithms. Softw. Eng. Int. J. 2(2), 19–36 (2012)
McCabe, T.: A complexity measure. IEEE Trans. Softw. Eng. 2(4), 308–320 (1976)
Mittal, H., Bhatia, P.: Software maintainability assessment based on fuzzy logic technique. ACM SIGSOFT Softw. Eng. Notes 34(3), 1–5 (2009)
Muthanna, S., Kontogiannis, K., Ponnambalam, K., Stacey, B.: A maintainability model for industrial software systems using design level metrics. In Proceedings of 7th Working Conference on Reverse Engineering, pp. 248–256 (2000)
Oman, P. W., Hagemeister, J.: Metrics for assessing a software system’s maintainability. In Proceedings of the Conference on Software Maintenance, IEEE Computer Society Press, Los Alamitos, CA, pp. 337–344 (1992)
Ramil, J.F., Cortazar, D.I., Mens, T.: What does it take to develop a million lines of open source code? OSS 299, 170–184 (2009)
Riaz, M., Mendes, E., Tempero, E.: A systematic review of software maintainability prediction and metrics. In Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement, pp. 367–377 (2009)
Shepperd, M., Cartwright, M., Kadoda, G.: On building prediction systems for software engineers. Empir. Softw. Eng. 5, 175–182 (2000)
Singh, Y., Bhatia, P.K., Sangwan, O.: Predicting software maintenance using fuzzy model. ACM SIGSOFT Softw. Eng. Notes 34(4), 1–6 (2009)
Welker, K.D.: The software maintainability index revisited, crosstalk. J. Def. Softw. Eng. 14, 18–21 (2001)
Welker, K.D., Oman, P.W., Atkinson, G.G.: Development and application of an automated source code maintainability index. J. Softw. Maint. Evol. R 9, 127–159 (1997)
Yu, L.: Indirectly predicting the maintenance effort of open-source software. J. Softw. Maint. Evol. Res. Pract. 18(5), 311–332 (2006)
Zhou, Y., Leung, H.: Predicting object-oriented software maintainability using multivariate adaptive regression splines. J. Syst. Softw. 80, 1349–1361 (2007)
Zhou, Y., Xu, B.: Predicting the maintainability of open source software using design metrics. Wuhan Univ. J. Nat. Sci. 13(1), 14–21 (2008)
We thank the anonymous reviewers for their helpful comments on an earlier draft of this manuscript. This research is supported by the U.S. National Science Foundation (NSF CNS Award # 1126747). It is partially supported by the Oversea Fund of State Key Laboratory for Novel Software Technology of Nanjing University, National Natural Science Foundation, China (No. 61100039, 61021062, 61272188, 91318301), and Natural Science Foundation of Jiangsu Province, China (No. BK20131277).
About this article
Cite this article
Zhang, W., Huang, L., Ng, V. et al. SMPLearner: learning to predict software maintainability. Autom Softw Eng 22, 111–141 (2015). https://doi.org/10.1007/s10515-014-0161-3
- Software Maintainability
- Maintenance effort
- Software metric
- Machine learning