Abstract
With the increase in the adoption rate of machine learning algorithms in multiple sectors, the need for accurate measurement and assessment is imperative, especially when classifiers are applied to real world applications. Determining which are the most appropriate evaluation metrics to effectively assess and evaluate the performance of a binary, multi-class and multi-labelled classifier needs to be further understood. Another significant challenge impacting research is that results from models that are similar in nature cannot be adequately compared if the criteria for the measurement and evaluation of these models are not standardized. This review paper aims at highlighting the various evaluation metrics being applied in research and the non-standardization of evaluation metrics to measure the classification results of the model. Although Accuracy, Precision, Recall and F1-Score are the most applied evaluation metrics, there are certain limitations when considering these metrics in isolation. Other metrics such as ROC\AUC and Kappa statistics have proven to provide additional insightful into the effectiveness of an algorithms adequacy and should also be considered when evaluating the effectiveness of binary, multi-class and multi-labelled classifiers. The adoption of a standardized and consistent evaluation methodology should be explored as an area of future work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wehle, H.: Machine learning, deep learning, and AI: what’s the difference. TY - BOOK AU - Wehle, Hans-Dieter PY, 2017/07/24 SP - T1, ER (2017)
Sayed, H., Abdel-Fattah, M.A., Kholief, S.: Predicting potential banking customer churn using apache spark ML and MLlib packages: a comparative study. Int. J. Adv. Comput. Sci. Appl. 9, 674–677 (2018). https://doi.org/10.14569/ijacsa.2018.091196
Fabris, F., de Magalhães, J.P., Freitas, A.A.: A review of supervised machine learning applied to ageing research. Biogerontology 18(2), 171–188 (2017). https://doi.org/10.1007/s10522-017-9683-y
Mahbobi, Tiemann: Regression Basics 2015. https://opentextbc.ca/introductorybusinessstatistics/chapter/regression-basics-2/. Accessed 6 June 2022
Kotsiantis, S., Kanellopoulos, D., Pintelas, P.: Handling imbalanced datasets: a review. Science 2006(30), 25–36 (1979)
Jain, H., Khunteta, A., Srivastava, S., Jain, H., Khunteta, A., Srivastava, S.: Churn prediction in telecommunication using logistic regression and logit boost. Procedia Comput. Sci. 167, 101–112 (2020). https://doi.org/10.1016/j.procs.2020.03.187
Arivazhagan, B., Sankara Subramanian, D.R.S., Scholar, R.: Customer churn prediction model using regression with Bayesian boosting technique in data mining. In: IjaemaCom 2020, vol. XII, pp. 1096–104 (2020)
Sebastian, H.T., Wagh, R.: Churn analysis in telecommunication using logistic regression. Orient. J. Comput. Sci. Technol. 10, 207–212 (2017)
Parmar, P.: Telecom churn prediction model using XgBoost classifier and logistic regression algorithm. Int. Res. J. Eng. Technol. (IRJET) 08, 1100–1105 (2021)
Kavitha, V., Kumar, H., Kumar, M., Harish, M.: Churn prediction of customer in telecom industry using machine learning algorithms. Int. J. Eng. Res. Technol. (IJERT) 9, 181–184 (2020). https://doi.org/10.17577/ijertv9is050022
Nisha, S., Garg, K.: Churn prediction in telecommunication industry using decision tree. Int. J. Eng. Res. 6, 439–443 (2017). https://doi.org/10.17577/ijertv6is040379
Pamina, J., Beschi Raja, J., Sathya Bama, S., Soundarya, S., Sruthi, M.S., Kiruthika, S., et al.: An effective classifier for predicting churn in telecommunication. J. Adv. Rese. Dyn. Control Syst. 11, 221–229 (2019)
Hossin, M., Sulaiman, M.N.: A review on evaluation metrics for data classification evaluations. Int. J. Data Min. Knowl. Manag. Process 5, 01–11 (2015). https://doi.org/10.5121/ijdkp.2015.5201
Lalwani, P., Mishra, M.K., Chadha, J.S., Sethi, P.: Customer churn prediction system: a machine learning approach. Computing 104, 271–294 (2022). https://doi.org/10.1007/s00607-021-00908-y
Karanovic, M., Popovac, M., Sladojevic, S., Arsenovic, M., Stefanovic, D.: Telecommunication services churn prediction - deep learning approach. In: 2018 26th Telecommunications Forum, TELFOR 2018 - Proceedings, Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/TELFOR.2018.8612067
Ahmad, A.K., Jafar, A., Aljoumaa, K.: Customer churn prediction in telecom using machine learning in big data platform. J. Big Data 6(1), 1–24 (2019). https://doi.org/10.1186/s40537-019-0191-6
Ullah, I., Raza, B., Malik, A.K., Imran, M., Islam, S.U., Kim, S.W.: A churn prediction model using random forest: analysis of machine learning techniques for churn prediction and factor identification in telecom sector. IEEE Access 7, 60134–60149 (2019). https://doi.org/10.1109/ACCESS.2019.2914999
Cao, S., Liu, W., Chen, Y., Zhu, X.: Deep learning based customer churn analysis. n.d
Joolfoo, M., Jugurnauth, R., Joofloo, K.: Customer churn prediction in telecom using big data analytics. IOP Conf. Ser. Mater. Sci. Eng. 768 (2020). https://doi.org/10.1088/1757-899X/768/5/052070
Kavita, M., Sharma, N., Aggarwal, G.: Churn prediction of customer in telecommunications and e-commerce industry using machine learning. Palarch’s J. Archaeol. Egypt Egyptol. 17, 6–15 (2020)
Senthilnayaki, B.: Customer churn prediction. Iarjset 8, 527–531 (2021). https://doi.org/10.17148/iarjset.2021.8692
Singh, D., Jatana, V., Kanchana, M.: Survey paper on churn prediction on telecom. SSRN Electron. J. 27, 395–403 (2021). https://doi.org/10.2139/ssrn.3849664
Jain, H., Khunteta, A., Srivastava, S.: Telecom churn prediction using seven machine learning experiments integrating features engineering and normalisation. Comput. Sci. Sch. Basic Appl. Sci. (2021). Poornima University
Xu, T., Ma, Y., Kim, K.: Telecom churn prediction system based on ensemble learning using feature grouping. Appl. Sci. 11 (2021). https://doi.org/10.3390/app11114742
Baldominos, A., Cervantes, A., Saez, Y., Isasi, P.: A comparison of machine learning and deep learning techniques for activity recognition using mobile devices. Sensors 19 (2019). https://doi.org/10.3390/s19030521
Almaguer-Angeles, F., Murphy, J., Murphy, L., Portillo-Dominguez, A.O.: Choosing machine learning algorithms for anomaly detection in smart building IoT scenarios. In: IEEE 5th World Forum on Internet of Things, WF-IoT 2019 - Conference Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 491–495 (2019). https://doi.org/10.1109/WF-IoT.2019.8767357
Lantz, B.: Machine Learning with R: Expert Techniques for Predictive Modeling. Packt Publishing Ltd. (2019)
Vafeiadis, T., Diamantaras, K.I., Sarigiannidis, G., Chatzisavvas, K.C.: A comparison of machine learning techniques for customer churn prediction. Simul. Model. Pract. Theory 55, 1–9 (2015). https://doi.org/10.1016/j.simpat.2015.03.003
Vakili, M., Ghamsari, M., Rezaei, M.: Performance analysis and comparison of machine and deep learning algorithms for IoT data classification. n.d
Majnik, M., Bosnić, Z.: ROC analysis of classifiers in machine learning: a survey. Intell. Data Anal. 17, 531–558 (2013). https://doi.org/10.3233/IDA-130592
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Naidu, G., Zuva, T., Sibanda, E.M. (2023). A Review of Evaluation Metrics in Machine Learning Algorithms. In: Silhavy, R., Silhavy, P. (eds) Artificial Intelligence Application in Networks and Systems. CSOC 2023. Lecture Notes in Networks and Systems, vol 724. Springer, Cham. https://doi.org/10.1007/978-3-031-35314-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-35314-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35313-0
Online ISBN: 978-3-031-35314-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)