Abstract
Real-time anomaly detection of massive data streams is an important research topic nowadays due to the fact that a lot of data is generated in continuous temporal processes. Holt-Winters (HW) and Taylor’s Double Holt-Winters (TDHW) forecasting models are used to predict the normal behavior of the periodic streams, and to detect anomalies when the deviations of observed and predicted values exceeded some predefined measures. In this work, we propose an enhancement of this approach. We implement the Genetic Algorithm (GA) to periodically optimize HW and TDHW smoothing parameters in addition to the two sliding windows parameters that improve Hyndman’s MASE measure of deviation, and value of the threshold parameter that defines no anomaly confidence interval. We also propose a new optimization function based on the input training datasets with the annotated anomaly intervals, in order to detect the right anomalies and minimize the number of false ones. The proposed method is evaluated on the known anomaly detection benchmarks NUMENTA and Yahoo datasets with annotated anomalies and real log data generated by the National education information system (NEIS) (http://ednevnik.edu.mk/) in Macedonia.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
De Gooijer, J.G., Hyndman, R.J.: 25 years of time series forecasting. Int. J. Forecast. 22(3), 443–473 (2006)
Adhikari, R., Agrawal, R.K.: An introductory study on time series modeling and forecasting. LAP Lambert Academic Publishing, Germany (2013)
Chandola, V., Banerjee, A., Kumar, V.: Outlier detection: a survey. ACM Comput. Surv. 41(3), 1–58 (2009)
Brutlag, J.D.: Aberrant behavior detection in time series for network monitoring. In: LISA 2000 Proceedings of the 14th USENIX Conference on System Administration, pp. 139–146. ACM, Louisiana (2000)
Ekberg, J., Ylinen, J., Loula, P.: Network behaviour anomaly detection using Holt-Winters algorithm. In: 6th International Conference on Internet Technology and Secured Transactions, pp. 627–631. IEEE, Piscataway (2011)
Andrysiak, T., Saganowski, Ł., Maszewski, M.: Time series forecasting using Holt-Winters model applied to anomaly detection in network traffic. In: Pérez García, H., Alfonso-Cendón, J., Sánchez González, L., Quintián, H., Corchado, E. (eds.) SOCO/CISIS/ICEUTE -2017. AISC, vol. 649, pp. 567–576. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-67180-2_55
Ravinder, H.V.: Determining the optimal values of exponential smoothing constants - does solver really work? Am. J. Bus. Educ. 6(3), 347–360 (2013)
Li, G., Wang, J., Liang, J., Yue, C.: The application of a double CUSUM algorithm in industrial data stream anomaly detection. Symmetry 10(7), 264 (2018). 2–14
Agapie, A., Agapie, A.: Forecasting the economic cycles based on an extension of the Holt-Winters model. a genetic algorithms approach. In: Proceedings of the IEEE/IAFE Computational Intelligence for Financial Engineering (CIFEr), pp. 96–99. IEEE, New York City (1997)
Eusébio, E., Camus, C., Curvelo, C.: Metaheuristic approach to the Holt-Winters optimal short term load forecast. Renew. Energy Power Qual. J. 10(13), 708–713 (2015)
Shahin, A.A.: Using multiple seasonal Holt-Winters exponential smoothing to predict cloud resource provisioning. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 7(11), 91–96 (2016)
Ortiz, R.R.L.: The accuracy rate of Holt-Winters model with particle swarm optimization in forecasting exchange rates. J. Comput. 11(3), 216–224 (2016)
Cortez, P., Rocha, M., Neves, J.: Genetic and evolutionary algorithms for time series forecasting. In: Monostori, L., Váncza, J., Ali, M. (eds.) IEA/AIE 2001. LNCS (LNAI), vol. 2070, pp. 393–402. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45517-5_44
Hyndman, R.J., Koehler, A.B.: Another look at forecast-accuracy metrics for intermittent demand. Int. J. Forecast. 22(4), 679–688 (2006)
Scrucca, L.: GA: a package for genetic algorithms in R. J. Stat. Softw. 53(4), 1–53 (2013)
de Assis, M.V.O., Carvalho, L.F., Rodrigues, J.J.P.C., Proença, M.L.: Holt-Winters statistical forecasting and ACO metaheuristic for traffic characterization. In: 2013 IEEE International Conference on Communications (ICC), Budapest, Hungary, pp. 2524–2528 (2013)
Maarof, M.Z.M., Ismail, Z., Fadzli, M.: Optimization of SARIMA model using genetic algorithm method in forecasting Singapore tourist arrivals to Malaysia. Appl. Math. Sci. 8(170), 8481–8491 (2014)
Ahmad, S., Purdy, S.: Real-time anomaly detection for streaming analytics. arXiv, pp. 1–10 (2016)
Hamamoto, A.H., Carvalho, L.F., Sampaio, L.D.H., Abrão, T., Proencüa Jr., M.L.: Network anomaly detection system using genetic algorithm and fuzzy logic. Expert. Syst. Appl.: Int. J. 99(C), 390–402 (2017)
Hyndman, R.J., Athanasopoulos, G.: Forecasting: Principles and Practice, 2nd edn. Texts Monash University, Australia (2018)
Hasani, Z., Jakimovski, B., Kon-Popovska, M., Velinov, G.: Real-time analytic of SQL queries based on log analytic. In: Loshkovska, S., Koceski, S. (eds.) ICT Innovations 2015, Web Proceedings, pp. 78–87. ICT ACT, Macedonia (2015)
Hasani, Z.: Implementation of infrastructure for streaming outlier detection in big data. In: Rocha, Á., Correia, A.M., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST 2017. AISC, vol. 570, pp. 503–511. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56538-5_51
Hasani, Z.: Robust anomaly detection algorithms for real-time big data: comparison of algorithms. In: 6th Mediterranean Conference on Embedded Computing (MECO), pp. 1–6. IEEE, Montenegro (2017)
Yahoo: S5 - A Labeled Anomaly Detection Dataset, version 1.0(16M). https://webscope.sandbox.yahoo.com/catalog.php?datatype=s%5c&did=70. Accessed 28 Apr 2018
NUMENTA anomaly benchmark with labeled anomalies. https://github.com/numenta/NAB/tree/master/data/artificialWithAnomaly. Accessed 28 Apr 2018
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Hasani, Z., Jakimovski, B., Velinov, G., Kon-Popovska, M. (2018). An Adaptive Anomaly Detection Algorithm for Periodic Data Streams. In: Yin, H., Camacho, D., Novais, P., Tallón-Ballesteros, A. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2018. IDEAL 2018. Lecture Notes in Computer Science(), vol 11314. Springer, Cham. https://doi.org/10.1007/978-3-030-03493-1_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-03493-1_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03492-4
Online ISBN: 978-3-030-03493-1
eBook Packages: Computer ScienceComputer Science (R0)