Abstract
Anomaly detection in large-scale software systems is important to guarantee smooth operation of the system. Upon detection of an anomaly, it is vital to identify the root cause behind the anomaly to decipher actionable information and prevent future incidents. Isolation of root causes becomes inherently difficult as the number of components and parameters in each component increase. This paper discusses successful application of three drift detection techniques, namely meta algorithm, fixed cumulative window model and Page-Hinckley test to identify the parameters that correlate to system abnormalities in a large scale complex software system. Out of these, change detection meta algorithm produced the best result.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: SDM, vol. 7, pp. 443–448. SIAM (2007)
Brown, A., Kar, G., Keller, A.: An active approach to characterizing dynamic dependencies for problem determination in a distributed environment. In: Proceedings of 2001 IEEE/IFIP International Symposium on Integrated Network Management, pp. 377–390. IEEE (2001)
Chen, M.Y., Kiciman, E., Fratkin, E., Fox, A., Brewer, E.: Pinpoint: problem determination in large, dynamic internet services. In: Proceedings of the International Conference on Dependable Systems and Networks, DSN 2002, pp. 595–604. IEEE (2002)
Domingos, P., Hulten, G.: Mining high-speed data streams. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 71–80. ACM (2000)
Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with Drift Detection. Springer, Heidelberg (2004)
Gama, J., liobait, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Comput. Surv. (CSUR) 46(4), 44 (2014)
Grell, S., Nano, O.: Experimenting with complex event processing for large scale Internet services monitoring. In: Proceedings of iCEP (2008)
Hall, M.A.: Correlation-based feature selection for machine learning. Ph.D. thesis, The University of Waikato (1999)
Hangal, S., Lam, M.S.: Tracking down software bugs using automatic anomaly detection. In: Proceedings of the 24th International Conference on Software Engineering, pp. 291–301. ACM (2002)
Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 97–106. ACM (2001)
Julisch, K.: Clustering intrusion detection alarms to support root cause analysis. ACM Trans. Inf. Syst. Secur. (TISSEC) 6(4), 443–471 (2003)
Kifer, D., Ben-David, S., Gehrke, J.: Detecting change in data streams. In: Proceedings of the Thirtieth International Conference on Very Large Data Bases, vol. 30 (2004)
Lakhina, S., Joseph, S., Verma, B.: Feature reduction using principal component analysis for effective anomaly based intrusion detection on NSL-KDD (2010)
Mouss, H., Mouss, D., Mouss, N., Sefouhi, L.: Test of page-hinckley, an approach for fault detection in an agro-alimentary production system. In: 5th Asian Control Conference, vol. 2, pp. 815–818. IEEE (2004)
Page, E.: Continuous inspection schemes. Biometrika 41(1/2), 100–115 (1954)
Pechenizkiy, M., Bakker, J., liobait, I., Ivannikov, A., Krkkinen, T.: Online mass flow prediction in CFB boilers with explicit detection of sudden concept drift. ACM SIGKDD Explor. Newsletter 11(2), 109–116 (2010)
Sebastiao, R., Gama, J.: A study on change detection methods. In: Progress in Artificial Intelligence, 14th Portuguese Conference on Artificial Intelligence, EPIA, pp. 12–15 (2009)
Sebastiao, R., Gama, J.: Change detection in learning histograms from data streams. In: Portuguese Conference on Artificial Intelligence, pp. 112–123. Springer, Heidelberg (2007)
Sebastiao, R., Gama, J., Rodrigues, P.P., Bernardes, J.: Monitoring incremental histogram distribution for change detection in data streams. In: Knowledge Discovery from Sensor Data, pp. 25–42. Springer, Heidelberg (2010)
Sez, C., Rodrigues, P.P., Gama, J., Robles, M., Garca-Gmez, J.M.: Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality. Data Min. Knowl. Discov. 29(4), 950–975 (2015)
Zheng, T., Yang, J., Woodside, M., Litoiu, M., Iszlai, G.: Tracking time-varying parameters in software systems with extended kalman filters. In: Proceedings of the 2005 Conference of the Centre for Advanced Studies on Collaborative Research, pp. 334–345. IBM Press (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Prabodha, L.H.C., Vithanage, W.R.R., Ranaweera, L.T., Dissanayake, D.M.M.A.I.B., Ranathunga, S. (2018). Monitoring Health of Large Scale Software Systems Using Drift Detection Techniques. In: Barolli, L., Terzo, O. (eds) Complex, Intelligent, and Software Intensive Systems. CISIS 2017. Advances in Intelligent Systems and Computing, vol 611. Springer, Cham. https://doi.org/10.1007/978-3-319-61566-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-61566-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61565-3
Online ISBN: 978-3-319-61566-0
eBook Packages: EngineeringEngineering (R0)