Abstract
Although appropriate feature extraction can improve the performance of anomaly detection, it is a challenging task due to the complex interaction between features, the mixture of irrelevant features and relevant features, and the unavailability of data tags. When conventional anomaly detection methods deal with the problem of anomaly detection of high dimensional data, the performance of anomaly detection will be degraded due to the existence of irrelevant features. This paper proposed a method of feature extraction and anomaly detection for high dimensional data based on elastic network, which can filter irrelevant features and improve the accuracy and efficiency of anomaly detection. In this paper, an outlier scoring method was used to score the outliers of the original data, and then outliers and the original data were input into the elastic network for sparse regression. After feature extraction of elastic network, those irrelevant features to abnormal data are ignored, thus reducing the dimension of data. Finally, high-dimensional data are detected efficiently according to extracted features. In the experimental stage, we used the high-dimensional anomaly dataset provided by ODDS to detect the performance of the proposed method based on AUC detection accuracy, ROC curve, feature number, convergence speed and other indicators. The results show that the proposed method not only can effectively extract the features related to high-dimensional anomaly data, but also the detection accuracy of outliers has been greatly improved.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hawkins, D.M.: Identification of Outliers, 1st edn, pp. 11–12. Springer, Berlin (1980). https://doi.org/10.1007/978-94-015-3994-4
Li, L., Hao, Z., Peng, H., et al.: Nearest neighbors based density peaks approach to intrusion detection. Chaos Solitons Fractals 110, 33–40 (2018)
Wang, C., Dong, H.: Credit card fraud forecasting model based on clustering analysis and integrated support vector machine. Cluster Comput. 16, 1–6 (2018)
Gao, X., Fang, Y.: Penalized weighted least squares for outlier detection and robust regression (2016)
Chen, T., Martin, E., Montague, G.: Robust probabilistic PCA with missing data and contribution analysis for outlier detection. Comput. Stat. Data Anal. 53(10), 3706–3716 (2009)
Aggarwal, C.C.: Linear models for outlier detection. In: Aggarwal, C.C. (ed.) Outlier Analysis, pp. 75–99. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6396-2_3
Dalatu, P.I., Fitrianto, A., Mustapha, A.: A comparative study of linear and nonlinear regression models for outlier detection (2016)
Pérez, B., Molina, I., Peña, D.: Outlier detection and robust estimation in linear regression models with fixed group effects. J. Stat. Comput. Simul. 84(12), 2652–2669 (2014)
Xu, H., Mao, R., Liao, H., et al.: Closest neighbors excluded outlier detection. In: Online Analysis & Computing Science, pp. 105–110. IEEE (2016)
Liu, J., Wang, G.: Outlier detection based on local minima density. In: IEEE Information Technology, Networking, Electronic & Automation Control Conference, pp. 718–723. IEEE (2016)
Vwema, P., Yadava, R.D.S.: Fuzzy c-means clustering based outlier detection for SAW electronic nose. In: Convergence in Technology, pp. 513–519. IEEE (2017)
Bolón-Canedo, V., Sánchez-Maro, N., Alonso-Betanzos, A.: Recent advances and emerging challenges of feature selection in the context of big data. Knowl.-Based Syst. 86, 33–45 (2015)
Du, L., Shen, Y.D.: Unsupervised feature selection with adaptive structure learning (2015)
Yu, K., Wu, X., Ding, W., et al.: Scalable and accurate online feature selection for big data. ACM Trans. Knowl. Discovery Data 11, 16 (2015)
Zhang, C., Wang, G., Zhou, Y., et al.: Feature selection for high dimensional imbalanced class data based on F-measure optimization. In: International Conference on Security. IEEE (2018)
Feng, S., Sen, P.N.: Percolation on elastic networks: new exponent and threshold. Phys. Rev. Lett. 52(3), 216–219 (1984)
Liu, F., Ting, K.M., Zhou, Z.H.: Isolation-based anomaly detection. ACM Trans. Knowl. Discovery Data 6(1), 1556–4681 (2012)
Yang, Z., Zheng, Y., Gao, Y., et al.: Abnormal data detection for an e-business using object-oriented approach. In: Integration and Innovation Orient to E-Society, vol. 1 (2007)
Zimek, A., Gaudet, M., Campello, R.J.G.B., et al.: Subsampling for efficient and effective unsupervised outlier detection ensembles. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM (2013)
Acknowledgement
The research was supported by the State Grid Liaoning Electric Power Supply CO., LTD, and we are grateful for the financial support for the “Key Technology and Application Research of the Self-Service Grid Big Data Governance (SGLNXT00YJJS1800110)”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Shen, Y., Bo, J., Li, K., Chen, S., Qiao, L., Li, J. (2019). High-Dimensional Data Anomaly Detection Framework Based on Feature Extraction of Elastic Network. In: Zhai, X., Chen, B., Zhu, K. (eds) Machine Learning and Intelligent Communications. MLICOM 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 294. Springer, Cham. https://doi.org/10.1007/978-3-030-32388-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-32388-2_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32387-5
Online ISBN: 978-3-030-32388-2
eBook Packages: Computer ScienceComputer Science (R0)