MAC: Multilevel Autonomous Clustering for Topologically Distributed Anomaly Detection
Anomaly detection in networks is an important cybersecurity threat detection capability. Anomalies in networks are often not localized to a single point, but are spread over a range of nodes. In this case of distributed anomalies, the anomalies are typically too subtle to detect at an individual-node level, and so require anomaly detection over groups of nodes. But it is usually not known a priori on which subset of nodes to focus, and it is infeasible to check all 2N subsets of nodes in a network. This renders distributed anomaly detection extremely challenging. An emerging strategy for detecting such anomalies is to apply a detection technique to a hierarchy of clusters of nodes in the network. However, developing such a hierarchy is challenging in large, decentralized networks with no central controller. In this work, we present Multilevel Autonomous Clustering (MAC), a novel local algorithm for self-organized, hierarchical clustering in distributed networks. MAC enables individual devices in a distributed system to determine their cluster membership at multiple levels using only local information, without centralized computation or information about the entire network. The result is an approach to hierarchical graph clustering that is both practical to use in large, real-world systems, as well as effective for distributed anomaly detection. The algorithm is evaluated on both synthetic and real-world networks. Its effectiveness for anomaly detection is demonstrated on various test problems.
KeywordsGraphs Anomaly detection Local clustering Hierarchical Distributed networks Decentralized algorithms
Dr. Alyson Fox, Dr. Eisha Nathan, and Dr. Tim La Fond, of Lawrence Livermore National Laboratory (LLNL), provided significant feedback and suggestions for improving the MAC algorithm. Professor Marija Ilic (MIT LIDS, MIT Lincoln Laboratory) provided valuable support and guidance. Within LLNL, Global Security’s E Program, Critical Infrastructure Cybersecurity Group and Engineering’s CED, Computational Electromagnetics Group both provided programmatic support. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
- 1.Chatzigiannakis, V., Papavassiliou, S., Grammatikou, M., Maglaris, B.: Hierarchical anomaly detection in distributed large-scale sensor networks. In: 11th IEEE Symposium on Computers and Communications, pp. 761–767 (2006)Google Scholar
- 3.Rajasegarar, S., Leckie, C., Palaniswami, M., Bezdek, J.C.: Distributed anomaly detection in wireless sensor networks. In: 10th IEEE Singapore International Conference on Communication Systems, pp. 1–5 (2006)Google Scholar
- 4.Tanachaiwiwat, S., Helmy, A.: Correlation analysis for alleviating effects of inserted data in wireless sensor networks. In: The Second Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, pp. 97–108 (2005)Google Scholar
- 6.Robinson, J., Lonergan, M., Singh, L., Candido, A., Sayal, M.: SHARD: a framework for sequential, hierarchical anomaly ranking and detection. In: Tan, P.N., Chawla, S., Ho, C.K., Bailey, J. (eds.) PAKDD. LNCS, vol. 7302, pp. 243–255. Springer, Heidelberg (2012)Google Scholar
- 8.Bridges, R.A., Collins, J.P., Ferragut, E.M., Laska, J.A., Sullivan, B.D.: Multi-level anomaly detection on time-varying graph data. In: ASONAM 2015, pp. 579–583 (2015)Google Scholar
- 11.Schaeffer, S.E.: Stochastic local clustering for massive graphs. In: Ho, T.B., Cheung, D., Liu, H. (eds.) PAKDD. LNCS, vol. 3518, pp. 354–360. Springer, Heidelberg (2005)Google Scholar
- 13.Schaeffer, S.E., Marinoni, S.: Dynamic local clustering for hierarchical Ad Hoc networks. In: 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, vol. 2, pp. 667–672 (2006)Google Scholar
- 18.Leskovec, J., Krevl, A.: SNAP datasets: stanford large network dataset collection (2014). http://snap.stanford.edu/data
- 20.Bollobás, B.: 6. The evolution of random graphs – the giant component. Random Graphs, Cambridge Studies in Advanced Mathematics, vol. 73, 2nd edn., pp. 130–159. Cambridge University Press, Cambridge (2001)Google Scholar