MAC: Multilevel Autonomous Clustering for Topologically Distributed Anomaly Detection

  • M. A. ParthaEmail author
  • C. V. Ponce
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 881)


Anomaly detection in networks is an important cybersecurity threat detection capability. Anomalies in networks are often not localized to a single point, but are spread over a range of nodes. In this case of distributed anomalies, the anomalies are typically too subtle to detect at an individual-node level, and so require anomaly detection over groups of nodes. But it is usually not known a priori on which subset of nodes to focus, and it is infeasible to check all 2N subsets of nodes in a network. This renders distributed anomaly detection extremely challenging. An emerging strategy for detecting such anomalies is to apply a detection technique to a hierarchy of clusters of nodes in the network. However, developing such a hierarchy is challenging in large, decentralized networks with no central controller. In this work, we present Multilevel Autonomous Clustering (MAC), a novel local algorithm for self-organized, hierarchical clustering in distributed networks. MAC enables individual devices in a distributed system to determine their cluster membership at multiple levels using only local information, without centralized computation or information about the entire network. The result is an approach to hierarchical graph clustering that is both practical to use in large, real-world systems, as well as effective for distributed anomaly detection. The algorithm is evaluated on both synthetic and real-world networks. Its effectiveness for anomaly detection is demonstrated on various test problems.


Graphs Anomaly detection Local clustering Hierarchical Distributed networks Decentralized algorithms 



Dr. Alyson Fox, Dr. Eisha Nathan, and Dr. Tim La Fond, of Lawrence Livermore National Laboratory (LLNL), provided significant feedback and suggestions for improving the MAC algorithm. Professor Marija Ilic (MIT LIDS, MIT Lincoln Laboratory) provided valuable support and guidance. Within LLNL, Global Security’s E Program, Critical Infrastructure Cybersecurity Group and Engineering’s CED, Computational Electromagnetics Group both provided programmatic support. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.


  1. 1.
    Chatzigiannakis, V., Papavassiliou, S., Grammatikou, M., Maglaris, B.: Hierarchical anomaly detection in distributed large-scale sensor networks. In: 11th IEEE Symposium on Computers and Communications, pp. 761–767 (2006)Google Scholar
  2. 2.
    Palpanas, T., Papadopoulos, D., Kalogeraki, V., Gunopulos, D.: Distributed deviation detection in sensor networks. ACM SIGMOD Rec. 32(4), 77–82 (2003)CrossRefGoogle Scholar
  3. 3.
    Rajasegarar, S., Leckie, C., Palaniswami, M., Bezdek, J.C.: Distributed anomaly detection in wireless sensor networks. In: 10th IEEE Singapore International Conference on Communication Systems, pp. 1–5 (2006)Google Scholar
  4. 4.
    Tanachaiwiwat, S., Helmy, A.: Correlation analysis for alleviating effects of inserted data in wireless sensor networks. In: The Second Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, pp. 97–108 (2005)Google Scholar
  5. 5.
    De Paola, A., Gaglio, S., Re, G.L., Milazzo, F., Ortolani, M.: Adaptive distributed outlier detection for WSNs. IEEE Trans. Cybern. 45(5), 902–913 (2015)CrossRefGoogle Scholar
  6. 6.
    Robinson, J., Lonergan, M., Singh, L., Candido, A., Sayal, M.: SHARD: a framework for sequential, hierarchical anomaly ranking and detection. In: Tan, P.N., Chawla, S., Ho, C.K., Bailey, J. (eds.) PAKDD. LNCS, vol. 7302, pp. 243–255. Springer, Heidelberg (2012)Google Scholar
  7. 7.
    Rassam, M., Zainal, A., Maarof, M.: An efficient distributed anomaly detection model for wireless sensor networks. AASRI Procedia 5, 9–14 (2013)CrossRefGoogle Scholar
  8. 8.
    Bridges, R.A., Collins, J.P., Ferragut, E.M., Laska, J.A., Sullivan, B.D.: Multi-level anomaly detection on time-varying graph data. In: ASONAM 2015, pp. 579–583 (2015)Google Scholar
  9. 9.
    Bagrow, J.P., Bollt, E.M.: Local method for detecting communities. Phys. Rev. E 72(4), 046108 (2005)CrossRefGoogle Scholar
  10. 10.
    Clauset, A.: Finding local community structure in networks. Phys. Rev. E 72(2), 026132 (2005)CrossRefGoogle Scholar
  11. 11.
    Schaeffer, S.E.: Stochastic local clustering for massive graphs. In: Ho, T.B., Cheung, D., Liu, H. (eds.) PAKDD. LNCS, vol. 3518, pp. 354–360. Springer, Heidelberg (2005)Google Scholar
  12. 12.
    Schaeffer, S.E.: Graph clustering. Comput. Sci. Rev. 1, 27–64 (2007)CrossRefGoogle Scholar
  13. 13.
    Schaeffer, S.E., Marinoni, S.: Dynamic local clustering for hierarchical Ad Hoc networks. In: 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, vol. 2, pp. 667–672 (2006)Google Scholar
  14. 14.
    Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech.: Theory Exp. 2008(10), P10008 (2008)CrossRefGoogle Scholar
  15. 15.
    Zimmerman, R.D., Murillo-Sanchez, C.E., Thomas, R.J.: MATPOWER: steady-state operations, planning and analysis tools for power systems research and education. IEEE Trans. Power Syst. 26(1), 12–19 (2011)CrossRefGoogle Scholar
  16. 16.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Clauset, A., Newman, M.E.J., Moore, C.: Finding community structure in very large networks. Phys. Rev. E 70(6), 066111 (2004)CrossRefGoogle Scholar
  18. 18.
    Leskovec, J., Krevl, A.: SNAP datasets: stanford large network dataset collection (2014).
  19. 19.
    Hong, J., Liu, C., Govindarasu, M.: Integrated anomaly detection for cyber security of the substations. IEEE Trans. Smart Grid 5(4), 1643–1653 (2014)CrossRefGoogle Scholar
  20. 20.
    Bollobás, B.: 6. The evolution of random graphs – the giant component. Random Graphs, Cambridge Studies in Advanced Mathematics, vol. 73, 2nd edn., pp. 130–159. Cambridge University Press, Cambridge (2001)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Lawrence Livermore National LaboratoryLivermoreUSA
  2. 2.Massachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations