Using Unsupervised Learning for Network Alert Correlation

  • Reuben Smith
  • Nathalie Japkowicz
  • Maxwell Dondo
  • Peter Mason
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5032)


Alert correlation systems are post-processing modules that enable intrusion analysts to find important alerts and filter false positives efficiently from the output of Intrusion Detection Systems. Typically, however, these modules require high levels of human involvement in creating the system and/or maintaining it, as patterns of attacks change as often as from month to month. We present an alert correlation system based on unsupervised machine learning algorithms that is accurate and low maintenance. The system is implemented in two stages of correlation. At the first stage, alerts are grouped together such that each group forms one step of an attack. At the second stage, the groups created at the first stage are combined such that each combination of groups contains the alerts of precisely one full attack. We tested various implementations of the system. The most successful one relies in the first stage on a new unsupervised algorithm inspired by an existing novelty detection system, and the EM algorithm in the second stage. Our experimental results show that, with our model, the number of alerts that an analyst has to deal with is significantly reduced.


Cluster Algorithm Intrusion Detection Intrusion Detection System Hide Unit Attack Step 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Northcutt, S., et al.: SHADOW: Second heuristic analysis for defensive online warfareGoogle Scholar
  2. 2.
    Danyliw, R.: ACID: Analysis console for intrusion detectionsGoogle Scholar
  3. 3.
    Haines, J., Ryder, D.K., Tinnel, L., Taylor, S.: Validation of sensor alert correlators. IEEE Security and Privacy, 46–56 (2003)Google Scholar
  4. 4.
    Valdes, A., Skinner, K.: Probabilistic alert correlation. In: Lee, W., Mé, L., Wespi, A. (eds.) RAID 2001. LNCS, vol. 2212, pp. 54–68. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  5. 5.
    Hätälä, A., Särs, C., Addams-Moring, R., Virtanen, T.: Event data exchange and intrusion alert correlation in heterogeneous networks. In: Proceedings of the 8th Colloquium for Information Systems Security Education (CISSE), Westpoint, NY, CISSE, June 2004, pp. 84–92 (2004)Google Scholar
  6. 6.
    Smith, R., Japkowicz, N., Dondo, M.: Clustering using an autoassociator: A case study in network event correlation. In: Proceedings of the 17th IASTED International Conference on Parallel and Distributed Computing and Systems, Phoenix, AZ, November 2005, pp. 613–618. ACTA Press (2005)Google Scholar
  7. 7.
    Japkowicz, N., Smith, R.: Autocorrel ii: Unsupervised network event correlation using neural networks. Contractor Report CR 2005-155, DRDC Ottawa, Ottawa, ON (October 2005)Google Scholar
  8. 8.
    Julisch, K., Dacier, M.: Mining intrusion detection alarms for actionable knowledge. In: Proceedings of SIGKDD 2002, the 8th International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, July 2002, pp. 366–375. ACM Press, New York (2002)CrossRefGoogle Scholar
  9. 9.
    Dain, O., Cunningham, R.K.: Fusing a heterogeneous alert stream into scenarios. In: Proceedings of the 2001 ACM Workshop on Data Mining for Security Applications, Philadelphia, PA, November 2001, pp. 1–13. ACM Press, New York (2001)Google Scholar
  10. 10.
    Zanero, S., Savaresi, S.M.: Unsupervised learning techniques for an intrusion detection system. In: Proceedings of the 2004 ACM Symposium on Applied Computing, Nicosia, Cyprus, pp. 412–419. ACM, New York (2004)CrossRefGoogle Scholar
  11. 11.
    Laskov, P., Dussel, P., Rieck, C.S.: Learning intrusion detection: Supervised or unsupervised? In: Roli, F., Vitulano, S. (eds.) ICIAP 2005. LNCS, vol. 3617, pp. 50–57. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incoming data via the EM algorithm. J. Royal Stat. Soc., Series B 39(1), 1–36 (1977)zbMATHMathSciNetGoogle Scholar
  13. 13.
    Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences, vol. 30. Springer, Berlin (1995); (Second Extended Edition 1997) Google Scholar
  14. 14.
    Roesch, M.: Snort—lightweight intrusion detection for networks. In: Proceedings of LISA 1999: 13th Systems Administration Conference, Seattle, Washington, November 7–12, 1999, pp. 229–238. The USENIX Association (1999)Google Scholar
  15. 15.
    Lippmann, R., Haines, J.W., Fried, D.J., Korba, J., Das, K.: The 1999 darpa off-line intrusion detection evaluation. Computer Networks 34(4), 579–595 (2000)CrossRefGoogle Scholar
  16. 16.
    Northcutt, S.: Network Intrusion Detection: An Analyst’s Handbook. New Riders Publishing, Indianapolis (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Reuben Smith
    • 1
  • Nathalie Japkowicz
    • 1
  • Maxwell Dondo
    • 2
  • Peter Mason
    • 2
  1. 1.School of Information Technology and Engineering (SITE)University of Ottawa ON Canada 
  2. 2.Defence Research and Development Canada (DRDC) Ottawa ON Canada 

Personalised recommendations