Detecting Network Anomalies Using CUSUM and EM Clustering

  • Wei Lu
  • Hengjian Tong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5821)


Intrusion detection has been extensively studied in the last two decades. However, most existing intrusion detection techniques detect limited number of attack types and report a huge number of false alarms. The hybrid approach has been proposed recently to improve the performance of intrusion detection systems (IDSs). A big challenge for constructing such a multi-sensor based IDS is how to make accurate inferences that minimize the number of false alerts and maximize the detection accuracy, thus releasing the security operator from the burden of high volume of conflicting event reports. We address this issue and propose a hybrid framework to achieve an optimal performance for detecting network traffic anomalies. In particular, we apply SNORT as the signature based intrusion detector and the other two anomaly detection methods, namely non-parametric CUmulative SUM (CUSUM) and EM based clustering, as the anomaly detector. The experimental evaluation with the 1999 DARPA intrusion detection evaluation dataset shows that our approach successfully detects a large portion of the attacks missed by SNORT while also reducing the false alarm rate.


Intrusion detection clustering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, J.P.: Computer Security Threat Monitoring and Surveillance. Technical Report, James P. Anderson Co., Fort Washington, Pennsylvania (1999)Google Scholar
  2. 2.
    Denning, D.E.: An Intrusion Detection Model. IEEE Transactions on Software Engineering 2, 222–232 (1987)CrossRefGoogle Scholar
  3. 3.
    Axelsson, S.: The Base-Rate Fallacy and the Difficulty of Intrusion Detection. ACM Transactions on Information and System Security (TISSEC) 3(3), 186–201 (2000)CrossRefGoogle Scholar
  4. 4.
    Barreno, M., Nelson, B., Sears, R., Joseph, A.D., Tygarcan, J.D.: Can Machine Learning be Secure? In: Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, pp. 16–25 (2006)Google Scholar
  5. 5.
    Sabhnani, M., Serpen, G.: Analysis of a Computer Security Dataset: Why Machine Learning Algorithms Fail on KDD Dataset for Misuse Detection. Intelligent Data Analysis 8(4), 403–415 (2004)Google Scholar
  6. 6.
    Patcha, A., Park, J.M.: An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technologies Trends. Computer Networks: The International Journal of Computer and Telecommunications Networking 51(12), 3448–3470 (2007)CrossRefGoogle Scholar
  7. 7.
    Barbarra, D., Couto, J., Jajodia, S., Popyack, L., Wu, N.: ADAM: Detecting Intrusions by Data Mining. In: Proceedings of the 2001 IEEE, Workshop on Information Assurance and Security, West Point, NY (June 2001)Google Scholar
  8. 8.
    Lunt, T.F., Tamaru, A., Gilham, F., Jagannathm, R., Jalali, C., Neumann, P.G., Javitz, H.S., Valdes, A., Garvey, T.D.: A Real-time Intrusion Detection Expert System (IDES). Technical Report, Computer Science Laboratory, SRI International, Menlo Park, USA (February 1992)Google Scholar
  9. 9.
    Anderson, D., Frivold, T., Tamaru, A., Valdes, A.: Next Generation Intrusion Detection Expert System (NIDES). Software Users Manual, Beta-Update release. Computer Science Laboratory, SRI International, Menlo Park, CA, USA, Technical Report SRI-CSL-95-0 (May 1994)Google Scholar
  10. 10.
    Porras, P., Neumann, P.: EMERALD: Event Monitoring Enabling Responses to Anomalous Live Disturbances. In: Proceedings of the 20th NIST-NCSC National Information Systems Security Conference, Baltimore, MD, USA, pp. 353–365 (1997)Google Scholar
  11. 11.
    Tombini, E., Debar, H., Mé, L., Ducassé, M.: A Serial Combination of Anomaly and Misuse IDSes Applied to HTTP traffic. In: Proceedings of the 20th Annual Computer Security Applications Conference, Tucson, AZ, USA (2004)Google Scholar
  12. 12.
    Zhang, J., Zulkernine, M.: A Hybrid Network Intrusion Detection Technique using Random Forests. In: Proceedings of the 1st International Conference on Availability, Reliability and Security, pp. 262–269. Vienna University of Technology (2006)Google Scholar
  13. 13.
    Peng, J., Feng, C., Rozenblit, J.W.: A Hybrid Intrusion Detection and Visualization System. In: Proceedings of the 13th Annual IEEE International Symposium and Workshop on Engineering of Computer Based Systems, pp. 505–506 (2006)Google Scholar
  14. 14.
    Depren, O., Topallar, M., Anarim, E., Ciliz, M.K.: An Intelligent Intrusion Detection System (IDS) for Anomaly and Misuse Detection in Computer Networks. Expert Systems with Applications 29(4), 713–722Google Scholar
  15. 15.
    Qin, M., Hwang, K., Cai, M., Chen, Y.: Hybrid Intrusion Detection with Weighted Signature Generation over Anomalous Internet Episodes. IEEE Transactions on Dependable and Secure Computing 4(1), 41–55Google Scholar
  16. 16.
    Xiang, C., Lim, S.M.: Design of Multiple-level Hybrid Classifier for Intrusion Detection System. In: Proceedings of the IEEE Workshop Machine Learning for Signal Processing, pp. 117–122 (2005)Google Scholar
  17. 17.
    Thames, J.L., Abler, R., Saad, A.: Hybrid Intelligent Systems for Network Security. In: Proceedings of the 44th ACM Annual Southeast Regional Conference, pp. 286–289Google Scholar
  18. 18.
    Peddabachigari, S., Abraham, A., Grosan, C., Thomas, J.: Modeling Intrusion Detection System using Hybrid Intelligent Systems. Special issue on Network and Information Security: A Computational Intelligence Approach. Journal of Network and Computer Applications 30(1), 114–132 (2007)CrossRefGoogle Scholar
  19. 19.
    Shon, T., Moon, J.: A Hybrid Machine Learning Approach to Network Anomaly Detection. International Journal on Information Sciences 177(18), 3799–3821 (2007)CrossRefGoogle Scholar
  20. 20.
    Sabhnani, M.R., Serpen, G.: Application of Machine Learning Algorithms to KDD Intrusion Detection Dataset within Misuse Detection Context. In: Proceedings of International Conference on Machine Learning: Models, Technologies, and Applications, pp. 209–215 (2003)Google Scholar
  21. 21.
  22. 22.
    Wang, H.N., Zhang, D.L., Hin, K.G.: Detecting SYN flooding attacks. In: Proceedings of IEEE INFOCOM 2002 (June 2002)Google Scholar
  23. 23.
    Lu, W., Traore, I.: Unsupervised anomaly detection using an evolutionary extension of K-means algorithm. International Journal on Information and Computer Security 2(2), 107–139 (2008)CrossRefGoogle Scholar
  24. 24.
    Peng, T., Leckie, C., Ramamohanarao, K.: Detecting distributed denial of service attacks using source IP address monitoring. Draft (November 2002)Google Scholar
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
    Mahoney, M.V., Chan, P.K.: An analysis of the 1999 DARPA/Lincoln Laboratory evaluation data for network anomaly detection. In: Proceedings of the 6th International Symposium on Recent Advances in Intrusion Detection, Pittsburgh, PA, USA, pp. 220–237 (2003)Google Scholar
  30. 30.
    McHugh, J.: Testing intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory. ACM Transactions on Information and System Security 3(4), 262–294 (2000)CrossRefGoogle Scholar
  31. 31.
    Gaffney, J.E., Ulvila, J.W.: Evaluation of intrusion detectors: a decision theory Approach. In: Proceeding of IEEE Symposium on Security and Privacy, pp. 50–61 (2001)Google Scholar
  32. 32.
  33. 33.
    Lu, W., Ghorbani, A.A.: Network anomaly detection based on wavelet analysis. EURASIP Journal on Advances in Signal Processing (2008) (in press)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Wei Lu
    • 1
    • 2
  • Hengjian Tong
    • 3
  1. 1.Faculty of Computer ScienceUniversity of New Brunswick, FrederictonCanada
  2. 2.Department of Electrical and Computer EngineeringUniversity of VictoriaVictoriaCanada
  3. 3.Department of Computing Science and Technology, School of Computer ScienceChina University of GeosciencesWuhanP.R. China

Personalised recommendations