Anomalous Payload-Based Network Intrusion Detection

  • Ke Wang
  • Salvatore J. Stolfo
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3224)


We present a payload-based anomaly detector, we call PAYL, for intrusion detection. PAYL models the normal application payload of network traffic in a fully automatic, unsupervised and very effecient fashion. We first compute during a training phase a profile byte frequency distribution and their standard deviation of the application payload flowing to a single host and port. We then use Mahalanobis distance during the detection phase to calculate the similarity of new data against the pre-computed profile. The detector compares this measure against a threshold and generates an alert when the distance of the new input exceeds this threshold. We demonstrate the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset we collected on the Columbia CS department network. In once case nearly 100% accuracy is achieved with 0.1% false positive rate for port 80 traffic.


  1. 1.
    Armstrong, D., Carter, S., Frazier, G., Frazier, T.: A Controller-Based Autonomic Defense System. In: Proc. of DISCEX (2003)Google Scholar
  2. 2.
    Damashek, M.: Gauging similarity with n-grams: language independent categorization of text. Science 267(5199), 843–848 (1995)CrossRefGoogle Scholar
  3. 3.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A Sense of self for Unix Processes. In: Proc. of IEEE Symposium on Computer Security and Privacy (1996)Google Scholar
  4. 4.
    Ghosh, A.K., Schwartzbard, A.: A study in Using Neural Networks for Anomaly and Misuse Detection. In: Proc. 8th USENIX Security Symposium (1999)Google Scholar
  5. 5.
    Hoagland, J.: SPADE, Silican Defense, (2000)
  6. 6.
    Javits, H.S., Valdes, A.: The NIDES statistical component: Description and justification. Technical report, SRI International, Computer Science Laboratory (1993)Google Scholar
  7. 7.
    Knuth, D.E.: the Art of Computer Programming, 2nd edn. Fundamental Algorithms, vol. 1. Addison Wesley, Reading (1973)Google Scholar
  8. 8.
    Kruegel, C., Toth, T., Kirda, E.: Service Specific Anomaly Detection for Network Intrusion Detection. In: Symposium on Applied Computing (SAC), Spain (March 2002)Google Scholar
  9. 9.
    Lee, W., Stolfo, S.: A Framework for Constructing Features and Models for Intrusion Detection Systems. ACM Transactions on Information and System Security 3(4) (November 2000)Google Scholar
  10. 10.
    Lippmann, R., et al.: The 1999 DARPA Off-Line Intrusion Detection Evaluation. Computer Networks 34(4), 579–595 (2000)CrossRefGoogle Scholar
  11. 11.
    Locasto, M., Parekh, J., Stolfo, S., Keromytis, A., Malkin, T., Misra, V.: Collaborative Distributed Intrusion Detection, Columbia University Tech Report, CUCS-012-04 (2004)Google Scholar
  12. 12.
    Mahoney, M.: Network Traffic Anomaly Detection Based on Packet Bytes. In: Proc. ACMSAC (2003)Google Scholar
  13. 13.
    Mahoney, M., Chan, P.K.: Learning Nonstationary Models of Normal Network Traffic for Detecting Novel Attacks. In: Proc. SIGKDD 2002, pp. 376–385 (2002)Google Scholar
  14. 14.
    Mahoney, M., Chan, P.K.: Learning Models of Network Traffic for Detecting Novel Attacks, Florida Tech, Technical report 2002-08,
  15. 15.
    Mahoney, M., Chan, P.K.: An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection. In: Vigna, G., Krügel, C., Jonsson, E. (eds.) RAID 2003. LNCS, vol. 2820, pp. 220–237. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  16. 16.
    Moore, D., Shannon, C., Voelker, G., Savage, S.: Internet Quarantine: Requirements for Containing Selp-Propagating Code. In: Proc. Infocom (2003)Google Scholar
  17. 17.
    V. Paxson, Bro: A system for detecting network intruders in real-time. In: USENIX Security Symposium (1998) Google Scholar
  18. 18.
    Porras, P., Neumann, P.: EMERALD: Event Monitoring Enabled Responses to Anomalous Live Disturbances. In: National Information Systems Security Conference (1997)Google Scholar
  19. 19.
    Robertson, S., Siegel, E., Miller, M., Stolfo, S.: Surveillance Detection in High Bandwidth Environments. In: Proceedings of the 2003 DARPA DISCEX III Conference (2003)Google Scholar
  20. 20.
    Roesch, M.: Snort: Lightweight intrusion detection for networks. In: USENIX LISA Conference (1999)Google Scholar
  21. 21.
    Staniford, S., Paxson, V., Weaver, N.: How to Own the Internet in Your Spare Time. In: Proceedings of the 11th USENIX Security Symposium (2002)Google Scholar
  22. 22.
    Stolfo, S.: Worm and Attack Early Warning: Piercing Stealthy Reconnaissance. IEEE Privacy and Security (May/June 2004) (to appear)Google Scholar
  23. 23.
    Taylor, C., Alves-Foss, J.: NATE – Network Analysis of Anomalous Traffic Events, A Low-Cost approach. In: New Security Paradigms Workshop (2001)Google Scholar
  24. 24.
    Vigna, G., Kemmerer, R.: NetSTAT: A Network-based intrusion detection approach. In: Computer Security Application Conference (1998)Google Scholar
  25. 25.
    Lane, T., Broadley, C.E.: Approaches to online learning and concept drift for user identification in computer security. In: 4th International Conference on Knowledge Discovery and Data Mining (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Ke Wang
    • 1
  • Salvatore J. Stolfo
    • 1
  1. 1.Computer Science DepartmentColumbia UniversityNew York

Personalised recommendations