Anomalous Payload-Based Network Intrusion Detection

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3224)


We present a payload-based anomaly detector, we call PAYL, for intrusion detection. PAYL models the normal application payload of network traffic in a fully automatic, unsupervised and very effecient fashion. We first compute during a training phase a profile byte frequency distribution and their standard deviation of the application payload flowing to a single host and port. We then use Mahalanobis distance during the detection phase to calculate the similarity of new data against the pre-computed profile. The detector compares this measure against a threshold and generates an alert when the distance of the new input exceeds this threshold. We demonstrate the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset we collected on the Columbia CS department network. In once case nearly 100% accuracy is achieved with 0.1% false positive rate for port 80 traffic.


Intrusion Detection Mahalanobis Distance Anomaly Detection Network Packet Network Intrusion Detection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Armstrong, D., Carter, S., Frazier, G., Frazier, T.: A Controller-Based Autonomic Defense System. In: Proc. of DISCEX (2003)Google Scholar
  2. 2.
    Damashek, M.: Gauging similarity with n-grams: language independent categorization of text. Science 267(5199), 843–848 (1995)CrossRefGoogle Scholar
  3. 3.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A Sense of self for Unix Processes. In: Proc. of IEEE Symposium on Computer Security and Privacy (1996)Google Scholar
  4. 4.
    Ghosh, A.K., Schwartzbard, A.: A study in Using Neural Networks for Anomaly and Misuse Detection. In: Proc. 8th USENIX Security Symposium (1999)Google Scholar
  5. 5.
    Hoagland, J.: SPADE, Silican Defense, (2000)
  6. 6.
    Javits, H.S., Valdes, A.: The NIDES statistical component: Description and justification. Technical report, SRI International, Computer Science Laboratory (1993)Google Scholar
  7. 7.
    Knuth, D.E.: the Art of Computer Programming, 2nd edn. Fundamental Algorithms, vol. 1. Addison Wesley, Reading (1973)Google Scholar
  8. 8.
    Kruegel, C., Toth, T., Kirda, E.: Service Specific Anomaly Detection for Network Intrusion Detection. In: Symposium on Applied Computing (SAC), Spain (March 2002)Google Scholar
  9. 9.
    Lee, W., Stolfo, S.: A Framework for Constructing Features and Models for Intrusion Detection Systems. ACM Transactions on Information and System Security 3(4) (November 2000)Google Scholar
  10. 10.
    Lippmann, R., et al.: The 1999 DARPA Off-Line Intrusion Detection Evaluation. Computer Networks 34(4), 579–595 (2000)CrossRefGoogle Scholar
  11. 11.
    Locasto, M., Parekh, J., Stolfo, S., Keromytis, A., Malkin, T., Misra, V.: Collaborative Distributed Intrusion Detection, Columbia University Tech Report, CUCS-012-04 (2004)Google Scholar
  12. 12.
    Mahoney, M.: Network Traffic Anomaly Detection Based on Packet Bytes. In: Proc. ACMSAC (2003)Google Scholar
  13. 13.
    Mahoney, M., Chan, P.K.: Learning Nonstationary Models of Normal Network Traffic for Detecting Novel Attacks. In: Proc. SIGKDD 2002, pp. 376–385 (2002)Google Scholar
  14. 14.
    Mahoney, M., Chan, P.K.: Learning Models of Network Traffic for Detecting Novel Attacks, Florida Tech, Technical report 2002-08,
  15. 15.
    Mahoney, M., Chan, P.K.: An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection. In: Vigna, G., Krügel, C., Jonsson, E. (eds.) RAID 2003. LNCS, vol. 2820, pp. 220–237. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  16. 16.
    Moore, D., Shannon, C., Voelker, G., Savage, S.: Internet Quarantine: Requirements for Containing Selp-Propagating Code. In: Proc. Infocom (2003)Google Scholar
  17. 17.
    V. Paxson, Bro: A system for detecting network intruders in real-time. In: USENIX Security Symposium (1998) Google Scholar
  18. 18.
    Porras, P., Neumann, P.: EMERALD: Event Monitoring Enabled Responses to Anomalous Live Disturbances. In: National Information Systems Security Conference (1997)Google Scholar
  19. 19.
    Robertson, S., Siegel, E., Miller, M., Stolfo, S.: Surveillance Detection in High Bandwidth Environments. In: Proceedings of the 2003 DARPA DISCEX III Conference (2003)Google Scholar
  20. 20.
    Roesch, M.: Snort: Lightweight intrusion detection for networks. In: USENIX LISA Conference (1999)Google Scholar
  21. 21.
    Staniford, S., Paxson, V., Weaver, N.: How to Own the Internet in Your Spare Time. In: Proceedings of the 11th USENIX Security Symposium (2002)Google Scholar
  22. 22.
    Stolfo, S.: Worm and Attack Early Warning: Piercing Stealthy Reconnaissance. IEEE Privacy and Security (May/June 2004) (to appear)Google Scholar
  23. 23.
    Taylor, C., Alves-Foss, J.: NATE – Network Analysis of Anomalous Traffic Events, A Low-Cost approach. In: New Security Paradigms Workshop (2001)Google Scholar
  24. 24.
    Vigna, G., Kemmerer, R.: NetSTAT: A Network-based intrusion detection approach. In: Computer Security Application Conference (1998)Google Scholar
  25. 25.
    Lane, T., Broadley, C.E.: Approaches to online learning and concept drift for user identification in computer security. In: 4th International Conference on Knowledge Discovery and Data Mining (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  1. 1.Computer Science DepartmentColumbia UniversityNew York

Personalised recommendations