An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection

  • Matthew V. Mahoney
  • Philip K. Chan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2820)

Abstract

The DARPA/MIT Lincoln Laboratory off-line intrusion detection evaluation data set is the most widely used public benchmark for testing intrusion detection systems. Our investigation of the 1999 background network traffic suggests the presence of simulation artifacts that would lead to overoptimistic evaluation of network anomaly detection systems. The effect can be mitigated without knowledge of specific artifacts by mixing real traffic into the simulation, although the method requires that both the system and the real traffic be analyzed and possibly modified to ensure that the system does not model the simulated traffic independently of the real traffic.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Lippmann, R., et al.: The 1999 DARPA Off-Line Intrusion Detection Evaluation. Computer Networks 34(4), 579–595 (2000), Data is available at http://www.ll.mit.edu/IST/ideval/ CrossRefGoogle Scholar
  2. 2.
    Lippmann, R.P., Haines, J.: Analysis and Results of the, DARPA Off-Line Intrusion Detection Evaluation, in Recent Advances in Intrusion Detection. In: Third International Workshop, Proc. RAID 2000, pp. 162–182 (2000)Google Scholar
  3. 3.
    Haines, J.W., Lippmann, R.P., Fried, D.J., Zissman, M.A., Tran, E., Boswell, S.B.: 1999 DARPA Intrusion Detection Evaluation: Design and Procedures. MIT Lincoln Laboratory, Lexington (2001)Google Scholar
  4. 4.
    D. Barbara, Wu, S. Jajodia, "Detecting Novel Network Attacks using Bayes Estimators", Proc. SIAM Intl. Data Mining Conference, 2001. Google Scholar
  5. 5.
    Valdes, A., Skinner, K.: Adaptive, Model-based Monitoring for Cyber Attack Detection. In: Proc. RAID 2000, pp. 80–92 (2000)Google Scholar
  6. 6.
    Mahoney, M., Chan, P.K.: PHAD: Packet Header Anomaly Detection for Identifying Hostile Network Traffic, Florida Tech. technical report CS-2001-2004, http://cs.fit.edu/~tr/
  7. 7.
    Mahoney, M., Chan, P.K.: Learning Nonstationary Models of Normal Network Traffic for Detecting Novel Attacks. In: Proc. SIGKDD 2002, pp. 376–385 (2002)Google Scholar
  8. 8.
    Mahoney, M., Chan, P.K.: Learning Models of Network Traffic for Detecting Novel Attacks, Florida Tech. technical report CS-2002-2008, http://cs.fit.edu/~tr/
  9. 9.
    Mahoney, M.: Network Traffic Anomaly Detection Based on Packet Bytes. In: Proc. ACMSAC (2003)Google Scholar
  10. 10.
    Eskin, E.: Anomaly Detection over Noisy Data using Learned Probability Distributions. In: Proc. Intl. Conf. Machine Learning (2000)Google Scholar
  11. 11.
    Eskin, E., Arnold, A., Prerau, M., Portnoy, L., Stolfo, S.: A Geometric Framework for Unsupervised Anomaly Detection: Detecting Intrusions in Unlabeled Data. In: Barbara, D., Jajodia, S. (eds.) Applications of Data Mining in Computer Security, Kluwer, Dordrecht (2002)Google Scholar
  12. 12.
    Ghosh, A.K., Schwartzbard, A.: A Study in Using Neural Networks for Anomaly and Misuse Detection. In: Proc. 8’th USENIX Security Symposium 1999 (1999)Google Scholar
  13. 13.
    Liao, Y., Vemuri, V.R.: Use of Text Categorization Techniques for Intrusion Detection. In: Proc. 11th USENIX Security Symposium, pp. 51–59 (2002)Google Scholar
  14. 14.
    Neumann, P.G., Porras, P.A.: Experience with EMERALD to DATE. In: Proc. 1st USENIX Workshop on Intrusion Detection and Network Monitoring, pp. 73–80 (1999)Google Scholar
  15. 15.
    Schwartzbard, A., Ghosh, A.K.: A Study in the Feasibility of Performing Host-based Anomaly Detection on Windows NT. In: Proc. RAID 1999 (1999)Google Scholar
  16. 16.
    Sekar, R., Gupta, A., Frullo, J., Shanbhag, T., Zhou, S., Tiwari, A., Yang, H.: Specification Based Anomaly Detection: A New Approach for Detecting Network Intrusions. In: Proc. ACM CCS (2002)Google Scholar
  17. 17.
    Sekar, R., Uppuluri, P.: Synthesizing Fast Intrusion Prevention/Detection Systems from High-Level Specifications. In: Proc. 8th USENIX Security Symposium 1999 (1999)Google Scholar
  18. 18.
    Tyson, M., Berry, P., Williams, N., Moran, D., Blei, D.: DERBI: Diagnosis, Explanation and Recovery from computer Break-Ins. (2000), http://www.ai.sri.com/~derbi/
  19. 19.
    Vigna, G., Eckmann, S.T., Kemmerer, R.A.: The STAT Tool Suite. In: Proc. 2000 DARPA Information Survivability Conference and Exposition (DISCEX), IEEE Press, Los Alamitos (2000)Google Scholar
  20. 20.
    Vigna, G., Kemmerer, R.: NetSTAT: A Network-based Intrusion Detection System. Journal of Computer Security 7(1), IOS Press (1999)Google Scholar
  21. 21.
    Elkan, C.: Results of the KDD 1999 Classifier Learning Contest (1999), http://www.cs.ucsd.edu/users/elkan/clresults.html
  22. 22.
    Portnoy, L.: Intrusion Detection with Unlabeled Data Using Clustering, Undergraduate Thesis, Columbia University (2000)Google Scholar
  23. 23.
    Yamanishi, K., Takeuchi, J., Williams, G.: On-line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms. In: Proc. KDD, pp. 320–324 (2000)Google Scholar
  24. 24.
    Paxson, V.: The Internet Traffic Archive (2002), http://ita.ee.lbl.gov/
  25. 25.
    Forrest, S.: Computer Immune Systems, Data Sets and Software (2002), http://www.cs.unm.edu/~immsec/data-sets.htm
  26. 26.
    McHugh, J.: Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory. In: Proc. ACM TISSEC, vol. 3(4), pp. 262–294 (2000)Google Scholar
  27. 27.
    Hoagland, J.: SPADE, Silicon Defense (2000), http://www.silicondefense.com/software/spice/
  28. 28.
    Ptacek, T.H., Newsham, T.N.: Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection(1998), http://www.robertgraham.com/mirror/Ptacek-Newsham-Evasion-98.html
  29. 29.
    Roesch, M.: Snort - Lightweight Intrusion Detection for Networks. In: Proc. USENIX Lisa 1999 (1999)Google Scholar
  30. 30.
    Mahoney, M.: Source code for PHAD, ALAD, LERAD, NETAD, SAD, EVAL, TF, TM, and AFIL is available at, http://cs.fit.edu/~mmahoney/dist/
  31. 31.
    Adamic, L.A.: Zipf, Power-laws, and Pareto - A Ranking Tutorial (2002), http://ginger.hpl.hp.com/shl/papers/ranking/ranking.html
  32. 32.
    Huberman, B.A., Adamic, L.A.: The Nature of Markets in the World Wide Web (1999), http://ideas.uqam.ca/ideas/data/Papers/scescecf9521.html
  33. 33.
    Mahoney, M.: A Machine Learning Approach to Detecting Attacks by Identifying Anomalies in Network Traffic, Ph.D. dissertation, Florida Institute of Technology (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Matthew V. Mahoney
    • 1
  • Philip K. Chan
    • 1
  1. 1.Computer Science DepartmentFlorida Institute of TechnologyMelbourne

Personalised recommendations