The 1998 Lincoln Laboratory IDS Evaluation
In 1998 (and again in 1999), the Lincoln Laboratory of MIT conducted a comparative evaluation of Intrusion Detection Systems developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of unresolved issues associated with its design and execution. Some of methodologies used in the evaluation are questionable and may have biased its results. One of the problems with the evaluation is that the evaluators have published relatively little concerning some of the more critical aspects of their work, such as validation of their test data. The purpose of this paper is to attempt to identify the shortcomings of the Lincoln Lab effort in the hope that future efforts of this kind will be placed on a sounder footing. Some of the problems that the paper points out might well be resolved if the evaluators publish a detailed description of their procedures and the rationale that led to their adoption, but other problems clearly remain.
KeywordsEvaluation IDS ROC Analysis
Unable to display preview. Download preview PDF.
- Stefan Axelsson. The base-rate fallacy and its implications for the difficulty of intrusion detection. In 6th ACM Conference on Computer and Communications Security, pages 1–7, 1999.Google Scholar
- Steven M. Bellovin. Packets found on an internet. Computer Communications Review, 23(3):26–31, July 1993.Google Scholar
- James P. Egan. Signal detection Theory and ROC Analysis. Academic Press, 1975.Google Scholar
- Isaac Graf et al. Results of DARPA 1998 offline intrusion detection evaluation. Presentation at MIT Lincoln Laboratory PI Meeting (available at) http://ideval.ll.mit.edu/results-html-dir/, 15 December 1998.
- D. A. James and S. J. Young. A fast lattice-based approach to vocabulary independent wordspotting. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 337–380, 1994.Google Scholar
- Kristopher Kendall. A database of computer attacks for the evaluation of intrusion detection systems. BS/MS thesis, Massachusetts Institute of Technology, June 1999.Google Scholar
- Richard P. Lippmann, Eric I. Chang, and Charles R. Jankowski. Wordspotter training using figure-of-merit back propagation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 385–388, 1994.Google Scholar
- Richard P. Lippmann et al. MIT Lincoln Laboratory offline component of DARPA 1998 intrusion detection evaluation. Presentation at MIT Lincoln Laboratory PI Meeting (available at) http://ideval.ll.mit.edu/intro-html-dir/, 14 December 1998.
- Richard P. Lippmann et al. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In DISCEX 2000. IEEE Computer Society Press, January 2000.Google Scholar
- Alvin Martin. Personal communications, January 2000.Google Scholar
- Stephen L. Moshier. Personal communications, January 2000.Google Scholar
- Vern Paxson. Bro: A system for detecting network intruders in real-time. Computer Networks, 31(23–24):2435–2463, December 1999.Google Scholar
- Stacy J. Prowell, Carmen J. Trammell, Richard C. Linger, and Jesse H. Poore. Cleanroom Software Engineering: Technology and Process. Addison-Wesley, Reading, Mass., 1998.Google Scholar
- John A. Swets. Measuring the accuracy of diagnostic systems. Science, 24(48):1285–1293, 3 June 1988.Google Scholar
- Daniel Weber. A taxonomy of computer intrusions. MS thesis, Massachusetts Institute of Technology, 1998.Google Scholar
- Q. E. Whiting-O’Keefe, Curtis Henke, and Donald W. Simborg. Choosing the correct unit of analysis in medical care experiments. Medical Care, 22(12):1101–1114, December 1984.Google Scholar