Abstract
Much of the intrusion detection research focuses on signature (misuse) detection, where models are built to recognize known attacks. However, signature detection, by its nature, cannot detect novel attacks. Anomaly detection focuses on modeling the normal behavior and identifying significant deviations, which could be novel attacks. In this chapter we explore two machine learning methods that can construct anomaly detection models from past behavior. The first method is a rule learning algorithm that characterizes normal behavior in the absence of labeled attack data. The second method uses a clustering algorithm to identify outliers.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
C. Aggarwal and P. Yu. Outlier detection for high dimensional data. In Proc. SIGMOD, 2001.
R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of items in large databases. In Proc. ACM SIGMOD Conf., pages 207–216, 1993.
F. Apap, A. Honig, S. Hershkop, E. Eskin, and S. Stolfo. Detecting malicious software by monitoring anomalous windows registry accesses. In Proc. Fifth Intl. Symp. Recent Advances in Intrusion Detection (RAID), 2002.
D. Barbara, N. Wu, and S. Jajodia. Detecting novel network intrusions using bayes estimators. In Proc. SIAM Intl. Conf. Data Mining, 2001.
M. Breunig, H. Kriegel, R. Ng, and J. Sander. Lof: Identifying density-based local outliers. In Proc. SIGMOD, 2000.
P. Clark and T. Niblett. The CN2 induction algorithm. Machine Learning, 3:261–285, 1989.
Silicon Defense. SPADE, 2001. http://www.silicondefense.com/software/spice/.
P. Domingos and M. Pazzani. On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning, 29:103–130, 1997.
R. Duda and P. Hart. Pattern classification and scene analysis. Wiley, New York, NY, 1973.
E. Eskin, A. Arnold, M. Prerau, L. Portnoy, and S. Stolfo. A geometric framework for unsupervised anomaly detection: Detecting intrusions in unlabeled data. In D. Barbara and S. Jajodia, editors, Applications of Data Mining in Computer Security. Kluwer, 2002.
S. Forrest, S. Hofmeyr, and A. Somayaji. Computer immunology. Comm. ACM, 4(10):88–96, 1997.
S. Forrest, S. Hofmeyr, A. Somayaji, and T. Longstaff. A sense of self for unix processes. In Proc. of 1996 IEEE Symp. on Computer Security and Privacy, 1996.
A. Ghosh, A. Schwartzbard, and M. Schatz. Learning program behavior profiles for in-trusion detection. In Proc. 1st USENIX Workshop on Intrusion Detection and Network Monitoring, 1999.
J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann, 2000.
K. Kendall. A database of computer attacks for the evaluation of intrusion detection systems. Master’s thesis, EECS Dept., MIT, 1999.
E. Knorr and T. Ng. Algorithms for mining distance-based outliers in large datasets. In Proc. VLDB, 1998.
C. Krugel, T. Toth, and E. Kirda. Service specific anomaly detection for network intrusion detection. In Proc. ACM Symp. on Applied Computing, 2002.
T. Lane and C. Brodley. Temporal sequence learning and data reduction for anomaly detection. ACM Trans. Information and System Security, 1999.
R. Lippmann, J. Haines, D. Fried, J. Korba, and K. Das. The 1999 DARPA off-line intrusion detection evaluation. Computer Networks, 34:579–595, 2000.
M. Mahoney and P. Chan. Learning models of network traffic for detecting novel attacks. Technical Report CS-2002-08, Florida Inst. of Tech., Melbourne, FL, 2002. http://www.cs.fit.edu/~pkc/papers/cs-2002-08.pdf.
M. Mahoney and P. Chan. Learning nonstationary models of normal network traffic for detecting novel attacks. In Proc. Eighth Intl. Conf. on Knowledge Discovery and Data Mining, pages 376–385, 2002.
M. Mahoney and P. Chan. Learning Rules for Anomaly Detection of Hostile Network Traffic. Technical Report CS-2003-16, Florida Inst. of Tech., Melbourne, FL, 2003. http://www.cs.fit.edu/~pkc/papers/cs-2003-16.pdf.
T. Mitchell. Machine Learning. McGraw Hill, 1997.
P. Neumann and P. Porras. Experience with EMERALD to date. In Proc. 1st USENIX Workshop on Intrusion Detection and Network Monitoring, pages 73–80, 1999.
T. Niblett. Constructing decision trees in noisy domain. In Proc. 2nd European Working Session on Learning, pages 67–78, 1987.
V. Paxson. Bro: A system for detecting network intruders in real-time. In Proc. 7th USENIX Security Symp., 1998.
V. Paxson and S. Floyd. The failure of poisson modeling. IEEE/ACM Transactions on Networking, 3:226–24, 1995.
J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1987.
L. Portnoy. Intrusion detection with unlabeled data using clustering. Undergraduate Thesis, Columbia University, 2000.
F. Provost and P. Domingos. Tree induction for probability-based rankings. Machine Learning, 2002.
S. Ramaswamy, R. Rastogi, and K. Shim. Efficient algorithms for mining outliers from large data sets. In Proc. SIGMOD, 2000.
M. Roesch. Snort — lightweight intrusion detection for networks. In USENIX LISA, 1999.
S. Salvador and P. Chan. Learning states and rules for time-series anomaly detection. Technical Report CS-2003-05, Florida Inst. of Tech., Melbourne, FL, 2003. http://www.cs.fit.edu/~pkc/papers/cs-2003-05.pdf.
R. Sekar, M. Bendre, D. Dhurjati, and P. Bollinen. A fast automaton-based method for detecting anomalous program behaviors. In Proc. IEEE Symp. Security and Privacy, 2001.
K. Sequira and M. Zaki. ADMIT: Anomaly-based data mining for intrusions. In Proc. KDD, 2002.
S. Staniford, J. Hoagland, and J. McAlerney. Practical automated detection of stealthy portscans. J. Computer Security, 2002.
A. Valdes and K. Skinner. Adaptive model-based monitoring for cyber attack detection. In Proc. RAID, pages 80–92, 2000.
I. Witten and T. Bell. The zero-frequency problem: estimating the probabilities of novel events in adaptive text compression. IEEE Trans. on Information Theory, 37(4): 1085–1094, 1991.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer Science+Business Media, Inc.
About this chapter
Cite this chapter
Chan, P.K., Mahoney, M.V., Arshad, M.H. (2005). Learning Rules and Clusters for Anomaly Detection in Network Traffic. In: Kumar, V., Srivastava, J., Lazarevic, A. (eds) Managing Cyber Threats. Massive Computing, vol 5. Springer, Boston, MA. https://doi.org/10.1007/0-387-24230-9_3
Download citation
DOI: https://doi.org/10.1007/0-387-24230-9_3
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-24226-2
Online ISBN: 978-0-387-24230-9
eBook Packages: Computer ScienceComputer Science (R0)