Abstract
Network intrusion detection is broadly divided into signature and anomaly detection. The former identifies patterns associated with known attacks and the latter attempts to learn a ‘normal’ pattern of activity and alerts when behaviors outside of those norms is detected. The n-gram methodology has arguably been the most successful technique for network anomaly detection. In this work we discover that when training data is sanitized, n-gram anomaly detection is not primarily anomaly detection, as it receives the majority of its performance from an implicit non-anomaly subsystem, that neither uses typical signatures nor is anomaly based (though it is closely related to both). We find that for our data, these “micro-signatures” provide the vast majority of the detection capability. This finding changes how we understand and approach n-gram based ‘anomaly’ detection. By understanding the foundational principles upon which it operates, we can then better explore how to optimally improve it.
The rights of this work are transferred to the extent transferable according to title 17 U.S.C. 105.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Smaha, S.E.: Haystack: an intrusion detection system. In: Aerospace Computer Security Applications Conference (1988)
Denning, D.E.: An intrusion-detection model. IEEE Trans. Softw. Eng. 2, 222–232 (1987)
Vaccaro, H.S., Liepins, G.E.: Detection of anomalous computer session activity. In: IEEE Symposium on Security and Privacy (1989)
Forrest, S., Hofmeyr, S., Somayaji, A.: Computer immunology. Commun. ACM 40(10), 88–96 (1997)
Damashek, D.: Gauging similarity with n-grams: language independent categorization of text. Science 267(5199), 843–848 (1995)
Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004). doi:10.1007/978-3-540-30143-1_11
The Unicode Standard Version 6.0- Core Specification, February 2011. http://www.unicode.org/versions/Unicode6.0.0/ch01.pdf
Wang, K., Parekh, Janak, J., Stolfo, Salvatore, J.: Anagram: a content anomaly detector resistant to mimicry attack. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 226–248. Springer, Heidelberg (2006). doi:10.1007/11856214_12
Hadžiosmanović, D., Simionato, L., Bolzoni, D., Zambon, E., Etalle, S.: N-Gram against the machine: on the feasibility of the N-Gram network analysis for binary protocols. In: Balzarotti, D., Stolfo, Salvatore, J., Cova, M. (eds.) RAID 2012. LNCS, vol. 7462, pp. 354–373. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33338-5_18
Axelsson, S.: The base-rate fallacy and the difficulty of intrusion detection. ACM Trans. Inf. Syst. Secur. 3(3), 186–205 (2000)
Chang, R., Harang, R.E., Payer, G.S.: Extremely lightweight intrusion detection (ELIDe), Army Research Laboratory (2013)
Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: Security and Privacy (2010)
Bolzoni, D., Zambon, E., Etalle, S., Hartel, P.: Poseidon: a 2-tier anomaly-based intrusion detection system, arXiv.preprint.cs/0511043 (2005)
Wressnegger, C., Schwenk, G., Arp, D., Rieck, K.: A close look on n-grams in intrusion detection: anomaly detection vs. classification. In: 2013 ACM workshop on Artificial intelligence and security (2013)
Robertson, W., Vigna, G., Kruegel, C., Kemmerer, R.A.: Using generalization and characterization techniques in the anomaly-based detection of web attacks. In: NDSS (2006)
Guangmin, L.: Modeling unknown web attacks in network anomaly detection. In: Third International Conference on Convergence and Hybrid Information Technology (2008)
Ingham, K.L., Somayaji, A., Burge, J., Forrest, S.: Learning DFA representations of HTTP for protecting web applications. Comput. Netw. 51(5), 1239–1255 (2007)
Görnitz, N., Kloft, M., Rieck, K., Brefeld, U.: Active learning for network intrusion detection. In: Proceedings of the 2nd ACM Workshop on Security and Artificial Intelligence (2009)
Axelsson, S.: Intrusion detection systems: a survey and taxonomy (2000)
Paxson, V.: Bro: a system for detecting network intruders in real-time. Comput. Netw. 31, 2435–2463 (1999)
Roesch, M.: Snort: lightweight intrusion detection for networks. In: LISA (1999)
Rieck, K., Laskov, P.: Detecting unknown network attacks using language models. In: Büschkes, R., Laskov, P. (eds.) Detection of Intrusions and Malware & Vulnerability Assessment. LNCS, pp. 74–90. Springer, Heidelberg (2006)
Rieck, K., Laskov, P., Müller, K.-R.: Efficient algorithms for similarity measures over sequential data: a look beyond kernels. In: Franke, K., Müller, K.-R., Nickolay, B., Schäfer, R. (eds.) DAGM 2006. LNCS, vol. 4174, pp. 374–383. Springer, Heidelberg (2006). doi:10.1007/11861898_38
Cretu-Ciocarlie, G.F., Stavrou, A., Locasto, M.E., Stolfo, S.J.: Adaptive anomaly detection via self-calibration and dynamic updating. In: Kirda, E., Jha, S., Balzarotti, D. (eds.) RAID 2009. LNCS, vol. 5758, pp. 41–60. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04342-0_3
Perdisci, R., Ariu, D., Fogla, P., Giacinto, G., Lee, W.: McPAD: a multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 53(6), 864–881 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Harang, R., Mell, P. (2017). Micro-signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection. In: Cuppens, F., Wang, L., Cuppens-Boulahia, N., Tawbi, N., Garcia-Alfaro, J. (eds) Foundations and Practice of Security. FPS 2016. Lecture Notes in Computer Science(), vol 10128. Springer, Cham. https://doi.org/10.1007/978-3-319-51966-1_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-51966-1_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-51965-4
Online ISBN: 978-3-319-51966-1
eBook Packages: Computer ScienceComputer Science (R0)