Micro-signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection

  • Richard HarangEmail author
  • Peter Mell
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10128)


Network intrusion detection is broadly divided into signature and anomaly detection. The former identifies patterns associated with known attacks and the latter attempts to learn a ‘normal’ pattern of activity and alerts when behaviors outside of those norms is detected. The n-gram methodology has arguably been the most successful technique for network anomaly detection. In this work we discover that when training data is sanitized, n-gram anomaly detection is not primarily anomaly detection, as it receives the majority of its performance from an implicit non-anomaly subsystem, that neither uses typical signatures nor is anomaly based (though it is closely related to both). We find that for our data, these “micro-signatures” provide the vast majority of the detection capability. This finding changes how we understand and approach n-gram based ‘anomaly’ detection. By understanding the foundational principles upon which it operates, we can then better explore how to optimally improve it.


Network intrusion detection Anomaly detection Microsignatures 


  1. 1.
    Smaha, S.E.: Haystack: an intrusion detection system. In: Aerospace Computer Security Applications Conference (1988)Google Scholar
  2. 2.
    Denning, D.E.: An intrusion-detection model. IEEE Trans. Softw. Eng. 2, 222–232 (1987)CrossRefGoogle Scholar
  3. 3.
    Vaccaro, H.S., Liepins, G.E.: Detection of anomalous computer session activity. In: IEEE Symposium on Security and Privacy (1989)Google Scholar
  4. 4.
    Forrest, S., Hofmeyr, S., Somayaji, A.: Computer immunology. Commun. ACM 40(10), 88–96 (1997)CrossRefGoogle Scholar
  5. 5.
    Damashek, D.: Gauging similarity with n-grams: language independent categorization of text. Science 267(5199), 843–848 (1995)CrossRefGoogle Scholar
  6. 6.
    Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-30143-1_11 CrossRefGoogle Scholar
  7. 7.
    The Unicode Standard Version 6.0- Core Specification, February 2011.
  8. 8.
    Wang, K., Parekh, Janak, J., Stolfo, Salvatore, J.: Anagram: a content anomaly detector resistant to mimicry attack. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 226–248. Springer, Heidelberg (2006). doi: 10.1007/11856214_12 CrossRefGoogle Scholar
  9. 9.
    Hadžiosmanović, D., Simionato, L., Bolzoni, D., Zambon, E., Etalle, S.: N-Gram against the machine: on the feasibility of the N-Gram network analysis for binary protocols. In: Balzarotti, D., Stolfo, Salvatore, J., Cova, M. (eds.) RAID 2012. LNCS, vol. 7462, pp. 354–373. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33338-5_18 CrossRefGoogle Scholar
  10. 10.
    Axelsson, S.: The base-rate fallacy and the difficulty of intrusion detection. ACM Trans. Inf. Syst. Secur. 3(3), 186–205 (2000)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Chang, R., Harang, R.E., Payer, G.S.: Extremely lightweight intrusion detection (ELIDe), Army Research Laboratory (2013)Google Scholar
  12. 12.
    Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: Security and Privacy (2010)Google Scholar
  13. 13.
    Bolzoni, D., Zambon, E., Etalle, S., Hartel, P.: Poseidon: a 2-tier anomaly-based intrusion detection system, arXiv.preprint.cs/0511043 (2005)
  14. 14.
    Wressnegger, C., Schwenk, G., Arp, D., Rieck, K.: A close look on n-grams in intrusion detection: anomaly detection vs. classification. In: 2013 ACM workshop on Artificial intelligence and security (2013)Google Scholar
  15. 15.
    Robertson, W., Vigna, G., Kruegel, C., Kemmerer, R.A.: Using generalization and characterization techniques in the anomaly-based detection of web attacks. In: NDSS (2006)Google Scholar
  16. 16.
    Guangmin, L.: Modeling unknown web attacks in network anomaly detection. In: Third International Conference on Convergence and Hybrid Information Technology (2008)Google Scholar
  17. 17.
    Ingham, K.L., Somayaji, A., Burge, J., Forrest, S.: Learning DFA representations of HTTP for protecting web applications. Comput. Netw. 51(5), 1239–1255 (2007)CrossRefzbMATHGoogle Scholar
  18. 18.
    Görnitz, N., Kloft, M., Rieck, K., Brefeld, U.: Active learning for network intrusion detection. In: Proceedings of the 2nd ACM Workshop on Security and Artificial Intelligence (2009)Google Scholar
  19. 19.
    Axelsson, S.: Intrusion detection systems: a survey and taxonomy (2000)Google Scholar
  20. 20.
    Paxson, V.: Bro: a system for detecting network intruders in real-time. Comput. Netw. 31, 2435–2463 (1999)CrossRefGoogle Scholar
  21. 21.
    Roesch, M.: Snort: lightweight intrusion detection for networks. In: LISA (1999)Google Scholar
  22. 22.
    Rieck, K., Laskov, P.: Detecting unknown network attacks using language models. In: Büschkes, R., Laskov, P. (eds.) Detection of Intrusions and Malware & Vulnerability Assessment. LNCS, pp. 74–90. Springer, Heidelberg (2006)Google Scholar
  23. 23.
    Rieck, K., Laskov, P., Müller, K.-R.: Efficient algorithms for similarity measures over sequential data: a look beyond kernels. In: Franke, K., Müller, K.-R., Nickolay, B., Schäfer, R. (eds.) DAGM 2006. LNCS, vol. 4174, pp. 374–383. Springer, Heidelberg (2006). doi: 10.1007/11861898_38 CrossRefGoogle Scholar
  24. 24.
    Cretu-Ciocarlie, G.F., Stavrou, A., Locasto, M.E., Stolfo, S.J.: Adaptive anomaly detection via self-calibration and dynamic updating. In: Kirda, E., Jha, S., Balzarotti, D. (eds.) RAID 2009. LNCS, vol. 5758, pp. 41–60. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-04342-0_3 CrossRefGoogle Scholar
  25. 25.
    Perdisci, R., Ariu, D., Fogla, P., Giacinto, G., Lee, W.: McPAD: a multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 53(6), 864–881 (2009)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.United States Army Research LaboratoryAdelphiUSA
  2. 2.National Institute of Standards and TechnologyGaithersburgUSA

Personalised recommendations