International Journal of Information Security

, Volume 16, Issue 5, pp 475–490 | Cite as

Detecting zero-day attacks using context-aware anomaly detection at the application-layer

  • Patrick Duessel
  • Christian Gehl
  • Ulrich Flegel
  • Sven Dietrich
  • Michael Meier
Regular Contribution


Anomaly detection allows for the identification of unknown and novel attacks in network traffic. However, current approaches for anomaly detection of network packet payloads are limited to the analysis of plain byte sequences. Experiments have shown that application-layer attacks become difficult to detect in the presence of attack obfuscation using payload customization. The ability to incorporate syntactic context into anomaly detection provides valuable information and increases detection accuracy. In this contribution, we address the issue of incorporating protocol context into payload-based anomaly detection. We present a new data representation, called \({c}_n\)-grams, that allows to integrate syntactic and sequential features of payloads in an unified feature space and provides the basis for context-aware detection of network intrusions. We conduct experiments on both text-based and binary application-layer protocols which demonstrate superior accuracy on the detection of various types of attacks over regular anomaly detection methods. Furthermore, we show how \({c}_n\)-grams can be used to interpret detected anomalies and thus, provide explainable decisions in practice.


Intrusion detection Machine learning Anomaly detection Protocol analysis Deep packet inspection 


  1. 1.
    Borisov, N., Brumley, D.J., Wang, H., Dunagan, J., Joshi, P., Guo, C.: Generic application-level protocol analyzer and its language. In: Proceedings of Network and Distributed System Security Symposium (NDSS) (2007)Google Scholar
  2. 2.
    Cretu, G., Stavrou, A., Locasto, M., Stolfo, S.J., Keromytis, A.D.: Casting out demons: sanitizing training data for anomaly sensors. In: ieeesp (2008)Google Scholar
  3. 3.
    Cui, W., Kannan, J., Wang. H.J.: Discoverer: automatic protocol reverse engineering from network traces. In: SS’07: Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium, pp. 1–14 (2007)Google Scholar
  4. 4.
    Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack. 11(61) (2003)Google Scholar
  5. 5.
    Düssel, P., Gehl, C., Laskov, P., Rieck, K.: Incorporation of application layer protocol syntax into anomaly detection. In: ICISS, pp. 188–202 (2008)Google Scholar
  6. 6.
    Folga, P., Sharif, M., Perdisci, R., Kolesnikov, O., Lee, W.: Polymorphic blending attacks. In: Proceedings of USENIX Security Symposium, pp. 241–256 (2006)Google Scholar
  7. 7.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 120–128. Oakland (1996)Google Scholar
  8. 8.
    Gao, D., Reiter, M.K., Song, D.: Behavioral distance measurement using hidden markov models. In: Recent Adances in Intrusion Detection (RAID), pp. 19–40 (2006)Google Scholar
  9. 9.
    Ingham, K.L., Somayaji, A., Burge, J., Forrest, S.: Learning DFA representations of HTTP for protecting web applications. Comput. Netw. 51(5), 1239–1255 (2007)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kloft, M., Laskov, P.: Security analysis of online centroid anomaly detection. Technical report UCB/EECS-2010-22. EECS Department, University of California, Berkeley (2010)Google Scholar
  11. 11.
    Kolesnikov, O., Dagon, D., Lee, W.: Advanced polymorphic worms: evading IDS by blending with normal traffic. In: Proceedings of USENIX Security Symposium (2004)Google Scholar
  12. 12.
    Kruegel, C., Vigna, G.: Anomaly detection of web-based attacks. In: Proceedings of 10th ACM Conference on Computer and Communications Security, pp. 251–261 (2003)Google Scholar
  13. 13.
    Kruegel, C., Toth, T., Kirda, E.: Service specific anomaly detection for network intrusion detection. In: Proceedings of ACM Symposium on Applied, Computing, pp. 201–208 (2002)Google Scholar
  14. 14.
    Kruegel, C., Vigna, G., Robertson, W.: A multi-model approach to the detection of web-based attacks. Comput. Netw. 48(5), 717–738 (2005)CrossRefGoogle Scholar
  15. 15.
    Lee, W., Stolfo, S.J.: A framework for constructing features and models for intrusion detection systems. ACM Trans. Inf. Syst. Secur. 3, 227–261 (2000)CrossRefGoogle Scholar
  16. 16.
    Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., Watkins, C.: Text classification using string kernels. J. Mach. Learn. Res. 2, 419–444 (2002)zbMATHGoogle Scholar
  17. 17.
    Mahoney, M.V., Chan, P.K.: PHAD: packet header anomaly detection for identifying hostile network traffic. Technical Report CS-2001-2, Florida Institute of Technology (2001)Google Scholar
  18. 18.
    Mahoney, M.V., Chan, P.K.: Learning nonstationary models of normal network traffic for detecting novel attacks. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 376–385 (2002)Google Scholar
  19. 19.
    Mahoney, M.V., Chan, P.K.: An analysis of the 1999 DARPA/Lincoln Laboratory evaluation data for network anomaly detection. In: Giovanni V, Kruegel, Christopher, Erland J (eds) Recent Advances in Intrusion Detection. Springer, Berlin, Heidelberg, pp. 220–237 (2003)Google Scholar
  20. 20.
    Müller, K.-R., Mika, S., Rätsch, G., Tsuda, K., Schölkopf, B.: An introduction to kernel-based learning algorithms. IEEE Neural Netw. 12(2), 181–201 (2001)CrossRefGoogle Scholar
  21. 21.
    Pang, R., Paxson, V., Sommer, R., Peterson, L.L.: binpac: A yacc for writing application protocol parsers. In: Proceedings of ACM Internet Measurement Conference, pp. 289–300 (2006)Google Scholar
  22. 22.
    Paxson, V.: Bro: a system for detecting network intruders in real-time. In: Proceedings of USENIX Security Symposium, pp. 31–51 (1998)Google Scholar
  23. 23.
    Paxson, V.: The bro 0.8 user manual. Lawrence Berkeley National Laboratory and ICSI Center for Internet Research (2004)Google Scholar
  24. 24.
    Perdisci, R., Ariu, D., Fogla, P., Giacinto, G., Lee, W.: McPAD: a multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 53(6), 864–881 (2009)Google Scholar
  25. 25.
    Rieck, K., Laskov, P.: Detecting unknown network attacks using language models. In: Detection of Intrusions and Malware, and Vulnerability Assessment, Proceedings of 3rd DIMVA Conference, LNCS, pp. 74–90 (2006)Google Scholar
  26. 26.
    Rieck, K., Laskov, P.: Language models for detection of unknown attacks in network traffic. J. Comput. Virol. 2(4), 243–256 (2007)Google Scholar
  27. 27.
    Rieck, K., Laskov, P.: Visualization and explanation of payload-based anomaly detection. In: Proceedings of European Conference on Computer Network Defense (EC2ND) (2009)Google Scholar
  28. 28.
    Roesch, M.: Snort: lightweight intrusion detection for networks. In: Proceedings of USENIX Large Installation System Administration Conference LISA, pp. 229–238 (1999)Google Scholar
  29. 29.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  30. 30.
    Song, Y., Keromytis, A.D., Stolfo, S.J.: Spectrogram: a mixture-of-markov-chains model for anomaly detection in web traffic. In: Proceedings of Network and Distributed System Security Symposium (NDSS) (2009)Google Scholar
  31. 31.
    Tax, D., Duin, R.: Data domain description by support vectors. In: Verleysen, M. (ed.) Proceedings ESANN, pp. 251–256. D. Facto Press, Brussels (1999)Google Scholar
  32. 32.
    Vishwanathan, S.V.N., Smola, A.J.: Fast kernels for string and tree matching. In: Tsuda, K., Schölkopf, B., Vert, J.F. (eds.) Kernels and Bioinformatics, pp. 113–130. MIT Press, Cambridge (2004)Google Scholar
  33. 33.
    Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Recent Adances in Intrusion Detection (RAID), pp. 203–222 (2004)Google Scholar
  34. 34.
    Wang, K., Parekh, J.J., Stolfo, S.J.: Anagram: a content anomaly detector resistant to mimicry attack. In: Recent Adances in Intrusion Detection (RAID), pp. 226–248 (2006)Google Scholar
  35. 35.
    Wireshark. Wireshark: network protocol analyzer. (2010)
  36. 36.
    Wondracek, G., Milani, C.P., Kruegel, C., Kirda, E.: Automatic network protocol analysis. In: 15th Symposium on Network and Distributed System Security (NDSS) (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.University of Bonn, Institute of Computer Science 4BonnGermany
  2. 2.Trifense GmbH - Intelligent Network DefenseVeltenGermany
  3. 3.Infineon Technologies AGNeubibergGermany
  4. 4.CUNY John Jay College of Criminal Justice, Mathematics and Computer Science DepartmentNew YorkUSA

Personalised recommendations