NORT: Runtime Anomaly-Based Monitoring of Malicious Behavior for Windows

  • Narcisa Andreea Milea
  • Siau Cheng Khoo
  • David Lo
  • Cristian Pop
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7186)


Protecting running programs from exploits has been the focus of many host-based intrusion detection systems. To this end various formal methods have been developed that either require manual construction of attack signatures or modelling of normal program behavior to detect exploits. In terms of the ability to discover new attacks before the infection spreads, the former approach has been found to be lacking in flexibility. Consequently, in this paper, we present an anomaly monitoring system, NORT, that verifies on-the-fly whether running programs comply to their expected normal behavior. The model of normal behavior is based on a rich set of discriminators such as minimal infrequent and maximal frequent iterative patterns of system calls, and relative entropy between distributions of system calls. Experiments run on malware samples have shown that our approach is able to effectively detect a broad range of attacks with very low overheads.


Intrusion Detection Frequent Pattern Relative Entropy System Call Runtime Overhead 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Agency, N.S.: Security-enhanced linux (2008),
  3. 3.
    Broder, A., Mitzenmacher, M.: Network applications of bloom filters: A survey. Internet Mathematics (2004)Google Scholar
  4. 4.
    Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. Tech. rep., University of Wisconsin, Madison (2003)Google Scholar
  5. 5.
    Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: India Software Engineering Conference. ACM (2008)Google Scholar
  6. 6.
    Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proc. IEEE S&P (2003)Google Scholar
  7. 7.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proc. IEEE S&P (1996)Google Scholar
  8. 8.
    Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: Proc. USENIX Security Symposium (2002)Google Scholar
  9. 9.
    Gopalakrishna, R., Spafford, E.H., Vitek, J.: Efficient intrusion detection using automaton inlining. In: Proc. IEEE S&P (2005)Google Scholar
  10. 10.
    Kaspersky: Adobe: the number one target for hackers in the first quarter (2010),
  11. 11.
    Kumar, S., Spafford, E.: A pattern matching model for misuse intrusion detection. In: Proc. National Computer Security Conference (1994)Google Scholar
  12. 12.
    Lee, C., Chen, F., Rosu, G.: Mining parametric specifications. In: Proc. ICSE (2011)Google Scholar
  13. 13.
    Lee, W., Stolfo, S.J.: Data mining approaches for intrusion detection. In: Proc. USENIX Security Symposium (1998)Google Scholar
  14. 14.
    Lee, W., Xiang, D.: Information-theoretic measures for anomaly detection. In: Proc. IEEE S&P (2001)Google Scholar
  15. 15.
    Lo, D., Khoo, S.C., Liu, C.: Efficient mining of iterative patterns for software specification discovery. In: Proc. ACM SIGKDD (2007)Google Scholar
  16. 16.
    Nucci, A., Bannerman, S.: Controlled chaos. IEEE Spectrum (December 2007),
  17. 17.
    Safyallah, H., Sartipi, K.: Dynamic analysis of software systems using execution pattern mining. In: Proc. IEEE ICPC (2006)Google Scholar
  18. 18.
    Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proc. IEEE S&P (2001)Google Scholar
  19. 19.
    Sophos: Top 10 malware (June 2008),
  20. 20.
    Szathmary, L., Napoli, A., Valtchev, P.: Towards rare itemset mining. In: Proc. IEEE International Conference on Tools with Artificial Intelligence (2007)Google Scholar
  21. 21.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proc. S&P (2001)Google Scholar
  22. 22.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proc. CCS (2002)Google Scholar
  23. 23.
    Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models. In: Proc. IEEE S&P (1999)Google Scholar
  24. 24.
    Wespi, A., Dacier, M., Debar, H.: Intrusion Detection Using Variable-Length Audit Trail Patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, pp. 110–129. Springer, Heidelberg (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Narcisa Andreea Milea
    • 1
  • Siau Cheng Khoo
    • 1
  • David Lo
    • 2
  • Cristian Pop
    • 3
  1. 1.National University of SingaporeSingapore
  2. 2.Singapore Management UniversitySingapore
  3. 3.MicrosoftRedmondUSA

Personalised recommendations