Abstract
Protecting running programs from exploits has been the focus of many host-based intrusion detection systems. To this end various formal methods have been developed that either require manual construction of attack signatures or modelling of normal program behavior to detect exploits. In terms of the ability to discover new attacks before the infection spreads, the former approach has been found to be lacking in flexibility. Consequently, in this paper, we present an anomaly monitoring system, NORT, that verifies on-the-fly whether running programs comply to their expected normal behavior. The model of normal behavior is based on a rich set of discriminators such as minimal infrequent and maximal frequent iterative patterns of system calls, and relative entropy between distributions of system calls. Experiments run on malware samples have shown that our approach is able to effectively detect a broad range of attacks with very low overheads.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Operation aurora (2010), http://en.wikipedia.org/wiki/Operation_Aurora#Response_and_aftermath
Agency, N.S.: Security-enhanced linux (2008), http://www.nsa.gov/selinux
Broder, A., Mitzenmacher, M.: Network applications of bloom filters: A survey. Internet Mathematics (2004)
Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. Tech. rep., University of Wisconsin, Madison (2003)
Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: India Software Engineering Conference. ACM (2008)
Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proc. IEEE S&P (2003)
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proc. IEEE S&P (1996)
Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: Proc. USENIX Security Symposium (2002)
Gopalakrishna, R., Spafford, E.H., Vitek, J.: Efficient intrusion detection using automaton inlining. In: Proc. IEEE S&P (2005)
Kaspersky: Adobe: the number one target for hackers in the first quarter (2010), http://www.kaspersky.com/news?id=207576094
Kumar, S., Spafford, E.: A pattern matching model for misuse intrusion detection. In: Proc. National Computer Security Conference (1994)
Lee, C., Chen, F., Rosu, G.: Mining parametric specifications. In: Proc. ICSE (2011)
Lee, W., Stolfo, S.J.: Data mining approaches for intrusion detection. In: Proc. USENIX Security Symposium (1998)
Lee, W., Xiang, D.: Information-theoretic measures for anomaly detection. In: Proc. IEEE S&P (2001)
Lo, D., Khoo, S.C., Liu, C.: Efficient mining of iterative patterns for software specification discovery. In: Proc. ACM SIGKDD (2007)
Nucci, A., Bannerman, S.: Controlled chaos. IEEE Spectrum (December 2007), http://www.spectrum.ieee.org/dec07/5722
Safyallah, H., Sartipi, K.: Dynamic analysis of software systems using execution pattern mining. In: Proc. IEEE ICPC (2006)
Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proc. IEEE S&P (2001)
Sophos: Top 10 malware (June 2008), http://www.sophos.com/security/top-10/
Szathmary, L., Napoli, A., Valtchev, P.: Towards rare itemset mining. In: Proc. IEEE International Conference on Tools with Artificial Intelligence (2007)
Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proc. S&P (2001)
Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proc. CCS (2002)
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models. In: Proc. IEEE S&P (1999)
Wespi, A., Dacier, M., Debar, H.: Intrusion Detection Using Variable-Length Audit Trail Patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, pp. 110–129. Springer, Heidelberg (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Milea, N.A., Khoo, S.C., Lo, D., Pop, C. (2012). NORT: Runtime Anomaly-Based Monitoring of Malicious Behavior for Windows. In: Khurshid, S., Sen, K. (eds) Runtime Verification. RV 2011. Lecture Notes in Computer Science, vol 7186. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29860-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-29860-8_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29859-2
Online ISBN: 978-3-642-29860-8
eBook Packages: Computer ScienceComputer Science (R0)