Abstract
Many intrusions amplify rights or circumvent defenses by issuing system calls in ways that the original process did not. Defense against these attacks emphasizes preventing attacking code from being introduced to the system and detecting or preventing execution of the injected code. Another approach, where this paper fits in, is to assume that both injection and execution have occurred, and to detect and prevent the executing code from subverting the target system.
We propose a method using waypoints: marks along the normal execution path that a process must follow to successfully access operating system services. Waypoints actively log trustworthy context information as the program executes, allowing our anomaly monitor to both monitor control flow and restrict system call permissions to conform to the legitimate needs of application functions. We describe our design and implementation of waypoints and present results showing that waypoint-based anomaly monitors can detect a subset of mimicry attacks and impossible paths.
This work was supported in part by a Syracuse University Graduate Fellowship Award.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baratloo, A., Tsai, T., Singh, N.: Libsafe: Protecting critical elements of stacks. Technical report, Avaya Labs Research (1999)
Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization. Technical report, CMU department of computer science (2002)
Cowan, C., Barringer, M., Beattie, S., Kroah-Hartman, G., Frantzen, M., Lokier, J.: Format- Guard: Automatic Protection From printf Format String Vulnerabilities. In: Proceedings of the 2001 USENIX Security Symposium, Washington D.C. (2001)
Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: StackGuard: Automatic Adaptive Detection and Prevention of Buffer- Overflow Attacks. In: Proceedings of the 7th USENIX Security Symposium, San Antonio, Texas (1998)
Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly Detection Using Call Stack Information. In: Proceedings of the 2003 IEEE Symposium on Security and Privacy, Berkeley, CA (2003)
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A Sense of Self for Unix Processes. In: Proceedings of the 1996 IEEE Symposium on Security and Privacy (1996)
Purczynski, W.: (kNoX – implementation of non-executable page protection mechanism)
Solar Designer: Non-Executable User Stack, http://www.openwall.com/linux/
Lhee, K., Chapin, S.J.: Type-Assisted Dynamic Buffer Overflow Detection. In: Proceedings of the 11th USENIX Security Symposium, San Francisco (2002)
the Pax team: design & implementation of PaX, http://pageexec.virtualave.net/docs/index.html
Vendicator: StackShield: A “stack smashing” technique protection tool for linux, http://www.angelfire.com/sk/stackshield/
Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the 2001 IEEE Symposium on Security and Privacy (2001)
Xu, J., Kalbarczyk, Z., Iyer, R.K.: Transparent Runtime Randomization for Security. In: Proceedings of the 22nd Symposium on Reliable and Distributed Systems (SRDS), Florence, Italy (2003)
Ghosh, A., Schwartzbard, A.: A study in using neural networks for anomaly and misuse detection. In: 8th USENIX security symposium (1999)
Lee, W., Stolfo, S.: Data mining approaches for intrusion detection. In: 7th USENIX security symposium, San Antonio, TX (1998)
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting Intrusions Using System Calls: Alternative Data Models. In: Proceedings of the 1999 IEEE Symposium on Security and Privacy (1999)
Wespi, A., Dacier, M., Debar, H.: Intrusion detection using variable-length audit trail patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, Springer, Heidelberg (2000)
Abadi, M., Fournet, C.: Access control based on execution history. In: Proceedings of the 2003 Network and Distributed System Security Symposium (2003)
Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference On Computer And Communication Security, Washington, DC, USA (2002)
Aleph One: Smashing The Stack For Fun And Profit, www.Phrack.org 49 (1996)
Nergal: The advanced return-into-lib(c) exploits, www.Phrack.org 58 (2001)
Box, D.: Essential.NET. The Common Language Runtime, vol. I. Addison-Wesley, Reading (2002)
Gong, L., Ellison, G., Dageforde, M.: Inside Java 2 Platform Security: Architecture, API Design, and Implementation, 2nd edn. Addison Wesley, Reading (1999)
Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the IEEE Symposium on Security and Privacy, p. 144. IEEE Computer Society, Los Alamitos (2001)
Kiriansky, V., Bruening, D., Amarasinghe, S.: Secure execution via program shepherding. In: Proceedings of the 11th USENIX Security Symposium, San Francisco, CA (2002)
Bernaschi, M., Gabrielli, E., Mancini, L.V.: Enhancements to the linux kernel for blocking buffer overflow based attacks. In: 4th Linux showcase & conference (2000)
Sekar, R., Venkatakrishnan, V., Basu, S., Bhatkar, S., DuVarney, D.C.: Model-carrying code: a practical approach for safe execution of untrusted applications. In: Proceedings of the nineteenth ACM symposium on Operating systems principles, pp. 15–28. ACM Press, New York (2003)
Somayaji, A., Hofmeyr, S., Forrest, S.: Principles of a Computer Immune System. In: Proceedings of the 1997 New Security Paradigms Workshop, UK (1997)
Red Hat security: Updated kon2 packages fix buffer overflow (2003)
Ashcraft, K., Engler, D.R.: Using programmer-written compiler extensions to catch security holes. In: Proceedings of the 2002 IEEE Symposium on Security and Privacy, Oakland, CA (2002)
Necula, G.C.: Proof-carrying code. In: Proceedings of the 24th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Langauges (POPL 1997), Paris, pp. 106–119 (1997)
Lhee, K., Chapin, S.J.: Buffer Overflow and Format String Overflow Vulnerabilities. Software – Practice & Experience 33, 423–460 (2003)
Cowan, C., Beattie, S., Johansen, J., Wagle, P.: Pointguard: Protecting pointers from buffer overflow vulnerabilities. In: Proceedings of the 12th USENIX Security Symposium (2003)
Barrantes, E.G., Ackley, D.H., Forrest, S., Palmer, T.S., Stefanovic, D., Zovi, D.D.: Randomized instruction set emulation to disrupt binary code injection attacks. In: Proceedings of the 10th ACM Conference On Computer And Communication Security (2003)
Kc, G.S., Keromytis, A.D., Prevelakis, V.: Countering Code-Injection Attacks With Instruction-Set Randomization. In: Proceedings of the 10th ACM Conference On Computer And Communication Security (2003)
Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: An efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, Washington D.C. (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Xu, H., Du, W., Chapin, S.J. (2004). Context Sensitive Anomaly Monitoring of Process Control Flow to Detect Mimicry Attacks and Impossible Paths. In: Jonsson, E., Valdes, A., Almgren, M. (eds) Recent Advances in Intrusion Detection. RAID 2004. Lecture Notes in Computer Science, vol 3224. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30143-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-30143-1_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23123-3
Online ISBN: 978-3-540-30143-1
eBook Packages: Springer Book Archive