Operating System Interface Obfuscation and the Revealing of Hidden Operations

  • Abhinav Srivastava
  • Andrea Lanzi
  • Jonathon Giffin
  • Davide Balzarotti
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6739)


Many software security solutions—including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors—rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel’s system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.


Virtual Machine System Call Execution Engine Kernel Code Guest Operating System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity: Principles, implementations, and applications. In: 12th ACM Conference on Computer and Communications Security, CCS (2005)Google Scholar
  2. 2.
    Baliga, A., Kamat, P., Iftode, L.: Lurking in the shadows: Identifying systemic threats to kernel data. In: IEEE Symposium on Security and Privacy (May 2007)Google Scholar
  3. 3.
    Blorge. Faulty drivers bypass Vistas kernel protection, (last accessed 15 Jan 2011)
  4. 4.
    Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization. In: Technical Report CMU-CS-02-197, Carnegie Mellon University, Pittsburg (December 2002)Google Scholar
  5. 5.
    David, F., Chan, E., Carlyle, J., Campbell, R.: Cloaker: hardware supported rootkit concealment. In: IEEE Symposium on Security and Privacy, Oakland, CA (May 2008)Google Scholar
  6. 6.
    Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: Malware analysis via hardware virtualization extensions. In: 15th ACM Conference on Computer and Communications Security, CCS (October 2008)Google Scholar
  7. 7.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for UNIX processes. In: IEEE Symposium on Security and Privacy (May 1996)Google Scholar
  8. 8.
    Ganapathy, V., Jaeger, T., Jha, S.: Automatic placement of authorization hooks in the Linux security modules framework. In: 12th ACM Conference on Computer and Communications Security (CCS), Alexandria, Virginia (November 2005)Google Scholar
  9. 9.
    Giffin, J.T., Jha, S., Miller, B.P.: Efficient context-sensitive intrusion detection. In: Network and Distributed System Security Symposium (NDSS), San Diego, CA (February 2004)Google Scholar
  10. 10.
    Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security 6(3), 151–180 (1998)CrossRefGoogle Scholar
  11. 11.
    Jiang, X., Wang, X.: “Out-of-the-box” monitoring of VM-based high-interaction honeypots. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 198–218. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  12. 12.
    Jones, S.T., Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H.: Antfarm: Tracking processes in a virtual machine environment. In: USENIX Annual Technical Conference (June 2006)Google Scholar
  13. 13.
    Kasslin, K.: Kernel malware: The attack from within. (last accessed January 15, 2011)
  14. 14.
    Krohn, M., Yip, A., Brodsky, M., Cliffer, N., Kaashoek, M.F., Kohler, E., Morris, R.: Information flow control for standard OS abstractions. In: Symposium on Operating System Principles, SOSP (October 2007)Google Scholar
  15. 15.
    Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: USENIX Security Symposium, Baltimore, MD (August 2005)Google Scholar
  16. 16.
    Last, J. V.: Stuxnet versus the iranian nuclear program. (last accessed January 15, 2011)
  17. 17.
    Linn, C.M., Rajagopalan, M., Baker, S., Collberg, C., Debray, S.K., Hartman, J.H.: Protecting against unexpected system calls. In: 14th USENIX Security Symposium (August 2005)Google Scholar
  18. 18.
    Martignoni, L., Christodorescu, M., Jha, S.: OmniUnpack: Fast, generic, and safe unpacking of malware. In: Annual Computer Security Applications Conference, ACSAC, Miami, FL (December 2007)Google Scholar
  19. 19.
    Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 78–97. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Mavinakayanahalli, A., Panchamukhi, P., Keniston, J., Keshavamurthy, A., Hiramatsu, M.: Probing the guts of kprobes. In: Linux Symposium (July 2006)Google Scholar
  21. 21.
    McAfee Security. System call interception, (last accessed January 15, 2011)
  22. 22.
    Mutz, D., Robertson, W., Vigna, G., Kemmerer, R.A.: Exploiting execution context for the detection of anomalous system calls. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 1–20. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  23. 23.
    Onoue, K., Oyama, Y., Yonezawa, A.: Control of system calls from outside of virtual machines. In: ACM Symposium on Applied Computing (March 2008)Google Scholar
  24. 24.
    packetstormsecurity. Adore rootkit, (last accessed January 15, 2011)
  25. 25.
    packetstormsecurity. Knark rootkit, (last accessed January 15, 2011)
  26. 26.
    Payne, B.D., Carbone, M., Sharif, M., Lee, W.: Lares: An architecture for secure active monitoring using virtualization. In: IEEE Symposium on Security and Privacy (May 2008)Google Scholar
  27. 27.
    PCNews. Verisign working to mitigate stuxnet digital signature theft, (last accessed January 15, 2011)
  28. 28.
    Petroni Jr., N.L., Fraser, T., Walters, A., Arbaugh, W.A.: An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In: 15th USENIX Security Symposium (August 2006)Google Scholar
  29. 29.
    Petroni Jr., N.L., Hicks, M.: Automated detection of persistent kernel control-flow attacks. In: ACM Conference on Computer and Communications Security, CCS (November 2007)Google Scholar
  30. 30.
    Provos, N.: Improving host security with system call policies. In: 12th USENIX Security Symposium (August 2003)Google Scholar
  31. 31.
    Riley, R., Jiang, X., Xu, D.: Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 1–20. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  32. 32., (last accessed January 15, 2011)
  33. 33.
    Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: IEEE Symposium on Security and Privacy (May 2001)Google Scholar
  34. 34.
    Seshadri, A., Luk, M., Qu, N., Perrig, A.: SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In: ACM Symposium on Operating Systems Principles, SOSP (October 2007)Google Scholar
  35. 35.
    Sharif, M., Singh, K., Giffin, J.T., Lee, W.: Understanding precision in host based intrusion detection. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 21–41. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  36. 36.
    Shinagawa, T., Eiraku, H., Tanimoto, K., Omote, K., Hasegawa, S., Horie, T., Hirano, M., Kourai, K., Oyama, Y., Kawai, E., Kono, K., Chiba, S., Shinjo, Y., Kato, K.: BitVisor: A thin hypervisor for enforcing I/O device security. In: ACM VEE, Washington, DC (March 2009)Google Scholar
  37. 37.
    Some Observations on Rootkits. Microsoft Malware Protection Center, (last accessed January 15, 2011)
  38. 38.
    Srivastava, A., Giffin, J.: Automatic discovery of parasitic malware. In: Jha, S., Sommer, R., Kreibich, C. (eds.) RAID 2010. LNCS, vol. 6307, pp. 97–117. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  39. 39.
    Srivastava, A., Giffin, J.: Efficient monitoring of untrusted kernel-mode execution. In: NDSS, San Diego, California (February 2011)Google Scholar
  40. 40.
    Sun Microsystem. Dtrace, (last accessed January 15, 2011)
  41. 41.
    Tan, K.M.C., Killourhy, K.S., Maxion, R.A.: Undermining an anomaly-based intrusion detection system using common exploits. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 54. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  42. 42.
    Tan, L., Zhang, X., Ma, X., Xiong, W., Zhou, Y.: AutoISES: Automatically inferring security specifications and detecting violations. In: USENIX Security Symposium (August 2008)Google Scholar
  43. 43.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: ACM CCS (November 2002)Google Scholar
  44. 44.
    Wang, Z., Jiang, X., Cui, W., Ning, P.: Countering kernel rootkits with lightweight hook protection. In: ACM CCS, Chicago, IL (November 2009)Google Scholar
  45. 45.
    Xu, H., Du, W., Chapin, S.J.: Context sensitive anomaly monitoring of process control flow to detect mimicry attacks and impossible paths. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 21–38. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  46. 46.
    Xu, M., Jiang, X., Sandhu, R., Zhang, X.: Towards a VMM-based usage control framework for OS kernel integrity protection. In: ACM SACMAT (June 2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Abhinav Srivastava
    • 1
  • Andrea Lanzi
    • 2
  • Jonathon Giffin
    • 1
  • Davide Balzarotti
    • 2
  1. 1.School of Computer ScienceGeorgia Institute of TechnologyUSA
  2. 2.Institute EurecomFrance

Personalised recommendations