Nitro: Hardware-Based System Call Tracing for Virtual Machines

  • Jonas Pfoh
  • Christian Schneider
  • Claudia Eckert
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7038)


Virtual machine introspection (VMI) describes the method of monitoring and analyzing the state of a virtual machine from the hypervisor level. This lends itself well to security applications, though the hardware virtualization support from Intel and AMD was not designed with VMI in mind. This results in many challenges for developers of hardware-supported VMI systems. This paper describes the design and implementation of our prototype framework, Nitro, for system call tracing and monitoring. Since Nitro is a purely VMI-based system, it remains isolated from attacks originating within the guest operating system and is not directly visible from within the guest. Nitro is extremely flexible as it supports all three system call mechanisms provided by the Intel x86 architecture and has been proven to work in Windows, Linux, 32-bit, and 64-bit environments. The high performance of our system allows for real-time capturing and dissemination of data without hindering usability. This is supported by extensive testing with various guest operating systems. In addition, Nitro is resistant to circumvention attempts due to a construction called hardware rooting. Finally, Nitro surpasses similar systems in both performance and functionality.


Virtual Machine System Call Page Fault Guest Operating System Virtual Hardware 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bahram, S., Jiang, X., Wang, Z., Grace, M., Li, J., Xu, D.: DKSM: Subverting virtual machine introspection for fun and profit. In: Proc. of 29th IEEE Int. Symp. on Reliable Distributed Systems (SRDS 2010), New Delhi, India (October 2010)Google Scholar
  2. 2.
    Bayer, U., Kruegel, C., Kirda, E.: TTAnalyze: A tool for analyzing malware. In: 15th European Inst. for Computer Antivirus Research (EICAR 2006) Conf., Hamburg, Germany (April 2006)Google Scholar
  3. 3.
    Chen, P.M., Noble, B.D.: When virtual is better than real. In: Proc. of the 8th Workshop on Hot Topics in Op. Sys., p. 133. IEEE, Washington, DC, USA (2001)Google Scholar
  4. 4.
    Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: malware analysis via hardware virtualization extensions. In: Proc. of the 15th ACM Conf. on Computer and Communications Security, pp. 51–62. ACM, New York (2008)Google Scholar
  5. 5.
    Garfinkel, T., Rosenblum, M.: A virtual machine introspection based architecture for intrusion detection. In: Proc. of Network and Distributed Systems Security Symp., pp. 191–206 (2003)Google Scholar
  6. 6.
    Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security 6(3), 151–180 (1998)CrossRefGoogle Scholar
  7. 7.
    Holz, T., Freiling, F., Willems, C.: Toward automated dynamic malware analysis using CWSandbox. IEEE Security & Privacy 5(2), 32–39 (2007)CrossRefGoogle Scholar
  8. 8.
    Jones, S.T., Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H.: VMM-based hidden process detection and identification using Lycosid. In: Proc. of the 4th Int. conf. on Virtual Execution Environments, pp. 91–100. ACM, New York (2008)Google Scholar
  9. 9.
    Kosoresow, A.P., Hofmeyr, S.A.: Intrusion detection via system call traces. IEEE Softw. 14(5), 35–42 (1997)CrossRefGoogle Scholar
  10. 10.
    Martignoni, L., Fattori, A., Paleari, R., Cavallaro, L.: Live and trustworthy forensic analysis of commodity production systems. In: Jha, S., Sommer, R., Kreibich, C. (eds.) RAID 2010. LNCS, vol. 6307, pp. 297–316. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Payne, B.D., Carbone, M., Sharif, M., Lee, W.: Lares: An architecture for secure active monitoring using virtualization. In: Proc. of 2008 IEEE Symp. on Security and Privacy, pp. 233–247. IEEE, Washington, DC, USA (2008)CrossRefGoogle Scholar
  12. 12.
    Pfoh, J., Schneider, C., Eckert, C.: A formal model for virtual machine introspection. In: Proc. of the 2nd ACM Workshop on Virtual Machine Security. ACM, New York (2009)Google Scholar
  13. 13.
    Pfoh, J., Schneider, C., Eckert, C.: Exploiting the x86 architecture to derive virtual machine state information. In: Proc. of the 4th Int. Conf. on Emerging Security Information, Systems and Technologies. IEEE, Venice (2010)Google Scholar
  14. 14.
    Rieck, K., Trinius, P., Willems, C., Holz, T.: Automatic analysis of malware behavior using machine learning. Tech. Rep. 18-2009, Berlin Inst. of Technology (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Jonas Pfoh
    • 1
  • Christian Schneider
    • 1
  • Claudia Eckert
    • 1
  1. 1.Technische Universität MünchenMunichGermany

Personalised recommendations