Dynamic Binary Instrumentation-Based Framework for Malware Defense

  • Najwa Aaraj
  • Anand Raghunathan
  • Niraj K. Jha
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5137)


Malware is at the root of a large number of information security breaches. Despite widespread effort devoted to combating malware, current techniques have proven to be insufficient in stemming the incessant growth in malware attacks. In this paper, we describe a tool that exploits a combination of virtualized (isolated) execution environments and dynamic binary instrumentation (DBI) to detect malicious software and prevent its execution. We define two isolated environments: (i) a Testing environment, wherein an untrusted program is traced during execution using DBI and subjected to rigorous checks against extensive security policies that express behavioral patterns of malicious software, and (ii) a Real environment, wherein a program is subjected to run-time monitoring using a behavioral model (in place of the security policies), along with a continuous learning process, in order to prevent non-permissible behavior.

We have evaluated the proposed methodology on both Linux and Windows XP operating systems, using several virus benchmarks as well as obfuscated versions thereof. Experiments demonstrate that our approach achieves almost complete coverage for original and obfuscated viruses. Average execution times go up to 28.57X and 1.23X in the Testing and Real environments, respectively. The high overhead imposed in the Testing environment does not create a severe impediment since it occurs only once and is transparent to the user. Users are only affected by the overhead imposed in the Real environment. We believe that our approach has the potential to improve on the state-of-the-art in malware detection, offering improved accuracy with low performance penalty.


Malware control-data flow execution context dynamic binary instrumentation virtualization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Computer Security Institute, CSI Survey 2007 (2007),
  2. 2.
    Virus Bulletin (2007),
  3. 3.
    Symantec Security Response (2007),
  4. 4.
    The difference between a virus, worm and trojan horse (2004),
  5. 5.
    Szor, P.: The Art of Computer Virus Research and Defense. Addison-Wesley Professional, Reading (2005)Google Scholar
  6. 6.
    Norman SandBox Pro-active virus protection (2004),
  7. 7.
    Gordon, S., Howard, F.: Antivirus software testing for the new millenium. In: Proc. National Information Systems Security Conf., pp. 125–139 (October 2000)Google Scholar
  8. 8.
    Westcoast labs: Checkmark certification (2007),
  9. 9.
    Zhou, Q.: A service-oriented solution framework for distributed virus detection and vulnerability remediation (VDVR) system. In: Proc. Int. Cryptology Conf. Services Computing, pp. 569–573 (July 2007)Google Scholar
  10. 10.
    Shin-Jia, H., Kuang-Hsi, C.: A proxy automatic signature scheme using a compiler in distributed systems for unknown virus detection. In: Proc. Int. Conf. Advanced Information Networking and Applications, pp. 649–654 (March 2005)Google Scholar
  11. 11.
    Yoo, I., Ultes-Nitsche, U.: Adaptive detection of worms/viruses in firewalls. In: Proc. Int. Conf. Security Technology (October 2004)Google Scholar
  12. 12.
    Henchiri, O., Japkowicz, N.: A feature selection and evaluation scheme for computer virus detection. In: Proc. Int. Conf. Data Mining, pp. 891–895 (December 2006)Google Scholar
  13. 13.
    Yin, H., Song, D., Egele, M., Kruegel, C., Kirda, E.: Panorama: Capturing system-wide information flow for malware detection and analysis. In: Proc. ACM Conf. Computer and Communication Security, pp. 116–127 (October 2007)Google Scholar
  14. 14.
    Rozinov, K.: Reverse code engineering: An in-depth analysis of the Bagle virus. In: Proc. Wkshp. Information Assurance and Security, pp. 380–387 (June 2005)Google Scholar
  15. 15.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: Proc. IEEE Symp. Security and Privacy, pp. 32–46 (May 2005)Google Scholar
  16. 16.
    Preda, M.D., Christodorescu, M., Jha, S., Debray, S.: A semantics-based approach to malware detection. In: Proc. Conf. Principles of Programming Languages, pp. 377–388 (January 2007)Google Scholar
  17. 17.
    Moser, A., Kruegel, C., Kirda, E.: Exploring multiple execution paths for malware analysis. In: Proc. IEEE Symp. Security and Privacy, pp. 231–245 (May 2007)Google Scholar
  18. 18.
    Goldberg, I., Wagner, D., Thomas, R., Brewer, E.A.: A secure environment for untrusted helper applications confining the wily hacker. In: Proc. Conf. USENIX Security Symp., pp. 1–13 (July 1996)Google Scholar
  19. 19.
    Peterson, D.S., Bishop, M., Pandey, R.: A flexible containment mechanism for executing untrusted code. In: Proc. Conf. USENIX Security Symp., pp. 207–225 (August 2002)Google Scholar
  20. 20.
    Lam, L.-C., Yu, Y., Chiueh, T.-C.: Secure mobile code execution service. In: Proc. Conf. Large Installation System Administration, pp. 53–62 (December 2006)Google Scholar
  21. 21.
    VMWare Inc., Palo Alto, VMWare browser appliance (2006),
  22. 22.
    Provos, N., Holz, T.: Virtual Honeypots: From Botnet Tracking to Intrusion Detection. Addison-Wesley, Reading (2007)Google Scholar
  23. 23.
    Intel vPro Processor Technology (2007),
  24. 24.
    Aaraj, N., Raghunathan, A., Jha, N.K.: Virtualization-assisted framework for prevention of software vulnerability based security attacks. Tech. Rep. CE-J07-001, Dept. of Electrical Engineering, Princeton University (December 2007)Google Scholar
  25. 25.
    Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: Building customized program analysis tools with dynamic instrumentation. In: Proc. Programming Language Design and Implementation Forum, pp. 190–200 (June 2005)Google Scholar
  26. 26.
    Hangal, S., Lam, M.S.: Tracking down software bugs using automatic anomaly detection. In: Proc. Int. Conf. Software Engineering, pp. 291–301 (May 2002)Google Scholar
  27. 27.
    Ernst, M.D., Cockrell, J., Griswold, W.G., Notkin, D.: Dynamically discovering likely program invariants to support program evolution. In: Proc. Int. Conf. Software Engineering, pp. 213–224 (May 1999)Google Scholar
  28. 28.
    Symantec corporation, Cupertino, The digital immune system (2007),
  29. 29.
    STP: A decision procedure for bitvectors and arrays (2007),
  30. 30.
    Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., Engler, D.R.: EXE: Automatically generating inputs of death. In: Proc. ACM Conf. Computer and Communications Security, pp. 322–335 (November 2006)Google Scholar
  31. 31.
    Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., Yin, H.: Automatically identifying trigger-based behavior in malware (2007),
  32. 32.
    XenSource: Delivering the Power of Xen (2007),
  33. 33.
    VMWare Inc., Palo Alto, Virtual Appliance Marketplace (2007),
  34. 34.
    VX Heavens (2007),
  35. 35.
    Computer Virus Codes (2007),
  36. 36.
  37. 37.
    UPX: the Ultimate Packer for eXecutables (2007),
  38. 38.
    Obfuscator download (2006),

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Najwa Aaraj
    • 1
  • Anand Raghunathan
    • 2
  • Niraj K. Jha
    • 1
  1. 1.Department of Electrical EngineeringPrinceton UniversityPrincetonUSA
  2. 2.NEC Laboratories AmericaPrinceton 

Personalised recommendations