Journal in Computer Virology

, Volume 3, Issue 3, pp 221–236 | Cite as

Secure and advanced unpacking using computer emulation

  • Sébastien JosseEmail author
Original Paper


The purpose of this article is firstly to present a secure unpacker which is specifically designed for a security analyst when studying viruses but also any anti-virus scanner. Such a tool is in fact required when assessing security requirements of an anti-virus scanner through a black box approach. During testing of anti-virus software, a security analyst needs to build virus populations required for several penetration tests. Virus unpacking is a first mandatory step before gaining the ability to apply obfuscation transformation or any information extraction algorithm on a viral set. A secure unpacker is also useful when checking security robustness against reverse engineering of any packed or protected security product. Several static and dynamic analysis tools already implement unpacking algorithms, but these often require human intervention and are not well designed to automatically unpack such a dangerous program as a virus. A new algorithm for automatically unpacking encrypted viruses is presented in this paper. Forensics techniques to reconstruct an unpacked executable and advanced heuristics are also presented in order to decrypt more sophisticated self-protected Malwares. We present several detection techniques which are specifically designed to deceive virtual machine monitors and discuss the security of our tool against these low-level viral attacks. Our secure unpacker figures among a set of several tools. We then present in this paper a proof-of-concept human analysis framework which implements most standard components of an anti-virus scanner (real-time scanner, emulator engine) and in addition proposes a reliable system for automatically gaining information about a virus and its interaction with the OS executive (stealth native API hooking), but focuses on human decision as a detection process without the same resource limitation constraint as product oriented anti-virus scanners. This framework is used as a basis/reference for the comparative analysis of security aspects of anti-virus scanners and deals with the robustness of their driver stack and the efficiency of their de-obfuscation and unpacking algorithms.


Malware analysis Anti-virus testing Forensics Software protection Fault injection Human driven analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aho, A.V., Corasik, M.J.: Efficient string matching: an aid to bibliographic search. Commun. ACM 18(6), (1975)Google Scholar
  2. 2.
  3. 3.
    Argos Howto: Howto: setting up Argos the 0day shellcode catcher. Retrieved from (2006)
  4. 4. project: Retrieved from (2006)
  5. 5.
    Butler, J.: DKOM (Direct Kernel Object Manipulation, slides). Retrieved from:, (2006)
  6. 6.
    Bayer, U., Kruegel, C., Kirda, E.: TTAnalyze: A tool for analyzing malware. In: proceedings of the 15th EICAR Conference, Hamburg, Germany, 29 April–3 May 2006. In: Broucek, V. et al. (ed.) J. Comput. Virol., EICAR 2006 Special Issue, 2006 (2005)Google Scholar
  7. 7.
    Bos, H.: A personal view on the future of Zero-day Worm Containment (slides) (2006)Google Scholar
  8. 8.
    Bellard, F.: QEMU, a Fast and Portable Dynamic Translator. In: Proceedings of the 2005 USENIX Conference (2005)Google Scholar
  9. 9.
    Betz, C.: MemParser tool. Retrieved from: (2006)
  10. 10.
    Beaucamp, P., Filiol, E.: On the possibility of practically obfuscating programs: towards a unified perspective of code protection. In: Proceedings of the First International Workshop in Theoretical Virology 2006, Nancy, May 2007, In: Bonfante, G., Marion, J.-Y., (eds.) WTCV’06 Special Issue, J. Comput. Virol. 3(1) 2007 (2006)Google Scholar
  11. 11.
    Brosch, T., Morgenstern, M.: Runtime packers: the hidden problem. Black Hat 2006 Conference (2006)Google Scholar
  12. 12.
    Bochs: Bochs, the open source IA-32 emulation project. Available at: (2007)
  13. 13.
    Brulez, N.: Anti Reverse Engineering Uncovered. Code Breakers Journal. Previously published at the Honeynet Project, Scan of the Month 33 (2005)
  14. 14.
    Burdach, M.: An Introduction to Windows memory forensic. Retrieved from:, September 2006 (2005)
  15. 15.
    Burdach, M.: Digital forensics of the physical memory. Retrieved from:, September 2006 (2005)
  16. 16.
    Burdach, M.: idetect, ProcEnum, WMFT tools. Retrieved from:, September 2006 (2005)
  17. 17.
    Burdach, M.: Digital Investigation. Retrieved from: (2006)
  18. 18.
    Burdach, M.: Finding Digital Evidence In Physical Memory (slides). Retrieved from: (2006)
  19. 19.
    Butler, J., Hoglund, G.: Rootkits: Subverting the Windows Kernel. Addison Wesley, ISBN 0-321-29431-9 (2006)Google Scholar
  20. 20.
    Bos, H., Portokalidis, G., Slowinska, A.: Argos: An Emulator for Fingerprinting Zero-Day Attacks. In: Proceedings EuroSys (2006)Google Scholar
  21. 21.
    Cohen, F.: Computer viruses, Ph.D. thesis, University of Southern California (1986)Google Scholar
  22. 22.
    Carvey, H.: Reassembling an image file from a memory dump. Retrieved from: (2006)
  23. 23.
    Carvey, H.: Ramdump, lsproc, lspm, ReadPE tools. Retrieved from: (2006)
  24. 24.
    Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., Veith, H.: Malware Normalization, Technical Report, University of Wisconsin, Madison, USA (2005)Google Scholar
  25. 25.
    Clam AntiVirus: Available at: (2007)
  26. 26.
    Cloakware: Retrieved from: (2007)
  27. 27.
    Cogswell, C., Russinovich, M.: RootkitRevealer. Available at: (2006)
  28. 28.
    DataRescue: Using the Universal PE Unpacker Plug-in included in IDA Pro 4.9 to unpack compressed executables. Retrieved from:, September 2006 (2005)
  29. 29.
    DataRescue: Using the IDA debugger to unpack an hostile PE executable. Retrieved from: (2006)
  30. 30.
    Elias: Detect if your program is running inside a Virtual Machine. 14 Mars 2005. Retrieved from: (Elias homepage), September 2006 (2005)
  31. 31.
    Filiol, F.: Strong cryptography armoured computer viruses forbidding code analysis: the Bradley virus. In: Proceedings of the 14th EICAR Conference, pp. 210–214 (2005)Google Scholar
  32. 32.
    Filiol E.: Techniques virales avancées, IRIS Series, Springer Verlag France, January 2007. An English translation is pending (due mid 2007) (2007)Google Scholar
  33. 33.
    Filiol, F., Josse, S.: A statistical model for undecidable viral detection. In: Proceedings of the 16th EICAR Conference, Budapest, Hungary, 5–8 May 2007. To appear in: Broucek, V. (ed.) Eicar 2007 Special Issue, J. Comput. Virol. 3(2) (2007)Google Scholar
  34. 34.
    Ferrie, P.: Attacks on virtual machine emulators. In: Proceedings of the 2006 AVAR Conference, Auckland, NZ (2006)Google Scholar
  35. 35.
    Garner, G.M.: Forensic Acquisition Utilities: Dd, md5lib, md5sum, VolumeDump, Wipe, ZlibU, nc, GetOpt. Retrieved from:, (2006)
  36. 36.
    Garner, G.M., Mora, R.: Kntlist tool. Retrieved from: (2006)
  37. 37.
    Irvin, C.E., Robin, J.S.: Analysis of the Intel Pentium’s ability to support a secure virtual machine monitor. In: Proceedings of Usenix00 Conference (2000)Google Scholar
  38. 38.
    Josse, S.: How to assess the security of your anti-virus? In: Proceedings of the 15th EICAR Conference, Hamburg, Germany, 29 April–3 May 2006. In: Broucek, V. et al. (ed.) J. Comput. Virol. EICAR 2006 Special Issue, 1(2) (2006)Google Scholar
  39. 39.
    Josse, S.: Secure and advanced unpacking using computer emulation. In: Proceedings of the AVAR 2006 Conference, Auckland, New Zealand (2006)Google Scholar
  40. 40.
    MackT’s ImportREC: Available at: (2006)
  41. 41.
    Microsoft PE-COFF: Microsoft Portable Executable and Common Object File Format Specification, revision 8.0, 2006. Retrieved from (2006)
  42. 42.
    Nebbett, G.: Windows NT/2000 Native API Reference. MTP Press (2000)Google Scholar
  43. 43.
    Newbigin, J.: Dd for Windows. Retrieved from: (2006)
  44. 44.
    Ollydbg: Available at: (2007)
  45. 45.
    Ollydbg Plugins: Available at: browse/OllydbgPlugins/ (2007)
  46. 46.
    Pennell, A.: Post-Mortem Debugging Your Application with Minidumps and Visual Studio. NET (2002)Google Scholar
  47. 47.
    Pennell, A.: Minidumps tool (2002)Google Scholar
  48. 48.
    Portokalidis, G.: Zero Hour Worm Detection and Containment using Honeypots. Master Thesis, University of Crete (2004)Google Scholar
  49. 49.
    PE iDentifier. Available at: (2007)
  50. 50.
    Plex86 x86 Virtual Machine Project: Available at: (2007)
  51. 51.
    QEMU Project: Available at: (2006)
  52. 52.
    Rutkowska, J.: Detecting Windows Server Compromises with Patchfinder 2. Retrieved from:, September 2006 (2004)
  53. 53.
    Rutkowska, J.: Red Pill... or how to detect VMM using (almost) one CPU instruction. Retrieved from:, September 2006 (2004)
  54. 54.
    Russinovich, M.E., Solomon, D.A.: Inside Microsoft Windows 2000, 3rd edn. Microsoft Press, ISBN 0-7356-1021-5 (2000)Google Scholar
  55. 55.
    Russinovich, M.E., Solomon, D.A.: Microsoft Windows Internals, 4th edn: Microsoft Windows Server 2003, Windows XP, and Windows 2000 (2004)Google Scholar
  56. 56.
    Szor, P.: Memory scanning under Windows NT. In: Proceedings of Virus Bulletin Conference (1999)Google Scholar
  57. 57.
    Stepan, A.E.: Defeating polymorphism: beyond emulation. In: Proceedings of Virus Bulletin Conference (2005)Google Scholar
  58. 58.
    Schuster, A.: Reconstructing a Binary. Part 1, part 2. Retrieved from: (2006)
  59. 59.
    Schuster, A.: Tool MemDump.PL (PERL script). Retrieved from: (2006)
  60. 60.
    Schuster, A.: Tool PTFinder.PL (Find Processes and Threads in a Microsoft Windows memory dump, PERL script). Retrieved from: (2006)
  61. 61.
    Schuster, A.: Improving list-walkers. Retrieved from: (2006)
  62. 62.
    Schuster, A.: Acquisition: dd. Retrieved from: (2006)
  63. 63.
    Schuster, A.: Adapting PTfinder to other Versions of Microsoft Windows. Retrieved from: (2006)
  64. 64.
    Schuster, A.: Converting Virtual into Physical Addresses. Retrieved from: (2006)
  65. 65.
    Schuster, A.: Searching for Processes and Threads. Retrieved from: (2006)
  66. 66.
    Schuster, A.: More on Processes and Threads. Retrieved from: (2006)
  67. 67.
    Tröger, J.: Specification-Driven Dynamic Binary Translation. Ph.D. Thesis from Queensland University of Technology, Brisbane, Australia (2004)Google Scholar
  68. 68.
    VMware ACE: Available at: ace/ (2007)
  69. 69.
    VX Heavens Virus Collection: Retrieved from (2006)
  70. 70.
    Weariless: Performing a hex dump of another process’s memory. Retrieved from:, September 2006 (2003)
  71. 71.
    Weariless: MDump tool. Retrieved from:, September 2006 (2003)
  72. 72.
    y0da’s LordPE: Available at: (2007)
  73. 73.
    Z0mbie: Automated reverse engineering: Mistfall engine. Retrieved from:, September 2006 (2000)
  74. 74.
    Z0mbie: VMWare has you. Retrieved from:, September 2006 (2001)

Copyright information

© Springer-Verlag France 2007

Authors and Affiliations

  1. 1.Silicomp-AQL, Security Evaluation LabCesson-SévignéFrance

Personalised recommendations