Skip to main content

Advertisement

SpringerLink
Log in

Secure and advanced unpacking using computer emulation

  • Original Paper
  • Published:
Journal in Computer Virology Aims and scope Submit manuscript

Abstract

The purpose of this article is firstly to present a secure unpacker which is specifically designed for a security analyst when studying viruses but also any anti-virus scanner. Such a tool is in fact required when assessing security requirements of an anti-virus scanner through a black box approach. During testing of anti-virus software, a security analyst needs to build virus populations required for several penetration tests. Virus unpacking is a first mandatory step before gaining the ability to apply obfuscation transformation or any information extraction algorithm on a viral set. A secure unpacker is also useful when checking security robustness against reverse engineering of any packed or protected security product. Several static and dynamic analysis tools already implement unpacking algorithms, but these often require human intervention and are not well designed to automatically unpack such a dangerous program as a virus. A new algorithm for automatically unpacking encrypted viruses is presented in this paper. Forensics techniques to reconstruct an unpacked executable and advanced heuristics are also presented in order to decrypt more sophisticated self-protected Malwares. We present several detection techniques which are specifically designed to deceive virtual machine monitors and discuss the security of our tool against these low-level viral attacks. Our secure unpacker figures among a set of several tools. We then present in this paper a proof-of-concept human analysis framework which implements most standard components of an anti-virus scanner (real-time scanner, emulator engine) and in addition proposes a reliable system for automatically gaining information about a virus and its interaction with the OS executive (stealth native API hooking), but focuses on human decision as a detection process without the same resource limitation constraint as product oriented anti-virus scanners. This framework is used as a basis/reference for the comparative analysis of security aspects of anti-virus scanners and deals with the robustness of their driver stack and the efficiency of their de-obfuscation and unpacking algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aho, A.V., Corasik, M.J.: Efficient string matching: an aid to bibliographic search. Commun. ACM 18(6), (1975)

  2. Argos project Retrieved from: https://gforce.cs.vu.nl/projects/argos/, http://www.few.vu.nl/argos/ (2006)

  3. Argos Howto: Howto: setting up Argos the 0day shellcode catcher. Retrieved from http://www.few.vu.nl/argos/ (2006)

  4. AV-Test.org project: Retrieved from http://www.av-test.org/ (2006)

  5. Butler, J.: DKOM (Direct Kernel Object Manipulation, slides). Retrieved from: http://www.blackhat.com/presentations/, (2006)

  6. Bayer, U., Kruegel, C., Kirda, E.: TTAnalyze: A tool for analyzing malware. In: proceedings of the 15th EICAR Conference, Hamburg, Germany, 29 April–3 May 2006. In: Broucek, V. et al. (ed.) J. Comput. Virol., EICAR 2006 Special Issue, 2006 (2005)

  7. Bos, H.: A personal view on the future of Zero-day Worm Containment (slides) (2006)

  8. Bellard, F.: QEMU, a Fast and Portable Dynamic Translator. In: Proceedings of the 2005 USENIX Conference (2005)

  9. Betz, C.: MemParser tool. Retrieved from: http://memparser.sourceforge.net/ (2006)

  10. Beaucamp, P., Filiol, E.: On the possibility of practically obfuscating programs: towards a unified perspective of code protection. In: Proceedings of the First International Workshop in Theoretical Virology 2006, Nancy, May 2007, In: Bonfante, G., Marion, J.-Y., (eds.) WTCV’06 Special Issue, J. Comput. Virol. 3(1) 2007 (2006)

  11. Brosch, T., Morgenstern, M.: Runtime packers: the hidden problem. Black Hat 2006 Conference (2006)

  12. Bochs: Bochs, the open source IA-32 emulation project. Available at: http://bochs.sourceforge.net/ http://bochs.sourceforge.net/ (2007)

  13. Brulez, N.: Anti Reverse Engineering Uncovered. Code Breakers Journal. http://www.CodeBreakers-Journal.com Previously published at the Honeynet Project, Scan of the Month 33 (2005)

  14. Burdach, M.: An Introduction to Windows memory forensic. Retrieved from: http://forensic.seccure.net, September 2006 (2005)

  15. Burdach, M.: Digital forensics of the physical memory. Retrieved from:http://forensic.seccure.net, September 2006 (2005)

  16. Burdach, M.: idetect, ProcEnum, WMFT tools. Retrieved from: http://forensic.seccure.net, September 2006 (2005)

  17. Burdach, M.: Digital Investigation. Retrieved from: http://forensic.seccure.net (2006)

  18. Burdach, M.: Finding Digital Evidence In Physical Memory (slides). Retrieved from: http://forensic.seccure.net (2006)

  19. Butler, J., Hoglund, G.: Rootkits: Subverting the Windows Kernel. Addison Wesley, ISBN 0-321-29431-9 (2006)

  20. Bos, H., Portokalidis, G., Slowinska, A.: Argos: An Emulator for Fingerprinting Zero-Day Attacks. In: Proceedings EuroSys (2006)

  21. Cohen, F.: Computer viruses, Ph.D. thesis, University of Southern California (1986)

  22. Carvey, H.: Reassembling an image file from a memory dump. Retrieved from: http://sourceforge.net/projects/windowsir (2006)

  23. Carvey, H.: Ramdump, lsproc, lspm, ReadPE tools. Retrieved from: http://sourceforge.net/projects/windowsir (2006)

  24. Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., Veith, H.: Malware Normalization, Technical Report, University of Wisconsin, Madison, USA (2005)

  25. Clam AntiVirus: Available at: http://www.clamav.net/ (2007)

  26. Cloakware: Retrieved from: http://www.cloakware.com/ (2007)

  27. Cogswell, C., Russinovich, M.: RootkitRevealer. Available at: http://www.sysinternals.com/ (2006)

  28. DataRescue: Using the Universal PE Unpacker Plug-in included in IDA Pro 4.9 to unpack compressed executables. Retrieved from: http://www.datarescue.com/idabase/, September 2006 (2005)

  29. DataRescue: Using the IDA debugger to unpack an hostile PE executable. Retrieved from: http://www.datarescue.com/idabase/ (2006)

  30. Elias: Detect if your program is running inside a Virtual Machine. 14 Mars 2005. Retrieved from: http://lgwm.org (Elias homepage), September 2006 (2005)

  31. Filiol, F.: Strong cryptography armoured computer viruses forbidding code analysis: the Bradley virus. In: Proceedings of the 14th EICAR Conference, pp. 210–214 (2005)

  32. Filiol E.: Techniques virales avancées, IRIS Series, Springer Verlag France, January 2007. An English translation is pending (due mid 2007) (2007)

  33. Filiol, F., Josse, S.: A statistical model for undecidable viral detection. In: Proceedings of the 16th EICAR Conference, Budapest, Hungary, 5–8 May 2007. To appear in: Broucek, V. (ed.) Eicar 2007 Special Issue, J. Comput. Virol. 3(2) (2007)

  34. Ferrie, P.: Attacks on virtual machine emulators. In: Proceedings of the 2006 AVAR Conference, Auckland, NZ (2006)

  35. Garner, G.M.: Forensic Acquisition Utilities: Dd, md5lib, md5sum, VolumeDump, Wipe, ZlibU, nc, GetOpt. Retrieved from: http://users.erols.com/gmgarner/forensics/, (2006)

  36. Garner, G.M., Mora, R.: Kntlist tool. Retrieved from: http://www.dfrws.org/2005/challenge/kntlist.html (2006)

  37. Irvin, C.E., Robin, J.S.: Analysis of the Intel Pentium’s ability to support a secure virtual machine monitor. In: Proceedings of Usenix00 Conference (2000)

  38. Josse, S.: How to assess the security of your anti-virus? In: Proceedings of the 15th EICAR Conference, Hamburg, Germany, 29 April–3 May 2006. In: Broucek, V. et al. (ed.) J. Comput. Virol. EICAR 2006 Special Issue, 1(2) (2006)

  39. Josse, S.: Secure and advanced unpacking using computer emulation. In: Proceedings of the AVAR 2006 Conference, Auckland, New Zealand (2006)

  40. MackT’s ImportREC: Available at: http://mackt.cjb.net/ (2006)

  41. Microsoft PE-COFF: Microsoft Portable Executable and Common Object File Format Specification, revision 8.0, 2006. Retrieved from http://msdn.microsoft.com/ (2006)

  42. Nebbett, G.: Windows NT/2000 Native API Reference. MTP Press (2000)

  43. Newbigin, J.: Dd for Windows. Retrieved from: http://uranus.it.swin.edu.au/~jn/linux/rawwrite/dd.htm (2006)

  44. Ollydbg: Available at: http://www.ollydbg.de/ (2007)

  45. Ollydbg Plugins: Available at: http://www.openrce.org/download/ browse/OllydbgPlugins/ (2007)

  46. Pennell, A.: Post-Mortem Debugging Your Application with Minidumps and Visual Studio. NET (2002)

  47. Pennell, A.: Minidumps tool (2002)

  48. Portokalidis, G.: Zero Hour Worm Detection and Containment using Honeypots. Master Thesis, University of Crete (2004)

  49. PE iDentifier. Available at: http://peid.tk (2007)

  50. Plex86 x86 Virtual Machine Project: Available at: http://plex86.sourceforge.net/ (2007)

  51. QEMU Project: Available at: http://fabrice.bellard.free.fr/qemu/ (2006)

  52. Rutkowska, J.: Detecting Windows Server Compromises with Patchfinder 2. Retrieved from: http://www.invisiblethings.org/papers/, September 2006 (2004)

  53. Rutkowska, J.: Red Pill... or how to detect VMM using (almost) one CPU instruction. Retrieved from: http://www.invisiblethings.org/papers/, September 2006 (2004)

  54. Russinovich, M.E., Solomon, D.A.: Inside Microsoft Windows 2000, 3rd edn. Microsoft Press, ISBN 0-7356-1021-5 (2000)

  55. Russinovich, M.E., Solomon, D.A.: Microsoft Windows Internals, 4th edn: Microsoft Windows Server 2003, Windows XP, and Windows 2000 (2004)

  56. Szor, P.: Memory scanning under Windows NT. In: Proceedings of Virus Bulletin Conference (1999)

  57. Stepan, A.E.: Defeating polymorphism: beyond emulation. In: Proceedings of Virus Bulletin Conference (2005)

  58. Schuster, A.: Reconstructing a Binary. Part 1, part 2. Retrieved from: http://computer.forensikblog.de/en/2006/04/reconstructing_a_binary.html (2006)

  59. Schuster, A.: Tool MemDump.PL (PERL script). Retrieved from: http://computer.forensikblog.de/ (2006)

  60. Schuster, A.: Tool PTFinder.PL (Find Processes and Threads in a Microsoft Windows memory dump, PERL script). Retrieved from: http://computer.forensikblog.de/en/topics/windows/memory_analysis/ (2006)

  61. Schuster, A.: Improving list-walkers. Retrieved from: http://computer.forensikblog.de/ (2006)

  62. Schuster, A.: Acquisition: dd. Retrieved from: http://computer.forensikblog.de/ (2006)

  63. Schuster, A.: Adapting PTfinder to other Versions of Microsoft Windows. Retrieved from: http://computer.forensikblog.de/ (2006)

  64. Schuster, A.: Converting Virtual into Physical Addresses. Retrieved from: http://computer.forensikblog.de/ (2006)

  65. Schuster, A.: Searching for Processes and Threads. Retrieved from: http://computer.forensikblog.de/ (2006)

  66. Schuster, A.: More on Processes and Threads. Retrieved from: http://computer.forensikblog.de/ (2006)

  67. Tröger, J.: Specification-Driven Dynamic Binary Translation. Ph.D. Thesis from Queensland University of Technology, Brisbane, Australia (2004)

  68. VMware ACE: Available at: http://www.vmware.com/products/ ace/ (2007)

  69. VX Heavens Virus Collection: Retrieved from http://vx.netlux.org/ (2006)

  70. Weariless: Performing a hex dump of another process’s memory. Retrieved from: http://www.codeproject.com/, September 2006 (2003)

  71. Weariless: MDump tool. Retrieved from: http://www.codeproject.com/, September 2006 (2003)

  72. y0da’s LordPE: Available at: http://y0da.cjb.net (2007)

  73. Z0mbie: Automated reverse engineering: Mistfall engine. Retrieved from: http://vx.netlux.org/, September 2006 (2000)

  74. Z0mbie: VMWare has you. Retrieved from: http://vx.netlux.org/, September 2006 (2001)

Download references

Author information

Authors and Affiliations

  1. Silicomp-AQL, Security Evaluation Lab, 1 rue de la Châtaigneraie, 51766, Cesson-Sévigné, France

    Sébastien Josse

Authors
  1. Sébastien Josse
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sébastien Josse.

Additional information

This paper is the extended version presented at the AVAR 2006 Conference [39].

Sébastien Josse is an I.T. consultant at Silicomp-AQL Security Evaluation Lab and also a Ph.D. student EDX Polytechnique Doctoral School within the ESAT Virology and Cryptology Lab in Rennes email: sebastien.josse@esat.terre.defense.gouv.fr.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Josse, S. Secure and advanced unpacking using computer emulation. J Comput Virol 3, 221–236 (2007). https://doi.org/10.1007/s11416-007-0046-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-007-0046-0

Keywords

Search

Navigation