Deobfuscation of Virtualization-Obfuscated Code Through Symbolic Execution and Compilation Optimization

  • Mingyue Liang
  • Zhoujun LiEmail author
  • Qiang Zeng
  • Zhejun Fang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10631)


Virtualization-obfuscation replaces native code in a binary with semantically equivalent and self-defined bytecode, which, upon execution, is interpreted by a custom virtual machine. It makes the code very difficult to analyze and is thus widely used in malware. How to deobfuscate such virtualization obfuscated code has been an important and challenging problem. We approach the problem from an innovative perspective by transforming it into a compilation optimization problem, and propose a novel technique that combines trace analysis, symbolic execution and compilation optimization to defeat virtualization obfuscation. We implement a prototype system and evaluate it against popular virtualization obfuscators; the results demonstrate that our method is effective in deobfuscation of virtualization-obfuscated code.


Deobfuscation Virtualization obfuscation Symbolic execution Compilation optimization 



This work was supported in part by National High Technology Research and Development Program of China (No. 2015AA016004), the National Key R&D Program of China (No. 2016QY04W0802).


  1. 1.
    Nagra, J., Collberg, C.: Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing for Software Protection. Pearson Education, London (2009)Google Scholar
  2. 2.
    Rolles, R.: Unpacking virtualization obfuscators. In: 3rd USENIX Workshop on Offensive Technologies (WOOT) (2009)Google Scholar
  3. 3.
    Sharif, M., Lanzi, A., Giffin, J., Lee, W.: Automatic reverse engineering of malware emulators. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 94–109. IEEE (2009)Google Scholar
  4. 4.
    Coogan, K., Lu, G., Debray, S.: Deobfuscation of virtualization-obfuscated software: a semantics-based approach. In: Proceedings of the 18th ACM Conference on Computer and Communications Security, pp. 275–284. ACM (2011)Google Scholar
  5. 5.
    HexEffect: Virtual deobfuscator.
  6. 6.
    VMProtect: Vmprotect software protection.
  7. 7.
    Oreans: Code virtualizer.
  8. 8.
    Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: SoK: deep packer inspection: a longitudinal study of the complexity of run-time packers. In: 2015 IEEE Symposium on Security and Privacy (SP), pp. 659–673. IEEE (2015)Google Scholar
  9. 9.
    Luk, C.K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. ACM SIGPLAN Not. 40, 190–200 (2005)CrossRefGoogle Scholar
  10. 10.
    Eagle, C.: The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler. No Starch Press, San Francisco (2011)Google Scholar
  11. 11.
    CEA: cea-sec/miasm: reverse engineering framework in Python.
  12. 12.
    Kalysch, A., Götzfried, J., Müller, T.: VMAttack: deobfuscating virtualization-based packed binaries. In: ARES (2017)Google Scholar
  13. 13.
    Yadegari, B., Johannesmeyer, B., Whitely, B., Debray, S.: A generic approach to automatic deobfuscation of executable code. In: 2015 IEEE Symposium on Security and Privacy (SP), pp. 674–691. IEEE (2015)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Mingyue Liang
    • 1
  • Zhoujun Li
    • 1
    Email author
  • Qiang Zeng
    • 2
  • Zhejun Fang
    • 3
  1. 1.Beihang UniversityBeijingChina
  2. 2.Temple UniversityPhiladelphiaUSA
  3. 3.CNCERT/CCBeijingChina

Personalised recommendations