Deobfuscation of Virtualization-Obfuscated Code Through Symbolic Execution and Compilation Optimization
Virtualization-obfuscation replaces native code in a binary with semantically equivalent and self-defined bytecode, which, upon execution, is interpreted by a custom virtual machine. It makes the code very difficult to analyze and is thus widely used in malware. How to deobfuscate such virtualization obfuscated code has been an important and challenging problem. We approach the problem from an innovative perspective by transforming it into a compilation optimization problem, and propose a novel technique that combines trace analysis, symbolic execution and compilation optimization to defeat virtualization obfuscation. We implement a prototype system and evaluate it against popular virtualization obfuscators; the results demonstrate that our method is effective in deobfuscation of virtualization-obfuscated code.
KeywordsDeobfuscation Virtualization obfuscation Symbolic execution Compilation optimization
This work was supported in part by National High Technology Research and Development Program of China (No. 2015AA016004), the National Key R&D Program of China (No. 2016QY04W0802).
- 1.Nagra, J., Collberg, C.: Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing for Software Protection. Pearson Education, London (2009)Google Scholar
- 2.Rolles, R.: Unpacking virtualization obfuscators. In: 3rd USENIX Workshop on Offensive Technologies (WOOT) (2009)Google Scholar
- 3.Sharif, M., Lanzi, A., Giffin, J., Lee, W.: Automatic reverse engineering of malware emulators. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 94–109. IEEE (2009)Google Scholar
- 4.Coogan, K., Lu, G., Debray, S.: Deobfuscation of virtualization-obfuscated software: a semantics-based approach. In: Proceedings of the 18th ACM Conference on Computer and Communications Security, pp. 275–284. ACM (2011)Google Scholar
- 5.HexEffect: Virtual deobfuscator. http://www.hexeffect.com/virtual_deob.html
- 6.VMProtect: Vmprotect software protection. http://vmpsoft.com/
- 7.Oreans: Code virtualizer. https://oreans.com/codevirtualizer.php
- 8.Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: SoK: deep packer inspection: a longitudinal study of the complexity of run-time packers. In: 2015 IEEE Symposium on Security and Privacy (SP), pp. 659–673. IEEE (2015)Google Scholar
- 10.Eagle, C.: The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler. No Starch Press, San Francisco (2011)Google Scholar
- 11.CEA: cea-sec/miasm: reverse engineering framework in Python. https://github.com/cea-sec/miasm
- 12.Kalysch, A., Götzfried, J., Müller, T.: VMAttack: deobfuscating virtualization-based packed binaries. In: ARES (2017)Google Scholar
- 13.Yadegari, B., Johannesmeyer, B., Whitely, B., Debray, S.: A generic approach to automatic deobfuscation of executable code. In: 2015 IEEE Symposium on Security and Privacy (SP), pp. 674–691. IEEE (2015)Google Scholar