Multi-stage Binary Code Obfuscation Using Improved Virtual Machine

  • Hui Fang
  • Yongdong Wu
  • Shuhong Wang
  • Yin Huang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7001)


A software obfuscator transforms a program into another executable one with the same functionality but unreadable code implementation. This paper presents an algorithm of multi-stage software obfuscation method using improved virtual machine techniques. The key idea is to iteratively obfuscate a program for many times in using different interpretations. An improved virtual machine (VM) core is appended to the protected program for byte-code interpretation. Adversaries will need to crack all intermediate results in order to figure out the structure of original code. Compared with existing obfuscators, our new obfuscator generates the protected code which performs more efficiently, and enjoys proven higher level security.


Virtual Machine Reverse Engineering Basic Block Original Program Digital Right Management 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, M., Plotkin, G.: On protection by layout randomization. In: 23rd IEEE Computer Security Foundations Symposium, pp. 337–351 (2010)Google Scholar
  2. 2.
    Anckaert, B., Madou, M., De Sutter, B., De Bus, B., De Bosschere, K., Preneel, B.: Program obfuscation: a quantitative approach. In: ACM Workshop on Quality of Protection, pp. 15–20 (2007)Google Scholar
  3. 3.
    Barak, B., Goldreich, O., Impagliazzo, R., Rudich, S., Sahai, A., Vadhan, S., Yang, K.: On the (Im)possibility of obfuscating programs. In: Kilian, J. (ed.) CRYPTO 2001. LNCS, vol. 2139, pp. 1–18. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  4. 4.
    Beaucamps, P., Filiol, E.: On the possibility of practically obfuscating programs towards a unified perspective of code protection. Journal in Computer Virology 3, 3–21 (2007)CrossRefGoogle Scholar
  5. 5.
    Bitansky, N., Canetti, R.: On Strong Simulation and Composable Point Obfuscation. In: Rabin, T. (ed.) CRYPTO 2010. LNCS, vol. 6223, pp. 520–537. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Canetti, R., Dakdouk, R.R.: Obfuscating Point Functions with Multibit Output. In: Smart, N.P. (ed.) EUROCRYPT 2008. LNCS, vol. 4965, pp. 489–508. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  7. 7.
    Canetti, R., Tauman Kalai, Y., Varia, M., Wichs, D.: On Symmetric Encryption and Point Obfuscation. In: Micciancio, D. (ed.) TCC 2010. LNCS, vol. 5978, pp. 52–71. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Cappaert, J., Preneel, B., Anckaert, B., Madou, M., De Bosschere, K.: Towards tamper resistant code encryption: Practice and experience. In: Chen, L., Mu, Y., Susilo, W. (eds.) ISPEC 2008. LNCS, vol. 4991, pp. 86–100. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Ceccato, M., Di Penta, M., Nagra, J., Falcarin, P., Ricca, F., Torchiano, M., Tonella, P.: The effectiveness of source code obfuscation -an experimental assessment. In: The 17th IEEE International Conference on Program Comprehension (ICPC), pp. 178–187. IEEE Computer Society, Los Alamitos (2009)Google Scholar
  10. 10.
    Collberg, C.: Tutorial: code transformation techniques for software protection. In: ACM SIGPLAN 2009 Conference on Programming Language Design and Implementation, PLDI 2009 (2009)Google Scholar
  11. 11.
    Collberg, C., Thomborson, C.: Watermarking, tamper-proofing, and obfuscation - tools for software protection. IEEE Transactions on Software Engineering 28, 735–746 (2002)CrossRefGoogle Scholar
  12. 12.
    Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations. Technical report (1997)Google Scholar
  13. 13.
    DataRescue. The ida pro disassembler and debugger (2005),
  14. 14.
    Ge, J.: Control flow based obfuscation. In: Proceedings of the 5th ACM Workshop on Digital Rights Management (DRM), pp. 83–92. ACM Press, New York (2005)CrossRefGoogle Scholar
  15. 15.
    Goldweisser, S.: On the impossibility of obfuscation with auxiliary input, pp. 553–562. IEEE Computer Society, Los Alamitos (2005)Google Scholar
  16. 16.
    Hohenberger, S., Rothblum, G.N., Shelat, A., Vaikuntanathan, V.: Securely Obfuscating Re-encryption. In: Vadhan, S.P. (ed.) TCC 2007. LNCS, vol. 4392, pp. 233–252. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Hohenberger, S., Waters, B.: Constructing Verifiable Random Functions with Large Input Spaces. In: Gilbert, H. (ed.) EUROCRYPT 2010. LNCS, vol. 6110, pp. 656–672. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  18. 18.
    Jhala, R., Majumdar, R.: Path slicing. In: Proceedings of ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2005, pp. 38–47. ACM, New York (2005)CrossRefGoogle Scholar
  19. 19.
    Kanzaki, Y., Monden, A., Nakamura, M.: A software protection method based on instruction camouflage. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences (Japanese Edition) J87-A(6):755-767, 47–59 (2004)Google Scholar
  20. 20.
    Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: ACM Conference on Computer and Communications Security (CCS), pp. 290–299. ACM Press, New York (2003)Google Scholar
  21. 21.
    Lynn, B., Prabhakaran, M., Sahai, A.: Positive Results and Techniques for Obfuscation. In: Cachin, C., Camenisch, J.L. (eds.) EUROCRYPT 2004. LNCS, vol. 3027, pp. 20–39. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  22. 22.
    Madou, M., Anckaert, B., De Bus, B., De Bosschere, K.: On the effectiveness of source code transformations for binary obfuscation. In: Proc. of the Int’l Conf. on Software Engineering Research and Practice (SERP 2006), pp. 527–533 (2006)Google Scholar
  23. 23.
    Madou, M., Anckaert, B., Moseley, P., Debray, S.K., De Sutter, B., De Bosschere, K.: Software protection through dynamic code mutation. In: Song, J.-S., Kwon, T., Yung, M. (eds.) WISA 2005. LNCS, vol. 3786, pp. 194–206. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  24. 24.
    Madou, M., Van Put, L., De Bosschere, K.: Understanding obfuscated code. In: 14th IEEE Int’l Conf. on Program Comprehension (ICPC), pp. 268–274 (2006)Google Scholar
  25. 25.
    Mit, M.E., Ernst, M.D.: Static and dynamic analysis: synergy and duality. In: WODA 2003: ICSE Workshop on Dynamic Analysis, pp. 24–27 (2003)Google Scholar
  26. 26.
    Monden, A., Monsifrot, A., Thomborson, C.: Security improvements for encrypted interpretation. In: Proc. 3rd Workshop on Application Specific Processors (WASP) Digest, pp. 19–26 (2004)Google Scholar
  27. 27.
    Naeem, N.A., Batchelder, M., Hendren, L.: Metrics for measuring the effectiveness of decompilers and obfuscator. In: 15th IEEE Int’l. Conf. on Program Comprehension, pp. 253–258 (2007)Google Scholar
  28. 28.
    Ogiso, T., Sakabe, Y., Soshi, M., Miyaji, A.: Software obfuscation on a theoretical basis and its implementation. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E86-A(1), 176–186 (2003)Google Scholar
  29. 29.
    Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: USENIX Security Symposium (2007)Google Scholar
  30. 30.
    Dalla Preda, M., Madou, M., De Bosschere, K., Giacobazzi, R.: Opaque Predicates Detection by Abstract Interpretation. In: Johnson, M., Vene, V. (eds.) AMAST 2006. LNCS, vol. 4019, pp. 81–95. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  31. 31.
    Rolles, R.: X86 virtualizer (2008),
  32. 32.
    Rolles, R.: Unpacking virtualization obfuscators. In: Proceedings of the 3rd USENIX Conference on Offensive Technologies, WOOT 2009, p. 1. USENIX Association (2009)Google Scholar
  33. 33.
    Schwarz, B., Debray, S.K., Andrews, G.R.: Disassembly of executable code revisited. In: 10th Working Conference on Reverse Engineering, pp. 45–54 (2002)Google Scholar
  34. 34.
    Sharif, M., Lanzi, A., Giffin, J., Lee, W.: Automatic reverse engineering of malware emulators. In: Proceedings of the 30th IEEE Symposium on Security and Privacy, pp. 94–109. IEEE Computer Society, Los Alamitos (2009)Google Scholar
  35. 35.
    Sivadasan, P., Sojan Lal, P.: Jconsthide: a framework for java source code constant hiding. CoRR (2009)Google Scholar
  36. 36.
    Smith, J.E., Nair, R.: Virtual machines: versatile platforms for systems and processes. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  37. 37.
    Oreans Technologies. Code virtualizer,
  38. 38.
    Udupa, S.K., Debray, S.K., Madou, M.: Deobfuscation: reverse engineering obfuscated code. In: 12th Working Conference on Reverse Engineering, pp. 45–54 (2005)Google Scholar
  39. 39.
    van Oorschot, P.C.: Revisiting Software Protection. In: Boyd, C., Mao, W. (eds.) ISC 2003. LNCS, vol. 2851, pp. 1–13. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  40. 40.
    VMPsoft. Vmprotect software,
  41. 41.
    Wang, C., Hill, J., Knight, J.C., Davidson, J.W.: Protection of software-based survivability mechanism. In: Proceedings of the International Conference on Dependable Systems and Networks (formerly: FTCS), DSN 2001, pp. 193–202. IEEE Computer Society, Los Alamitos (2001)CrossRefGoogle Scholar
  42. 42.
    Wee, H.: On obfuscating point functions. In: Proceedings of the 37th Annual ACM Symposium on Theory of Computing, STOC 2005, pp. 523–532. ACM, New York (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Hui Fang
    • 1
  • Yongdong Wu
    • 1
  • Shuhong Wang
    • 2
  • Yin Huang
    • 2
  1. 1.Institute for Infocomm ResearchSingapore
  2. 2.Sumavision Soft Tech Co., Ltd.BeijingChina

Personalised recommendations