Advertisement

Misleading Malware Similarities Analysis by Automatic Data Structure Obfuscation

  • Zhi Xin
  • Huiyu Chen
  • Hao Han
  • Bing Mao
  • Li Xie
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6531)

Abstract

Program obfuscation techniques have been widely used by malware to dodge the scanning from anti-virus detectors. However, signature based on the data structures appearing in the runtime memory makes traditional code obfuscation useless. Laika [2] implements this signature using Bayesian unsupervised learning, which clusters similar vectors of bytes in memory into the same class. We present a novel malware obfuscation technique that automatically obfuscate the data structure layout so that memory similarities between malware programs are blurred and hardly recognized. We design and implement the automatic data structure obfuscation technique as a GNU GCC compiler extension that can automatically distinguish the obfuscability of the data structures and convert part of the unobfuscable data structures into obfuscable. After evaluated by fourteen real-world malware programs, we present that our tool maintains a high proportion of obfuscated data structures as 60.19% for type and 60.49% for variable.

Keywords

data structure obfuscation malware 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Lin, Z., Zhang, X., Xu, D.: Automatic Reverse Engineering of Data Structures from Binary Execution. In: Proceedings of the 17th Annual Network and Distributed System Security Symposium (2010)Google Scholar
  2. 2.
    Cozzie, A., Stratton, F., Xue, H., King, S.T.: Digging for Data Structures. In: The 8th USENIX Symposium on Operating Systems Design and Implementation (2008)Google Scholar
  3. 3.
    Anubis: Analyzing Unknown Binaries (2009), http://anubis.seclab.tuwien.ac.at
  4. 4.
    CWSandbox (2009), http://www.cwsandbox.org/
  5. 5.
    Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM Conference on Computer and Communications Security (2003)Google Scholar
  6. 6.
    Christodorescu, M., Jha, S., Seshia, S.A., Songand, D., Bryant, R.E.: Semantics-Aware Malware Detection. In: Proceedings of the 2005 IEEE Symposium on Security and Privacy (2005)Google Scholar
  7. 7.
    Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: Proceedings of the 16th USENIX Security Symposium (2007)Google Scholar
  8. 8.
    Moser, A., Kruegel, C., Kirda, E.: Limits of Static Analysis for Malware Detection. In: 23rd Annual Computer Security Applications Conference (2007)Google Scholar
  9. 9.
    Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D.X.: Automatically Identifying Trigger-based Behavior in Malware. In: Lee, W., et al. (eds.) Book chapter in Botnet Analysis and Defense (2007)Google Scholar
  10. 10.
    Moser, A., Kruegel, C., Kirda, E.: Exploring Multiple Execution Paths for Malware Analysis. In: Proceedings of the 28th IEEE Symposium on Security and Privacy (2007)Google Scholar
  11. 11.
    Coogan, K., Debray, S.K., Kaochar, T., Townsend, G.M.: Automatic Static Unpacking of Malware Binaries. In: The 16th Working Conference on Reverse Engineering (2009)Google Scholar
  12. 12.
    Balakrishnan, G., Reps, T.: Analyzing Memory Accesses in x86 Executables. In: Duesterwald, E. (ed.) CC 2004. LNCS, vol. 2985, pp. 5–23. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. 13.
    Balakrishnan, G., Reps, T.W.: DIVINE: Discovering Variables IN Executables. In: Proceeding of Verification Model Checking and Abstract Interpretation (2007)Google Scholar
  14. 14.
    Szor, P.: The Art of Computer Virus Research and Defense. Addison Wesley, Reading (2005)Google Scholar
  15. 15.
    Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 16th USENIX Security Symposium (2003)Google Scholar
  16. 16.
    Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Impeding Malware Analysis Using Conditional Code Obfuscation. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (2008)Google Scholar
  17. 17.
    Pearce, S.: Viral polymorphism. VX Heavens (2003)Google Scholar
  18. 18.
    The Mental Drille Metamorphism in practice or How I made MetaPHOR and what I’ve learnt. VX Heavens (February 2002)Google Scholar
  19. 19.
    Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.A.: Behavior-based spyware detection. In: Proceedings of the 15th Conference on USENIX Security Symposium (2006)Google Scholar
  20. 20.
    Stallman, R.: Using GCC: the GNU compiler collection reference manual. GNU Press (2009)Google Scholar
  21. 21.
    TESO. Burneye ELF encryption program (January 2004), http://teso.scene.at
  22. 22.
    Detristan, T., Ulenspiegel, T., Malcom, Y., von Underduk, M.S.: Polymorphic Shellcode Engine Using Spectrum Analysis. Phrack 61 (2003)Google Scholar
  23. 23.
    Julus, L.: Metamorphism. VX heaven (March 2000), http://vx.netlux.org/lib/vlj00.html
  24. 24.
    Kruegel, C., Robertson, W., Vigna, G.: Detecting Kernel-Level Rootkits Through Binary Analysis. In: Proceedings of the 20th Annual Computer Security Applications Conference (2004)Google Scholar
  25. 25.
    Lin, Z., Riley, R.D., Xu, D.: Polymorphing Software by Randomizing Data Structure Layout. In: Proceedings of the 6th SIDAR Conference on Detection of Intrusions and Malware and Vulnerability Assessment (2009)Google Scholar
  26. 26.
    Balakrishnan, A., Schulze, C.: Code Obfuscation Literature Survey (2005), http://pages.cs.wisc.edu/~arinib/projects.htm
  27. 27.
    Colberg, Thomborson: Watermarking, Tamper-Proofing, and Obfuscation–Tools for Software Protection. IEEE Transactions on Software Engineering 28(8) (2002)Google Scholar
  28. 28.
    Bhatkar, S., Sekar, R., DuVarney, D.C.: Efficient techniques for comprehensive protection from memory error exploits. In: Proceedings of the 14th Conference on USENIX Security Symposium (2005)Google Scholar
  29. 29.
    Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: an efficient approach to combat a board range of memory error exploits. In: Proceedings of the 12th Conference on USENIX Security Symposium (2003)Google Scholar
  30. 30.
    Cifuentes, C., Gough, K.J.: Decompilation of Binary Programs. Software Practice & Experience (July 1995)Google Scholar
  31. 31.
    Ramalingam, G., Field, J., Tip, F.: Aggregate structure identification and its application to program analysis. In: Proceedings of the 26th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (1999)Google Scholar
  32. 32.
    Status of C99 features in GCC, GNU (1999), http://gcc.gnu.org/c99status.html
  33. 33.
    Richard Stevens, W.: Advanced Programming in the UNIX Environment. Addison-Wesley, Reading (1992)zbMATHGoogle Scholar
  34. 34.
    Shapiro, M., Horwitz, S.: The Effects of the Precision of Pointer Analysis. Lecture Notes in Computer Science (1997)Google Scholar
  35. 35.
    Collberg, C., Thomborson, C., Low, D.: A Taxonomy of Obfuscating Transformations. Technical Report 148, University of Auckland (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Zhi Xin
    • 1
  • Huiyu Chen
    • 1
  • Hao Han
    • 1
  • Bing Mao
    • 1
  • Li Xie
    • 1
  1. 1.State Key Laboratory for Novel Software Technology, Department of Computer Science and TechnologyNanjing UniversityChina

Personalised recommendations