Analyzing Memory Accesses in Obfuscated x86 Executables

  • Michael Venable
  • Mohamed R. Chouchane
  • Md Enamul Karim
  • Arun Lakhotia
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3548)

Abstract

Programmers obfuscate their code to defeat manual or automated analysis. Obfuscations are often used to hide malicious behavior. In particular, malicious programs employ obfuscations of stack-based instructions, such as call and return instructions, to prevent an analyzer from determining which system functions it calls. Instead of using these instructions directly, a combination of other instructions, such as PUSH and POP, are used to achieve the same semantics. This paper presents an abstract interpretation based analysis to detect obfuscation of stack instructions. The approach combines Reps and Balakrishnan’s value set analysis (VSA) and Lakhotia and Kumar’s Abstract Stack Graph, to create an analyzer that can track stack manipulations where the stack pointer may be saved and restored in memory or registers. The analysis technique may be used to determine obfuscated calls made by a program, an important first step in detecting malicious behavior.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Linn, C., Debray, S.: Obfuscation of Executable Code to Improve Resistance to Static Disassembly. In: 10th ACM Conference on Computer and Communications Security, CCS (2003)Google Scholar
  2. 2.
    Collberg, C., Thomborson, C., Low, D.: A Taxonomy of Obfuscating Transformations. Technical Report 148, Department of Computer Science, University of Auckland (1997)Google Scholar
  3. 3.
    Lakhotia, A., Kumar, E.U.: Abstract Stack Graph to Detect Obfuscated Calls in Binaries. In: Fourth IEEE International Workshop on Source Code Analysis and Manipulation(SCAM 2004), Chicago, Illinois (2004)Google Scholar
  4. 4.
    Ször, P., Ferrie, P.: Hunting for Metamorphic. In: Virus Bulletin Conference, Prague, Czech Republic (2001)Google Scholar
  5. 5.
    Lakhotia, A., Singh, P.K.: Challenges in Getting ’Formal’ with Viruses. Virus Bulletin (2003)Google Scholar
  6. 6.
    Ször, P.: The New 32-Bit Medusa. Virus Bulletin, 8–10 (2000)Google Scholar
  7. 7.
    Bergeron, J., Debbabi, M.: Detection of Malicious Code in Cots Software: A Short Survey. In: First International Software Assurance Certification Conference (ISACC 1999), Washington DC (1999)Google Scholar
  8. 8.
    Symantec, Understanding Heuristics: Symantec’s Bloodhound Technology, http://www.symantec.com/avcenter/reference/heuristc.pdf (Last accessed July 1, 2004)
  9. 9.
    Balakrishnan, G., Reps, T.: Analyzing Memory Accesses in X86 Executables. In: 13th International Conference on Compiler Construction (2004)Google Scholar
  10. 10.
    Cousot, P., Cousot, R.: Static Determination of Dynamic Properties of Programs. In: 2nd Int. Symp. on Programming, Dumod, Paris, France (1976)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Michael Venable
    • 1
  • Mohamed R. Chouchane
    • 1
  • Md Enamul Karim
    • 1
  • Arun Lakhotia
    • 1
  1. 1.Center for Advanced Computer StudiesUniversity of Louisiana at Lafayette

Personalised recommendations