Venable M., Chouchane M.R., Karim M.E., Lakhotia A. (2005) Analyzing Memory Accesses in Obfuscated x86 Executables. In: Julisch K., Kruegel C. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2005. Lecture Notes in Computer Science, vol 3548. Springer, Berlin, Heidelberg
Programmers obfuscate their code to defeat manual or automated analysis. Obfuscations are often used to hide malicious behavior. In particular, malicious programs employ obfuscations of stack-based instructions, such as call and return instructions, to prevent an analyzer from determining which system functions it calls. Instead of using these instructions directly, a combination of other instructions, such as PUSH and POP, are used to achieve the same semantics. This paper presents an abstract interpretation based analysis to detect obfuscation of stack instructions. The approach combines Reps and Balakrishnan’s value set analysis (VSA) and Lakhotia and Kumar’s Abstract Stack Graph, to create an analyzer that can track stack manipulations where the stack pointer may be saved and restored in memory or registers. The analysis technique may be used to determine obfuscated calls made by a program, an important first step in detecting malicious behavior.