Efficient Detection of the Return-Oriented Programming Malicious Code

  • Ping Chen
  • Xiao Xing
  • Hao Han
  • Bing Mao
  • Li Xie
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6503)


Return-Oriented Programming (ROP) is a code-reuse technique which helps the attacker construct malicious code by using the instruction snippets in existing libraries/executables. Such technique makes the ROP program contain no malicious instructions. Moreover, in recent research, Return-Oriented Programming without returns has been proposed, which can be used to mount an attack without any independent return instructions, therefore, ROP malicious code circumvents the existing defenses which are based on the assumption that the ROP malicious code should use the ret without corresponding call. In this paper, we found the intrinsic feature of the ROP shellcode, and proposed an efficient method which can detect the ROP malicious code (including the one without returns). Preliminary experimental results show that our method can efficiently detect ROP malicious code and have no false positives and negatives.


Malicious Code Library Function Frame Function Normal Program Target Address 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    The pax project (2004),
  2. 2.
    Abadi, M., Budiu, M., Ligatti, J.: Control-flow integrity. In: Proceedings of the 12th ACM Conference on Computer and Communications Security (CCS), pp. 340–353. ACM, New York (2005)Google Scholar
  3. 3.
    Bhatkar, E., Duvarney, D.C., Sekar, R.: Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, pp. 105–120 (2003)Google Scholar
  4. 4.
    Blazakis, D.: Interpreter exploitation: pointer inference and jit spraying. BHDC (2010),
  5. 5.
    Buchanan, E., Roemer, R., Shacham, H., Savage, S.: When good instructions go bad: generalizing return-oriented programming to risc. In: Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS), pp. 27–38 (2008)Google Scholar
  6. 6.
    Caballero, J., Johnson, N.M., McCamant, S., Song, D.: Binary code extraction and interface identification for security applications. In: Proceedings of the 17th Annual Network and Distributed System Security Symposium (2010)Google Scholar
  7. 7.
    Checkoway, S., Davi, L., Dmitrienko, A., Sadeghi, A.R., Shacham, H., Winandy, M.: Return-oriented programming without returns. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS (2010)Google Scholar
  8. 8.
    Checkoway, S., Feldman, A.J., Kantor, B., Halderman, J.A., Felten, E.W., Shacham, H.: Can dres provide long-lasting security? the case of return-oriented programming and the avc advantage. In: Proceedings of EVT/WOTE 2009. USENIX/ACCURATE/IAVoSS (2009)Google Scholar
  9. 9.
    Chen, P., Xiao, H., Shen, X., Yin, X., Mao, B., Xie, L.: Drop: Detecting return-oriented programming malicious code. In: Prakash, A., Sen Gupta, I. (eds.) ICISS 2009. LNCS, vol. 5905, pp. 163–177. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  10. 10.
    Datarescue: Interactive disassembler (ida) pro (2008),
  11. 11.
    Davi, L., Sadeghi, A.R., Winandy, M.: Dynamic integrity measurement and attestation: towards defense against return-oriented programming attacks. In: Proceedings of the 2009 ACM Workshop on Scalable Trusted Computing, pp. 49–54 (2009)Google Scholar
  12. 12.
    Davi, L., Sadeghi, A.R., Winandy, M.: Ropdefender: A detection tool to defend against return-oriented programming attacks. Technical Report HGI-TR-2010-001 (2010),
  13. 13.
    Durden, T.: Bypassing pax aslr protection. Phrack Magazine (2002)Google Scholar
  14. 14.
    Francillon, A., Perito, D., Castelluccia, C.: Defending embedded systems against control flow attacks. In: Proceedings of the First ACM Workshop on Secure Execution of Untrusted Code, SecuCode 2009, pp. 19–26. ACM, New York (2009)CrossRefGoogle Scholar
  15. 15.
    Francillon, A., Castelluccia., C.: Code injection attacks on harvard-architecture devices. In: Syverson, P., Jha, S. (eds.) Proceedings of CCS 2008 (2008)Google Scholar
  16. 16.
    Kolbitsch, C., Holz, T., Kruegel, C., Kirda, E.: Inspector gadget: Automated extraction of proprietary gadgets from malware binaries. In: Proceedings of the 30th IEEE Symposium on Security and Privacy (2010)Google Scholar
  17. 17.
    Kornau, T.: Return oriented programming for the arm architecture. Master’s thesis, Ruhr-Universitat Bochum (2010)Google Scholar
  18. 18.
    Li, J., Wang, Z., Jiang, X., Grace, M., Bahram, S.: Defeating return-oriented rootkits with “return-less” kernels. In: Proceedings of the 5th European Conference on Computer Systems, EuroSys 2010, pp. 195–208. ACM, New York (2010)Google Scholar
  19. 19.
    Lidner, F.F.: Developments in cisco ios forensics. CONFidence 2.0,
  20. 20.
    Lin, Z., Zhang, X., Xu, D.: Reuse-oriented camouflaging trojan: Vulnerability detection and attack construction. In: Proceedings of the 40th DSN-DCCS (2010)Google Scholar
  21. 21.
    Luk, C.K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 190–200. ACM, New York (2005)CrossRefGoogle Scholar
  22. 22.
  23. 23.
    Nergal: The advanced return-into-lib(c) exploits (pax case study). Phrack Magazine (2001),
  24. 24.
    Nethercote, N., Seward, J.: Valgrind: a framework for heavyweight dynamic binary instrumentation. In: Proceedings of the 2007 PLDI Conference, vol. 42(6), pp. 89–100 (2007)Google Scholar
  25. 25.
  26. 26.
    Roesch, M.: Snort - lightweight intrusion detection for networks. In: Proceedings of the 13th USENIX Conference on System Administration, pp. 229–238. USENIX Association, Berkeley (1999)Google Scholar
  27. 27.
    Shacham, H.: The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86). In: Proceedings of the 14th ACM Conference on Computer and Communications Security (CCS), pp. 552–561. ACM, New York (2007)Google Scholar
  28. 28.
    Symantec: Dynamic linking in linux and windows, part one (2006),
  29. 29.
    Team, P.: What the future holds for pax (2003),
  30. 30.
    Bletsch, T., Jiang, X., Freeh, V.: Jump-oriented programming: A new class of code-reuse attack. Technical Report TR-2010-8 (2010)Google Scholar
  31. 31.
    Wang, X., Pan, C.C., Liu, P., Zhu, S.: Sigfree: A signature-free buffer overflow attack blocker. IEEE Transactions on Dependable and Secure Computing 99(2) (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ping Chen
    • 1
    • 2
  • Xiao Xing
    • 1
    • 2
  • Hao Han
    • 1
    • 2
  • Bing Mao
    • 1
    • 2
  • Li Xie
    • 1
    • 2
  1. 1.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina
  2. 2.Department of Computer Science and TechnologyNanjing UniversityNanjingChina

Personalised recommendations