On Emulation-Based Network Intrusion Detection Systems

  • Ali Abbasi
  • Jos Wetzels
  • Wouter Bokslag
  • Emmanuele Zambon
  • Sandro Etalle
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8688)


Emulation-based network intrusion detection systems have been devised to detect the presence of shellcode in network traffic by trying to execute (portions of) the network packet payloads in an instrumented environment and checking the execution traces for signs of shellcode activity. Emulation-based network intrusion detection systems are regarded as a significant step forward with regards to traditional signature-based systems, as they allow detecting polymorphic (i.e., encrypted) shellcode. In this paper we investigate and test the actual effectiveness of emulation-based detection and show that the detection can be circumvented by employing a wide range of evasion techniques, exploiting weakness that are present at all three levels in the detection process. We draw the conclusion that current emulation-based systems have limitations that allow attackers to craft generic shellcode encoders able to circumvent their detection mechanisms.


Emulation IDS Shellcode Evasion Polymorphism 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Polychronakis, M., Anagnostakis, K.G., Markatos, E.P.: Network–Level polymorphic shellcode detection using emulation. In: Büschkes, R., Laskov, P. (eds.) DIMVA 2006. LNCS, vol. 4064, pp. 54–73. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Shimamura, M., Kono, K.: Yataglass: Network-level code emulation for analyzing memory-scanning attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 68–87. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  3. 3.
    Polychronakis, M., Anagnostakis, K., Markatos, E.: Comprehensive shellcode detection using runtime heuristics. In: Proc. of the 26th Annual Computer Security Applications Conference (ACSAC 2010), pp. 287–296. ACM (2010)Google Scholar
  4. 4.
    Snow, K., Krishnan, S., Monrose, F., Provos, N.: SHELLOS: Enabling Fast Detection and Forensic Analysis of Code Injection Attacks. In: USENIX Security Symposium (2011)Google Scholar
  5. 5.
    Egele, M., Wurzinger, P., Kruegel, C., Kirda, E.: Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 88–106. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Gu, B., Bai, X., Yang, Z., Champion, A., Xuan, D.: Malicious shellcode detection with virtual memory snapshots. In: Proc. of IEEE INFOCOM 2010, pp. 1–9. IEEE (2010)Google Scholar
  7. 7.
    Portokalidis, G., Slowinska, A., Bos, H.: Argos: An emulator for fingerprinting zero-day attacks for advertised honeypots with automatic signature generation. In: Proc. of ACM SIGOPS Operating Systems Review, vol. 40(4), pp. 15–27. ACM (2006)Google Scholar
  8. 8.
    Zhang, Q., Reeves, D., Ning, P., Iyer, S.: Analyzing network traffic to detect self-decrypting exploit code. In: Proc. of the 2nd ACM Symposium on Information, Computer and Communications Security (CCS 2007), pp. 4–12. ACM (2007)Google Scholar
  9. 9.
    Polychronakis, M., Anagnostakis, K.G., Markatos, E.P.: Emulation-based detection of non-self-contained polymorphic shellcode. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 87–106. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  10. 10.
    Honeynet Project, Dionaea, a low-interaction honeypot (2008),
  11. 11.
    Markatos, E., Anagnostakis, K.: Noah: A european network of affined honeypots for cyber-attack tracking and alerting. The Parliament Magazine 262 (2008)Google Scholar
  12. 12.
    Baecher, P., Koetter, M.: libemu (2009),
  13. 13.
    Branco, R., Barbosa, G., Neto, P.: Scientific but not academical overview of malware anti-debugging, anti-disassembly and anti-vm technologies. In: Black Hat Technical Security Conf., Las Vegas, Nevada (2012)Google Scholar
  14. 14.
    Sikorski, M., Honig, A.: Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press (2012)Google Scholar
  15. 15.
    Ferrie, P.: Attacks on more virtual machine emulators. Symantec Technology Exchange (2007)Google Scholar
  16. 16.
    Raffetseder, T., Kruegel, C., Kirda, E.: Detecting system emulators. In: Garay, J.A., Lenstra, A.K., Mambo, M., Peralta, R. (eds.) ISC 2007. LNCS, vol. 4779, pp. 1–18. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Bania, P.: Evading network-level emulation. arXiv preprint arXiv:0906.1963 (2009)Google Scholar
  18. 18.
    Skape, Using dual-mappings to evade automated unpackers (October 2008),
  19. 19.
    Linn, C., Rajagopalan, M., Baker, S., Collberg, C., Debray, S., Hartman, J.: Protecting against unexpected system calls. In: Proc. of the 14th USENIX Security Symposium, pp. 239–254 (2005)Google Scholar
  20. 20.
    Chung, S.P., Mok, A.K.: Swarm attacks against network-level emulation/analysis. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 175–190. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  21. 21.
    0vercl0k, RP++ ROP Sequences Finder (2013),
  22. 22.
    kingcopes: Attacking the Windows 7/8 Address Space Randomization (2013),
  23. 23.
    Polychronakis, M., Keromytis, A.D.: Rop payload detection using speculative code execution. In: 2011 6th International Conference on Malicious and Unwanted Software (MALWARE), pp. 58–65. IEEE (2011)Google Scholar
  24. 24.
    Kharn: Exploring RDA (2006),
  25. 25.
    Rivest, R., Shamir, A., Wagner, D.: Time-lock puzzles and timed-release crypto. Massachusetts Institute of Technology, Tech. Rep. (1996)Google Scholar
  26. 26.
    Nomenumbra: Countering behavior based malware analysis (2009),
  27. 27.
    Glynos, D.: Context-keyed Payload Encoding: Fighting the Next Generation of IDS. In: Proc. of Athens IT Security Conference, ATH.C0N 2010 (2010)Google Scholar
  28. 28.
    Aycock, J., de Graaf, R., Jacobson Jr., M.: Anti-disassembly using cryptographic hash functions. Journal in Computer Virology 2(1), 79–85 (2006)CrossRefGoogle Scholar
  29. 29.
    Davi, L., Sadeghi, A., Winandy, M.: ROPdefender: A detection tool to defend against return-oriented programming attacks. In: Proc. of the 6th ACM Symposium on Information, Computer and Communications Security (ASIACCS 2011), pp. 40–51. ACM (2011)Google Scholar
  30. 30.
    Chen, P., Xiao, H., Shen, X., Yin, X., Mao, B., Xie, L.: DROP: Detecting return-oriented programming malicious code. In: Prakash, A., Sen Gupta, I. (eds.) ICISS 2009. LNCS, vol. 5905, pp. 163–177. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  31. 31.
    Onarlioglu, K., Bilge, L., Lanzi, A., Balzarotti, D., Kirda, E.: G-Free: Defeating return-oriented programming through gadget-less binaries. In: Proc. of the 26th Annual Computer Security Applications Conference (ACSAC 2010), pp. 49–58. ACM (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ali Abbasi
    • 1
  • Jos Wetzels
    • 1
    • 2
  • Wouter Bokslag
    • 2
  • Emmanuele Zambon
    • 1
    • 3
  • Sandro Etalle
    • 1
    • 2
  1. 1.Services, Cyber security and Safety GroupUniversity of TwenteThe Netherlands
  2. 2.Eindhoven University of TechnologyThe Netherlands
  3. 3.SecurityMatters BVThe Netherlands

Personalised recommendations