Skip to main content
SpringerLink
Log in

On Emulation-Based Network Intrusion Detection Systems

  • Conference paper
Research in Attacks, Intrusions and Defenses (RAID 2014)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 8688))

Included in the following conference series:

Abstract

Emulation-based network intrusion detection systems have been devised to detect the presence of shellcode in network traffic by trying to execute (portions of) the network packet payloads in an instrumented environment and checking the execution traces for signs of shellcode activity. Emulation-based network intrusion detection systems are regarded as a significant step forward with regards to traditional signature-based systems, as they allow detecting polymorphic (i.e., encrypted) shellcode. In this paper we investigate and test the actual effectiveness of emulation-based detection and show that the detection can be circumvented by employing a wide range of evasion techniques, exploiting weakness that are present at all three levels in the detection process. We draw the conclusion that current emulation-based systems have limitations that allow attackers to craft generic shellcode encoders able to circumvent their detection mechanisms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Polychronakis, M., Anagnostakis, K.G., Markatos, E.P.: Network–Level polymorphic shellcode detection using emulation. In: Büschkes, R., Laskov, P. (eds.) DIMVA 2006. LNCS, vol. 4064, pp. 54–73. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  2. Shimamura, M., Kono, K.: Yataglass: Network-level code emulation for analyzing memory-scanning attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 68–87. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  3. Polychronakis, M., Anagnostakis, K., Markatos, E.: Comprehensive shellcode detection using runtime heuristics. In: Proc. of the 26th Annual Computer Security Applications Conference (ACSAC 2010), pp. 287–296. ACM (2010)

    Google Scholar 

  4. Snow, K., Krishnan, S., Monrose, F., Provos, N.: SHELLOS: Enabling Fast Detection and Forensic Analysis of Code Injection Attacks. In: USENIX Security Symposium (2011)

    Google Scholar 

  5. Egele, M., Wurzinger, P., Kruegel, C., Kirda, E.: Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 88–106. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  6. Gu, B., Bai, X., Yang, Z., Champion, A., Xuan, D.: Malicious shellcode detection with virtual memory snapshots. In: Proc. of IEEE INFOCOM 2010, pp. 1–9. IEEE (2010)

    Google Scholar 

  7. Portokalidis, G., Slowinska, A., Bos, H.: Argos: An emulator for fingerprinting zero-day attacks for advertised honeypots with automatic signature generation. In: Proc. of ACM SIGOPS Operating Systems Review, vol. 40(4), pp. 15–27. ACM (2006)

    Google Scholar 

  8. Zhang, Q., Reeves, D., Ning, P., Iyer, S.: Analyzing network traffic to detect self-decrypting exploit code. In: Proc. of the 2nd ACM Symposium on Information, Computer and Communications Security (CCS 2007), pp. 4–12. ACM (2007)

    Google Scholar 

  9. Polychronakis, M., Anagnostakis, K.G., Markatos, E.P.: Emulation-based detection of non-self-contained polymorphic shellcode. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 87–106. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  10. Honeynet Project, Dionaea, a low-interaction honeypot (2008), http://www.honeynet.org/project/Dionaea

  11. Markatos, E., Anagnostakis, K.: Noah: A european network of affined honeypots for cyber-attack tracking and alerting. The Parliament Magazine 262 (2008)

    Google Scholar 

  12. Baecher, P., Koetter, M.: libemu (2009), http://libemu.carnivore.it/

  13. Branco, R., Barbosa, G., Neto, P.: Scientific but not academical overview of malware anti-debugging, anti-disassembly and anti-vm technologies. In: Black Hat Technical Security Conf., Las Vegas, Nevada (2012)

    Google Scholar 

  14. Sikorski, M., Honig, A.: Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press (2012)

    Google Scholar 

  15. Ferrie, P.: Attacks on more virtual machine emulators. Symantec Technology Exchange (2007)

    Google Scholar 

  16. Raffetseder, T., Kruegel, C., Kirda, E.: Detecting system emulators. In: Garay, J.A., Lenstra, A.K., Mambo, M., Peralta, R. (eds.) ISC 2007. LNCS, vol. 4779, pp. 1–18. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  17. Bania, P.: Evading network-level emulation. arXiv preprint arXiv:0906.1963 (2009)

    Google Scholar 

  18. Skape, Using dual-mappings to evade automated unpackers (October 2008), http://www.uninformed.org/?v=10&a=1&t=sumry

  19. Linn, C., Rajagopalan, M., Baker, S., Collberg, C., Debray, S., Hartman, J.: Protecting against unexpected system calls. In: Proc. of the 14th USENIX Security Symposium, pp. 239–254 (2005)

    Google Scholar 

  20. Chung, S.P., Mok, A.K.: Swarm attacks against network-level emulation/analysis. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 175–190. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  21. 0vercl0k, RP++ ROP Sequences Finder (2013), https://github.com/0vercl0k/rp

  22. kingcopes: Attacking the Windows 7/8 Address Space Randomization (2013), http://kingcope.wordpress.com/2013/01/24/attacking-the-windows-78-address-space-randomization/

  23. Polychronakis, M., Keromytis, A.D.: Rop payload detection using speculative code execution. In: 2011 6th International Conference on Malicious and Unwanted Software (MALWARE), pp. 58–65. IEEE (2011)

    Google Scholar 

  24. Kharn: Exploring RDA (2006), http://www.awarenetwork.org/etc/alpha/?x=3

  25. Rivest, R., Shamir, A., Wagner, D.: Time-lock puzzles and timed-release crypto. Massachusetts Institute of Technology, Tech. Rep. (1996)

    Google Scholar 

  26. Nomenumbra: Countering behavior based malware analysis (2009), https://har2009.org/program/track/Other/57.en.html

  27. Glynos, D.: Context-keyed Payload Encoding: Fighting the Next Generation of IDS. In: Proc. of Athens IT Security Conference, ATH.C0N 2010 (2010)

    Google Scholar 

  28. Aycock, J., de Graaf, R., Jacobson Jr., M.: Anti-disassembly using cryptographic hash functions. Journal in Computer Virology 2(1), 79–85 (2006)

    Article  Google Scholar 

  29. Davi, L., Sadeghi, A., Winandy, M.: ROPdefender: A detection tool to defend against return-oriented programming attacks. In: Proc. of the 6th ACM Symposium on Information, Computer and Communications Security (ASIACCS 2011), pp. 40–51. ACM (2011)

    Google Scholar 

  30. Chen, P., Xiao, H., Shen, X., Yin, X., Mao, B., Xie, L.: DROP: Detecting return-oriented programming malicious code. In: Prakash, A., Sen Gupta, I. (eds.) ICISS 2009. LNCS, vol. 5905, pp. 163–177. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  31. Onarlioglu, K., Bilge, L., Lanzi, A., Balzarotti, D., Kirda, E.: G-Free: Defeating return-oriented programming through gadget-less binaries. In: Proc. of the 26th Annual Computer Security Applications Conference (ACSAC 2010), pp. 49–58. ACM (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Services, Cyber security and Safety Group, University of Twente, The Netherlands

    Ali Abbasi, Jos Wetzels, Emmanuele Zambon & Sandro Etalle

  2. Eindhoven University of Technology, The Netherlands

    Jos Wetzels, Wouter Bokslag & Sandro Etalle

  3. SecurityMatters BV, The Netherlands

    Emmanuele Zambon

Authors
  1. Ali Abbasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jos Wetzels
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wouter Bokslag
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emmanuele Zambon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sandro Etalle
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Angelos Stavrou

  2. VU University Amsterdam, The Netherlands

    Herbert Bos

  3. Stevens Institute of Technology, NJ, USA

    Georgios Portokalidis

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Abbasi, A., Wetzels, J., Bokslag, W., Zambon, E., Etalle, S. (2014). On Emulation-Based Network Intrusion Detection Systems. In: Stavrou, A., Bos, H., Portokalidis, G. (eds) Research in Attacks, Intrusions and Defenses. RAID 2014. Lecture Notes in Computer Science, vol 8688. Springer, Cham. https://doi.org/10.1007/978-3-319-11379-1_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-11379-1_19

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-11378-4

  • Online ISBN: 978-3-319-11379-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation