Network–Level Polymorphic Shellcode Detection Using Emulation

  • Michalis Polychronakis
  • Kostas G. Anagnostakis
  • Evangelos P. Markatos
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4064)


As state–of–the–art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to evade detection. Although recent results have been promising, most existing proposals can be defeated using only minor enhancements to the attack vector. We present a heuristic detection method that scans network traffic streams for the presence of polymorphic shellcode. Our approach relies on a NIDS–embedded CPU emulator that executes every potential instruction sequence, aiming to identify the execution behavior of polymorphic shellcodes. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem.


Memory Location Input Buffer Control Flow Graph Network Intrusion Detection System USENIX Security Symposium 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    sk: History and advances in windows shellcode. Phrack 11(62) (July 2004)Google Scholar
  2. 2.
    Kim, H.-A., Karp, B.: Autograph: Toward automated, distributed worm signature detection. In: Proceedings of the 13th USENIX Security Symposium, pp. 271–286 (2004)Google Scholar
  3. 3.
    Singh, S., Estan, C., Varghese, G., Savage, S.: Automated worm fingerprinting. In: Proceedings of the 6th Symposium on Operating Systems Design & Implementation (OSDI) (December 2004)Google Scholar
  4. 4.
    Newsome, J., Karp, B., Song, D.: Polygraph: Automatically Generating Signatures for Polymorphic Worms. In: Proceedings of the IEEE Security & Privacy Symposium, May 2005, pp. 226–241 (2005)Google Scholar
  5. 5.
    Tang, Y., Chen, S.: Defending against internet worms: a signature-based approach. In: Proceedings of the 24th Annual Joint Conference of IEEE Computer and Communication societies (INFOCOM) (2005)Google Scholar
  6. 6.
    Wang, K., Stolfo, S.J.: Anomalous Payload-Based Network Intrusion Detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Krügel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic Worm Detection Using Structural Information of Executables. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 207–226. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    Chinchani, R., van den Berg, E.: A fast static analysis approach to detect exploit code inside network flows. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 284–308. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  9. 9.
    Ször, P., Ferrie, P.: Hunting for metamorphic. In: Proceedings of the Virus Bulletin Conference, September 2001, pp. 123–144 (2001)Google Scholar
  10. 10.
    Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium (Security 2003) (August 2003)Google Scholar
  11. 11.
    Bania, P.: TAPiON (2005),
  12. 12.
    Roesch, M.: Snort: Lightweight intrusion detection for networks. In: Proceedings of USENIX LISA 1999 (November 1999), software available from:
  13. 13.
    Jordan, C.: Writing detection signatures. USENIX; login: 30(6), 55–61 (2005)Google Scholar
  14. 14.
  15. 15.
    Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack 11(61) (August 2003)Google Scholar
  16. 16.
    Rix: Writing ia32 alphanumeric shellcodes. Phrack 11(57) (August 2001)Google Scholar
  17. 17.
    Tóth, T., Krügel, C.: Accurate Buffer Overflow Detection via Abstract Payload Execution. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 274. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  18. 18.
    Akritidis, P., Markatos, E.P., Polychronakis, M., Anagnostakis, K.: STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis. In: Proceedings of the 20th IFIP International Information Security Conference (IFIP/SEC) (June 2005)Google Scholar
  19. 19.
    Crandall, J.R., Wu, S.F., Chong, F.T.: Experiences Using Minos as a Tool for Capturing and Analyzing Novel Worms for Unknown Vulnerabilities. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 32–50. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  20. 20.
    Pasupulati, A., Coit, J., Levitt, K., Wu, S., Li, S., Kuo, J., Fan, K.: Buttercup: On Network-based Detection of Polymorphic Buffer Overflow Vulnerabilities. In: Proceedings of the Network Operations and Management Symposium (NOMS), April 2004, pp. 235–248 (2004)Google Scholar
  21. 21.
    Kreibich, C., Crowcroft, J.: Honeycomb – creating intrusion detection signatures using honeypots. In: Proceedings of the Second Workshop on Hot Topics in Networks (HotNets-II) (November 2003)Google Scholar
  22. 22.
    Kolesnikov, O., Dagon, D., Lee, W.: Advanced polymorphic worms: Evading IDS by blending in with normal traffic, College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 (2004),
  23. 23.
    Payer, U., Teufl, P., Lamberger, M.: Hybrid Engine for Polymorphic Shellcode Detection. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 19–31. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM conference on Computer and communications security (CCS), pp. 290–299 (2003)Google Scholar
  25. 25.
    Aycock, J., de Graaf, R., Jacobson, M.: Anti-disassembly using cryptographic hash functions. Department of Computer Science, University of Calgary, Tech. Rep. 2005-793-24Google Scholar
  26. 26.
    Venable, M., Chouchane, M.R., Karim, M.E., Lakhotia, A.: Analyzing Memory Accesses in Obfuscated x86 Executables. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 1–18. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  27. 27.
    Collberg, C.S., Thomborson, C.: Watermarking, tamper-proffing, and obfuscation: tools for software protection. IEEE Transactions on Software Engineering 28(8), 735–746 (2002)CrossRefGoogle Scholar
  28. 28.
    Wang, C., Hill, J., Knight, J., Davidson, J.: Software tamper resistance: Obstructing static analysis of programs. University of Virginia, Tech. Rep. CS-2000-12 (2000)Google Scholar
  29. 29.
    Madou, M., Anckaert, B., Moseley, P., Debray, S., De Sutter, B., De Bosschere, K.: Software protection through dynamic code mutation. In: Song, J.-S., Kwon, T., Yung, M. (eds.) WISA 2005. LNCS, vol. 3786, pp. 194–206. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  30. 30.
    Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of the Ninth Working Conference on Reverse Engineering (WCRE) (2002)Google Scholar
  31. 31.
    Prasad, M., cker Chiueh, T.: A binary rewriting defense against stack based overflow attacks. In: Proceedings of the USENIX Annual Technical Conference (June 2003)Google Scholar
  32. 32.
    Kruegel, C., Robertson, W., Valeur, F., Vigna, G.: Static disassembly of obfuscated binaries. In: Proceedings of the USENIX Security Symposium, August 2004, pp. 255–270 (2004)Google Scholar
  33. 33.
    Cohen, F.B.: Operating system protection through program evolution. Computer and Security 12(6), 565–584 (1993)CrossRefGoogle Scholar
  34. 34.
    Metasploit Project (2006),
  35. 35.
    Cifuentes, C., Gough, K.J.: Decompilation of binary programs. Software—Practice and Experience 25(7), 811–829 (1995)CrossRefGoogle Scholar
  36. 36.
    Balakrishnan, G., Reps, T.: Analyzing Memory Accesses in x86 Executables. In: Duesterwald, E. (ed.) CC 2004. LNCS, vol. 2985, pp. 5–23. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  37. 37.
    Noir, GetPC code (was: Shellcode from ASCII) (June 2003),
  38. 38.
    Ionescu, C.: GetPC code (was: Shellcode from ASCII) (July 2003),
  39. 39.
  40. 40.
    Perriot, F., Ferrie, P., Ször, P.: Striking similarities. Virus Bulletin, 4–6 (May 2002)Google Scholar
  41. 41.
    Obscou: Building ia32 ’unicode-proof’ shellcodes. Phrack 11(61) (August 2003)Google Scholar
  42. 42.
    Tubella, J., González, A.: Control speculation in multithreaded processors through dynamic loop detection. In: Proceedings of the 4th International Symposium on High-Performance Computer Architecture (HPCA) (1998)Google Scholar
  43. 43.
    McCanne, S., Leres, C., Jacobson, V.: Libpcap (2006),
  44. 44.
    Wojtczuk, R.: Libnids (2006),
  45. 45.
  46. 46.
    Apache Chunked Encoding Overflow (2002),
  47. 47.
    Microsoft Windows RPC DCOM Interface Overflow (2003),
  48. 48.
    Microsoft Windows LSASS Remote Overflow (2004),
  49. 49.
    Bell, J.R.: Threaded code. Comm. of the ACM 16(6), 370–372 (1973)CrossRefGoogle Scholar
  50. 50.
    Bellard, F.: QEMU, a Fast and Portable Dynamic Translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005)Google Scholar
  51. 51.
    Bhatkar, S., DuVarney, D.C., Sekar, R.: Address Obfuscation: An Efficient Approach to Combat a Broad Range of Memory Error Exploits. In: Proceedings of the 12th USENIX Security Symposium (2003)Google Scholar
  52. 52.
    Anagnostakis, K., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., Keromytis, A.D.: Detecting Targeted Attacks Using Shadow Honeypots. In: Proceedings of the 14th USENIX Security Symposium, August 2005, pp. 129–144 (2005)Google Scholar
  53. 53.
    Dreger, H., Kreibich, C., Paxson, V., Sommer, R.: Enhancing the Accuracy of Network-Based Intrusion Detection with Host-Based Context. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 206–221. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Michalis Polychronakis
    • 1
  • Kostas G. Anagnostakis
    • 2
  • Evangelos P. Markatos
    • 1
  1. 1.Institute of Computer ScienceFoundation for Research & Technology – Hellas 
  2. 2.Institute for Infocomm ResearchSingapore

Personalised recommendations