Abstract
Current trends demonstrate an increasing use of polymorphism by attackers to disguise their exploits. The ability for malicious code to be easily, and automatically, transformed into semantically equivalent variants frustrates attempts to construct simple, easily verifiable representations for use in security sensors. In this paper, we present a quantitative analysis of the strengths and limitations of shellcode polymorphism, and describe the impact that these techniques have in the context of learning-based IDS systems. Our examination focuses on dual problems: shellcode encryption-based evasion methods and targeted “blending” attacks. Both techniques are currently being used in the wild, allowing real exploits to evade IDS sensors. This paper provides metrics to measure the effectiveness of modern polymorphic engines and provide insights into their designs. We describe methods to evade statistics-based IDS sensors and present suggestions on how to defend against them. Our experimental results illustrate that the challenge of modeling self-modifying shellcode by signature-based methods, and certain classes of statistical models, is likely an intractable problem.
Article PDF
Similar content being viewed by others
References
Abadi, M., Budiu, M., Erlingsson, U., & Ligatti, J. (2005). Control-flow integrity: principles, implementations, and applications. In Proceedings of the ACM conference on computer and communications security (CCS).
AlephOne (2001). Smashing the stack for fun and profit. Phrack, 7(49-14).
Anagnostakis, K. G., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., & Keromytis, A. D. (2005). Detecting targeted attacks using shadow honeypots. In Proceedings of the 14th USENIX security symposium.
Bania, P. (2009). Tapion polymorphic engine. http://pb.specialised.info/all/tapion/.
Baratloo, A., Singh, N., & Tsai, T. (2000). Transparent run-time defense against stack smashing attacks. In Proceedings of the USENIX annual technical conference.
Barrantes, E. G., Ackley, D. H., Forrest, S., Palmer, T. S., Stefanovic, D., & Zovi, D. D. (2003). Randomized instruction set emulation to distrupt binary code injection attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
Bhatkar, S., DuVarney, D. C., & Sekar, R. (2003). Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In Proceedings of the 12th USENIX security symposium (pp. 105–120).
Biondi, P. (2006). Shellforge project. http://www.secdev.org/projects/shellforge/.
Brumley, D., Newsome, J., Song, D., Wang, H., & Jha, S. (2006). Towards automatic generation of vulnerability-based signatures. In Proceedings of the IEEE symposium on security and privacy.
CERT (2001). Code red I/II worm. http://www.cert.org/advisories/CA-2001-19.html.
Chinchani, R., & Berg, E. V. D. (2005). A fast static analysis approach to detect exploit code inside network flows. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 284–304).
Costa, M., Crowcroft, J., Castro, M., & Rowstron, A. (2005). Vigilante: end-to-end containment of Internet worms. In Proceedings of the symposium on systems and operating systems principles (SOSP).
Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., & Zhang, Q. (1998). Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In Proceedings of the USENIX security symposium.
Crandall, J. R., Su, Z., Wu, S. F., & Chong, F. T. (2005a). On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
Crandall, J. R., Wu, S. F., & Chong, F. T. (2005b). Experiences using minos as a tool for capturing and analyzing novel worms for unknown vulnerabilities. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
Cui, W., Peinado, M., Wang, H. J., & Locasto, M. E. (2007). ShieldGen: automated data patch generation for unknown vulnerabilities with informed probing. In Proceedings of the IEEE symposium on security and privacy.
Detristan, T., Ulenspiegel, T., Malcom, Y., & von Underduk, M. S. (2003). Polymorphic shellcode engine using spectrum analysis. Phrack, 11(61-9).
Etoh, J. (2000). GCC extension for protecting applications from stack-smashing attacks. http://www.trl.ibm.com/projects/security/ssp.
Fogla, P., & Lee, W. (2006). Evading network anomaly detection systems: formal reasoning and practical techniques. In Proceedings of the 13th ACM conference on computer and communications security (CCS) (pp. 59–68). http://doi.acm.org/10.1145/1180405.1180414.
Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., & Lee, W. (2006). Polymorphic blending attacks. In Proceedings of the USENIX security conference.
Foster, J. C., Osipov, V., Bhalla, N., & Heinen, N. (2005). Buffer overflow attacks: detect, exploit, prevent. Syngress.
Joshi, A., King, S. T., Dunlap, G. W., & Chen, P. M. (2005). Detecting past and present intrusions through vulnerability-specific predicates. In Proceedings of the symposium on systems and operating systems principles (SOSP).
K2 (2003). ADMmutate documentation. http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz.
Kc, G. S., Keromytis, A. D., & Prevelakis, V. (2003). Countering code-injection attacks with instruction-set randomization. In Proceedings of the 10th ACM conference on computer and communications security (CCS) (pp. 272–280).
Kim, H. A., & Karp, B. (2004). Autograph: toward automated, distributed worm signature detection. In Proceedings of the USENIX security conference.
Kiriansky, V., Bruening, D., & Amarasinghe, S. (2002). Secure execution via program shepherding. In Proceedings of the 11th USENIX security symposium.
Kolesnikov, A., & Lee, W. (2006). Advanced polymorphic worms: evading IDS by blending in with normal traffic. In Proceedings of the USENIX security conference.
Kruegel, C., & Vigna, G. (2003). Anomaly detection of web-based attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
Krugel, C., Kirda, E., Mutz, D., Robertson, W., & Vigna, G. (2005). Polymorphic worm detection using structural information of executables. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 207–226).
Liang, Z., & Sekar, R. (2005). Fast and automated generation of attack signatures: a basis for building self-protecting servers. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
Locasto, M. E., Wang, K., Keromytis, A. D., & Stolfo, S. J. (2005). FLIPS: hybrid adaptive intrusion prevention. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 82–101).
Metasploit Development Team (2006). Metasploit project. http://www.metasploit.com.
Nethercote, N., & Seward, J. (2003). Valgrind: a program supervision framework. In Electronic notes in theoretical computer science (Vol. 89).
Newsome, J., & Song, D. (2005). Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of the 12th symposium on network and distributed system security (NDSS).
Newsome, J., Karp, B., & Song, D. (2005). Polygraph: automatically generating signatures for polymorphic worms. In Proceedings of the IEEE symposium on security and privacy.
Obscou (2003). Building IA32 ‘Unicode-Proof’ shellcodes. Phrack, 11(61-11).
Panda Labs (2007). MPack uncovered. http://pandalabs.pandasecurity.com/.
Polychronakis, M., Anagnostakis, K. G., & Markatos, E. P. (2006). Network-level polymorhpic shellcode detection using emulation. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
Rix (2001). Writing IA-32 alphanumeric shellcodes. Phrack, 11(57-15).
Russell, S., & Norvig, P. (2002). Artificial intelligence: a modern approach. New York: Prentice Hall.
SANS (2004a). IISMedia Exploit. http://www.sans.org/newsletters/cva/vol2_21.php.
SANS (2004b). Santy worm. http://isc.sans.org/diary.html?date=2004-12-21.
SANS (2004c). Webdav exploit. http://www.sans.org/resources/malwarefaq/webdav-exploit.php.
Siddharth, S. (2005). Evading NIDS. http://www.securityfocus.com/infocus/1852.
Sidiroglou, S., Giovanidis, G., & Keromytis, A. D. (2005). A dynamic mechanism for recovering from buffer overflow attacks. In Proceedings of the 8th information security conference (ISC) (pp. 1–15).
Singh, S., Estan, C., Varghese, G., & Savage, S. (2004). Automated worm fingerprinting. In Proceedings of symposium on operating systems design and implementation (OSDI).
Snort Development Team (2009). Snort project. http://www.snort.org/.
Song, Y., Locasto, M. E., Stavrou, A., Keromytis, A. D., & Stolfo, S. J. (2007). On the infeasibility of modeling polymorphic shellcode. In Proceedings of the ACM conference on computer and communications security (CCS).
Spinellis, D. (2003). Reliable identification of bounded-length viruses is NP-complete. IEEE Transactions on Information Theory, 49(1), 280–284.
Tcpdump (2009). http://www.tcpdump.org.
Toth, T., & Kruegel, C. (2002). Accurate buffer overflow detection via abstract payload execution. In Proceedings of the 5th international symposium on recent advances in intrusion detection (RAID) (pp. 274–291).
Wang, K., & Stolfo, S. J. (2004). Anomalous payload-based network intrusion detection. In Proceedings of the 7th international symposium on recent advances in intrusion detection (RAID) (pp. 203–222).
Wang, H. J., Guo, C., Simon, D. R., & Zugenmaier, A. (2004). Shield: vulnerability-driven network filters for preventing known vulnerability exploits. In Proceedings of the ACM SIGCOMM conference (pp. 193–204).
Wang, K., Cretu, G., & Stolfo, S. J. (2005). Anomalous payload-based worm detection and signature generation. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 227–246).
Wang, K., Parekh, J. J., & Stolfo, S. J. (2006a). Anagram: a content anomaly detector resistant to mimicry attack. In Proceedings of the 9th international symposium on recent advances in intrusion detection (RAID).
Wang, X., Pan, C. C., Liu, P., & Zhu, S. (2006b). SigFree: a signature-free buffer overflow attack blocker. In Proceedings of the 15th USENIX security symposium (pp. 225–240).
Yegneswaran, V., Giffin, J. T., Barford, P., & Jha, S. (2005). An architecture for generating semantics-aware signatures. In Proceedings of the 14th USENIX security symposium.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editors: Pavel Laskov and Richard Lippmann.
This material is based on research sponsored by the Air Force Research Laboratory under agreement number FA8750-06-2-0221. Army Research Office contract number W911NF0610151, and by NSF Grant 06-27473, with additional support from Google.
Rights and permissions
About this article
Cite this article
Song, Y., Locasto, M.E., Stavrou, A. et al. On the infeasibility of modeling polymorphic shellcode. Mach Learn 81, 179–205 (2010). https://doi.org/10.1007/s10994-009-5143-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-009-5143-5