Abstract
It is a well-known fact that polymorphism is one of the greatest find of malicious code authors. Applied in the context of Buffer Overflow attacks, the detection of such codes becomes very difficult. In view of this problematic, which constitutes a real challenge for all the international community, we propose in this paper a new formal language (based on temporal logics such as CTL) allowing to specify polymorphic codes, to detect them and to better understand their nature. The efficiency and the expressiveness of this language are shown via the specification of a variety of properties characterizing polymorphic shellcodes. Finally, to make the verification process automatic, this language is supported by a new IDS (Intrusion Detection System) that will also be presented in this paper.
Similar content being viewed by others
References
CAN-2002-0392 - apache chunked-encoding memory corruption vulnerability. http://www.securityfocus.com/bid/5033/discuss
Flawfinder. http://www.dwheeler.com/flawfinder
IA-32 intel architecture software developer’s manual-instruction set reference. http://www.intel.com/design/pentium4/manuals/index_new.htm
The lex & yacc page. http://dinosaur.compilertools.net/
Metasploit. http://www.metasploit.com/
MIT lincoln laboratory. http://www.ll.mit.edu/
National vulnerability database. http://nvd.nist.gov/statistics.cfm
Nessus. http://www.nessus.org
Openwall. http://www.openwall.com/
Retina. http://www.eeye.com
Stack Shield. http://www.angelfire.com/sk/stackshield/
US-CERT. http://www.us-cert.gov/
Adi K., Debbabi M., Mejri M.: A new logic for electronic commerce protocols. Theor. Comput. Sci. 291(3), 223–283 (2003)
Akritidis, P., Evangelos, P., Markatos, Polychronakis, M., Kostas, G., Anagnostakis: STRIDE: Polymorphic sled detection through instruction sequence analysis. In: SEC, pp. 375–392 (2005)
Aleph1. Smashing the stack for fun and profit. http://www.phrack.org/issues.html?issue=49&id=14
Bailleux, C., Grenie, C.: Protections contre l’exploitation des débordements de buffer - bibliothèques et compilateurs. http://www.miscmag.com/
Baratloo, A., Singh, N., Tsai, T.: Libsafe: Protecting critical elements of stacks
Beaucamps P., Filiol E.: On the possibility of practically obfuscating programs towards a unified perspective of code protection. J. Comput. Virol. 3(1), 3–21 (2007)
Bulba and Kil3r. Bypassing Stackguard and Stackshield. http://www.phrack.org/issues.html?issue=56&id=5
Cowan, C., Pu, C., Maier, D., Hintony, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In: SSYM. USENIX Association (1998)
Solar Designer. Getting around non-executable stack (and fix). http://www.securityfocus.com/archive/1/7480
Detristan, T., Ulenspiegel, T., Malcom, Y., Superbus M., Von Underduk.: Polymorphic shellcode engine using spectrum analysis. http://www.phrack.org/issues.html?issue=61&id=9
Filiol E.: Metamorphism, formal grammars and undecidable code mutation. Int. J. Comput. Sci. 2(1), 70–75 (2007)
Ben Ghorbel, M., Talbi M., Mejri, M.: Specification and detection of TCP/IP based attacks using the ADM-logic. In: ARES, pp. 206–212. IEEE Computer Society (2007)
Gushin, Y.: Nids polymorphic evasion - the end? http://www.ecl-labs.org/papers.html
K2. Admmutate. http://www.ktwo.ca/
Kolesnikov, O., Lee, W.: Advanced polymorphic worms: Evading IDS by blending in with normal traffic (2004)
Kripke S.A.: Semantical considerations in modal logic. Acta Philosophica Fenica 16, 83–94 (1963)
Krügel, C., Kirda, E., Mutz, D., Robertson, W.K., Vigna, G.: Polymorphic worm detection using structural information of executables. In: RAID, pp. 207–226 (2005)
Lespérance, P.L.: Detecting variants of known attacks using temporal logic. In: WPTACT (2005)
Lions, J.L.: ARIANE 5: Flight 501 failure. http://sunnyday.mit.edu/accidents/Ariane5accidentreport.html
McHugh J.: Testing intrusion detection systems: a critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory. ACM Trans. Inform. Syst. Security 3(4), 262–294 (2000)
Newsome, J., Karp, B., Xiaodong Song, D.: Polygraph: automatically generating signatures for polymorphic worms. In: IEEE Symposium on Security and Privacy, pp. 226–241 (2005)
Payer, U., Kraxberger, S.: Polymorphic code detection with GA optimized markov models. In: Communications and Multimedia Security, pp. 210–219 (2005)
Payer, U., Teufl, P., Lamberger, M.: Hybrid engine for polymorphic shellcode detection. In: Julisch, K., Krügel, C. (eds.) DIMVA, vol. 3548. Lecture Notes in Computer Science, pp. 19–31. Springer, Berlin (2005)
Plotkin, G.D.: A structural approach to operational semantics. Technical Report DAIMI FN-19, University of Aarhus (1981)
Writing, R.: IA32 alphanumeric shellcodes. http://www.phrack.org/issues.html?issue=57&id=15
Ruiu, D.: Snort preprocessor—multi-architecture mutated NOP sled detector
Sedalo, M.: JempiScode. http://goodfellas.shellcode.com.ar/proyectos.html
Stirling, C.: Modal and temporal logics for processes. In: Proceedings of the VIII Banff Higher order workshop conference on Logics for concurrency : structure versus automata, pp. 149–237. Springer, Berlin (1996)
Talbi, M.: IDS-logic. http://www.rennes.supelec.fr/ren/perso/mtalbi/outils/IDS-Logic.tar.gz
Toth, T., Krügel, C.: Accurate buffer overflow detection via abstract payload execution. In: RAID, pp. 274–291 (2002)
Wojtczuk, R.: The advanced return-into-lib(c) exploits: PAX case study. http://www.phrack.org/issues.html?issue=58&id=4
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Talbi, M., Mejri, M. & Bouhoula, A. Specification and evaluation of polymorphic shellcode properties using a new temporal logic. J Comput Virol 5, 171–186 (2009). https://doi.org/10.1007/s11416-008-0089-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11416-008-0089-x