Detecting Malicious Code by Model Checking
Abstract
The ease of compiling malicious code from source code in higher programming languages has increased the volatility of malicious programs: The first appearance of a new worm in the wild is usually followed by modified versions in quick succession. As demonstrated by Christodorescu and Jha, however, classical detection software relies on static patterns, and is easily outsmarted. In this paper, we present a flexible method to detect malicious code patterns in executables by model checking. While model checking was originally developed to verify the correctness of systems against specifications, we argue that it lends itself equally well to the specification of malicious code patterns. To this end, we introduce the specification language CTPL (Computation Tree Predicate Logic) which extends the well-known logic CTL, and describe an efficient model checking algorithm. Our practical experiments demonstrate that we are able to detect a large number of worm variants with a single specification.
Keywords
Model Checking Malware DetectionPreview
Unable to display preview. Download preview PDF.
References
- 1.Norman ASA. Norman sandbox whitepaper. Technical report (2003)Google Scholar
- 2.Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M.M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: Symposium on Requirements Engineering for Information Security (March 2001)Google Scholar
- 3.Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium (Security 2003), August 2003, pp. 169–186. USENIX Association (2003)Google Scholar
- 4.Christodorescu, M., Jha, S.: Testing malware detectors. In: Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 2004 (2004)Google Scholar
- 5.Clarke, E., Emerson, E.: Design and synthesis of synchronization skeletons using branching time temporal logic. In: Kozen, D. (ed.) Logic of Programs 1981. LNCS, vol. 131, pp. 52–71. Springer, Heidelberg (1982)CrossRefGoogle Scholar
- 6.Clarke, E., Grumberg, O., Long, D.: Model Checking. MIT Press, Cambridge (1999)Google Scholar
- 7.Clarke, E., Schlingloff, B.: Model Checking. In: Handbook of Automated Reasoning, pp. 1637–1790. Elsevier, Amsterdam (2001)Google Scholar
- 8.Emerson, E.: Temporal and Modal Logic. In: Handbook of Theoretical Computer Science, vol. B, pp. 995–1072. Elsevier, Amsterdam (1990)Google Scholar
- 9.Fast Small Good, http://www.xtreeme.prv.pl (Last accessed: December16, 2004)
- 10.Huth, M., Ryan, M.: Logic in Computer Science: Modelling and Reasoning about Systems. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
- 11.IDA Pro. http://www.datarescue.com/idabase/ (Last accessed: January 20, 2004)
- 12.IKARUS Software, http://www.ikarus-software.at/ (Last accessed: January 20, 2004)
- 13.Lakhotia, A., Singh, P.: Challenges in getting ’formal’ with viruses. Virus Bulletin (September 2003)Google Scholar
- 14.Singh, P., Lakhotia, A.: Static Verification of Worm and Virus Behavior in Binary Executables using Model Checking. In: 4th IEEE Information Assurance Workshop (June 2003)Google Scholar
- 15.Ultimate Packer for eXecutables, http://upx.sourceforge.net/ (Last accessed: December 16, 2004)