Pushdown model checking for malware detection

  • Fu SongEmail author
  • Tayssir Touili
TACAS 2012


The number of malware is growing extraordinarily fast. Therefore, it is important to have efficient malware detectors. Malware writers try to obfuscate their code by different techniques. Many well-known obfuscation techniques rely on operations on the stack such as inserting dead code by adding useless push and pop instructions, or hiding calls to the operating system, etc. Thus, it is important for malware detectors to be able to deal with the program’s stack. In this study, we propose a new model-checking approach for malware detection that takes into account the behavior of the stack. Our approach consists in: (1) Modeling the program using a pushdown system (PDS). (2) Introducing a new logic, called stack computation tree predicate logic (SCTPL), to represent the malicious behavior. SCTPL can be seen as an extension of the branching-time temporal logic CTL with variables, quantifiers, and predicates over the stack. (3) Reducing the malware detection problem to the model-checking problem of PDSs against SCTPL formulas. We show how our new logic can be used to precisely express malicious behaviors that could not be specified by existing specification formalisms. We then consider the model-checking problem of PDSs against SCTPL specifications. We reduce this problem to emptiness checking in Symbolic Alternating Büchi Pushdown Systems, and we provide an algorithm to solve this problem. We implemented our techniques in a tool and applied it to detect several viruses. Our results are encouraging.


Pushdown Systems Model Checking CTL Malware Detection 


  1. 1.
    Avast. Free avast antivirus. Version 6.0.1367
  2. 2.
    Avira. Version
  3. 3.
    Balakrishnan, G., Gruian, R., Reps, T.W., Teitelbaum, T.: CodeSurfer/x86-a platform for analyzing x86 executables. In: CC, pp. 250–254 (2005)Google Scholar
  4. 4.
    Balakrishnan, G., Reps, T.W., Kidd, N., Lal, A., Lim, J., Melski, D., Gruian, R., Yong, S.H., Chen, C.-H., Teitelbaum, T.: Model checking x86 executables with CodeSurfer/x86 and WPDS++. In: CAV, pp. 158–163 (2005)Google Scholar
  5. 5.
    Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: Symposium on Requirements Engineering for Information Security, pp. 1–8 (2001)Google Scholar
  6. 6.
    Bonfante, G., Kaczmarek, M., Marion, J.-Y.: Architecture of a morphological malware detector. J. Comput. Virol. 5, 263–270 (2009)CrossRefGoogle Scholar
  7. 7.
    Bouajjani, A., Esparza, J., Maler, O.: Reachability analysis of pushdown automata: application to model checking. In: CONCUR’97. LNCS 1243 (1997)Google Scholar
  8. 8.
    Brumley, D., Jager, I., Avgerinos, T., Schwartz, E.J.: BAP: A binary analysis platform. In: Computer Aided Verification (2011)Google Scholar
  9. 9.
    Bryant, R.E.: Symbolic boolean manipulation with ordered binary-decision diagrams. ACM Comput. Surv. 24(3), 293–318 (1992)CrossRefGoogle Scholar
  10. 10.
    Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: 12th USENIX Security, Symposium, pp. 169–186 (2003)Google Scholar
  11. 11.
    Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: ISEC, pp. 5–14 (2008)Google Scholar
  12. 12.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D.X., Bryant, R.E.: Semantics-aware malware detection. In: IEEE Symposium on Security and Privacy, pp. 32–46 (2005)Google Scholar
  13. 13.
    Eric, S.: 10 most destructive computer worms and viruses ever. (2010)
  14. 14.
    Esparza, J., Kucera, A., Schwoon, S.: Model-checking LTL with regular valuations for pushdown systems. In: TACS, pp. 316–339 (2001) Google Scholar
  15. 15.
    Esparza, J., Kucera, A., Schwoon, S.: Model checking LTL with regular valuations for pushdown systems. Inf. Comput. 186(2), 355–376 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Esparza, J., Schwoon, S.: A BDD-based model checker for recursive programs. In: CAV’01, pp. 324–336 (2001)Google Scholar
  17. 17.
    Gostev, A.: Kaspersky security bulletin, malware evolution 2010. Kaspersky Lab ZAO (2011)
  18. 18.
  19. 19.
    Hex-Rays. IDAPro (2011)Google Scholar
  20. 20.
    Holzer, A., Kinder, J., Veith, H.: Using verification technology to specify and detect malware. In: EUROCAST, pp. 497–504 (2007)Google Scholar
  21. 21.
    Kinder, J., Katzenbeisser, S., Schallhart, C., Veith, H.: Detecting malicious code by model checking. In: DIMVA, pp. 174–187 (2005)Google Scholar
  22. 22.
    Kinder, J., Katzenbeisser, S., Schallhart, C., Veith, H.: Proactive detection of computer worms using model checking. IEEE Trans. Dependable Secure Comput. 7(4), 424–438 (2010)CrossRefGoogle Scholar
  23. 23.
    Kinder, J., Veith, H.: Jakstab: a static analysis platform for binaries. In: CAV, pp. 423–427 (2008)Google Scholar
  24. 24.
    Lakhotia, A., Boccardo, D.R., Singh, A., Manacero, A.: Context-sensitive analysis of obfuscated x86 executables. In: PEPM, pp. 131–140 (2010)Google Scholar
  25. 25.
    Lakhotia, A., Kumar, E.U., Venable, M.: A method for detecting obfuscated calls in malicious binaries. IEEE Trans. Softw. Eng. 31(11), 955–968 (2005)CrossRefGoogle Scholar
  26. 26.
    Qihoo 360.
  27. 27.
    Singh, P.K., Lakhotia, A.: Static verification of worm and virus behavior in binary executables using model checking. In: IAW, pp. 298–300 (2003)Google Scholar
  28. 28.
    Song, F., Touili, T.: Efficient CTL model-checking for pushdown systems. In: CONCUR (2011)Google Scholar
  29. 29.
    Song, F., Touili, T.: Pushdown model checking for malware detection. In: TACAS, pp. 110–125 (2012)Google Scholar
  30. 30.
    Suwimonteerabuth, D., Schwoon, S., Esparza, J.: Efficient algorithms for alternating pushdown systems with an application to the computation of certificate chains. In: ATVA, pp. 141–153 (2006)Google Scholar
  31. 31.
    Uezato, Y., Minamide, Y.: Pushdown systems with stack manipulation. In: ATVA’13 (2013) (to appear)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Shanghai Key Laboratory of Trustworthy ComputingEast China Normal UniversityShanghaiPeople’s Republic of China
  2. 2.Liafa, CNRS and Université Paris DiderotParis Cedex 13 France

Personalised recommendations