Skip to main content

Advertisement

SpringerLink
Log in
Menu
Find a journal Publish with us
Search
Cart
Book cover

International Conference on Tools and Algorithms for the Construction and Analysis of Systems

TACAS 2012: Tools and Algorithms for the Construction and Analysis of Systems pp 110–125Cite as

  1. Home
  2. Tools and Algorithms for the Construction and Analysis of Systems
  3. Conference paper
Pushdown Model Checking for Malware Detection

Pushdown Model Checking for Malware Detection

  • Fu Song18 &
  • Tayssir Touili18 
  • Conference paper
  • 2028 Accesses

  • 24 Citations

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 7214)

Abstract

The number of malware is growing extraordinarily fast. Therefore, it is important to have efficient malware detectors. Malware writers try to obfuscate their code by different techniques. Many of these well-known obfuscation techniques rely on operations on the stack such as inserting dead code by adding useless push and pop instructions, or hiding calls to the operating system, etc. Thus, it is important for malware detectors to be able to deal with the program’s stack. In this paper we propose a new model-checking approach for malware detection that takes into account the behavior of the stack. Our approach consists in : (1) Modeling the program using a Pushdown System (PDS). (2) Introducing a new logic, called SCTPL, to represent the malicious behavior. SCTPL can be seen as an extension of the branching-time temporal logic CTL with variables, quantifiers, and predicates over the stack. (3) Reducing the malware detection problem to the model-checking problem of PDSs against SCTPL formulas. We show how our new logic can be used to precisely express malicious behaviors that could not be specified by existing specification formalisms. We then consider the model-checking problem of PDSs against SCTPL specifications. We reduce this problem to emptiness checking in Symbolic Alternating Büchi Pushdown Systems, and we provide an algorithm to solve this problem. We implemented our techniques in a tool, and we applied it to detect several viruses. Our results are encouraging.

Keywords

  • Model Check
  • Malicious Code
  • Malicious Behavior
  • Obfuscation Technique
  • Benign Program

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Work partially funded by ANR grant ANR-08-SEGI-006.

Download conference paper PDF

References

  1. Avast antivirus, free version, http://www.avast.com

  2. Avira antivirus, free version, http://www.avira.com

  3. IDA Pro, http://www.hex-rays.com/idapro/

  4. Qihoo 360 antivirus, http://www.360.cn

  5. Balakrishnan, G., Reps, T., Kidd, N., Lal, A., Lim, J., Melski, D., Gruian, R., Yong, S., Chen, C.-H., Teitelbaum, T.: Model Checking x86 Executables with CodeSurfer/x86 and WPDS++. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 158–163. Springer, Heidelberg (2005)

    CrossRef  Google Scholar 

  6. Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: SREIS (2001)

    Google Scholar 

  7. Bonfante, G., Kaczmarek, M., Marion, J.-Y.: Architecture of a Morphological Malware Detector. Journal in Computer Virology 5, 263–270 (2009)

    CrossRef  Google Scholar 

  8. Bryant, R.E.: Symbolic boolean manipulation with ordered binary-decision diagrams. ACM Comput. Surv. 24(3) (1992)

    Google Scholar 

  9. Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: 12th USENIX Security Symposium (2003)

    Google Scholar 

  10. Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: ISEC (2008)

    Google Scholar 

  11. Christodorescu, M., Jha, S., Seshia, S.A., Song, D.X., Bryant, R.E.: Semantics-aware malware detection. In: IEEE Symposium on Security and Privacy (2005)

    Google Scholar 

  12. Esparza, J., Kucera, A., Schwoon, S.: Model checking LTL with regular valuations for pushdown systems. Inf. Comput. 186(2) (2003)

    Google Scholar 

  13. Esparza, J., Schwoon, S.: A BDD-Based Model Checker for Recursive Programs. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 324–336. Springer, Heidelberg (2001)

    CrossRef  Google Scholar 

  14. Heavens, V.: http://vx.netlux.org

  15. Holzer, A., Kinder, J., Veith, H.: Using Verification Technology to Specify and Detect Malware. In: Moreno Díaz, R., Pichler, F., Quesada Arencibia, A. (eds.) EUROCAST 2007. LNCS, vol. 4739, pp. 497–504. Springer, Heidelberg (2007)

    CrossRef  Google Scholar 

  16. Kinder, J., Katzenbeisser, S., Schallhart, C., Veith, H.: Detecting Malicious Code by Model Checking. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 174–187. Springer, Heidelberg (2005)

    CrossRef  Google Scholar 

  17. Kinder, J., Katzenbeisser, S., Schallhart, C., Veith, H.: Proactive detection of computer worms using model checking. IEEE Transactions on Dependable and Secure Computing 7(4) (2010)

    Google Scholar 

  18. Lakhotia, A., Boccardo, D.R., Singh, A., Manacero, A.: Context-sensitive analysis of obfuscated x86 executables. In: PEPM (2010)

    Google Scholar 

  19. Lakhotia, A., Kumar, E.U., Venable, M.: A method for detecting obfuscated calls in malicious binaries. IEEE Trans. Software Eng. 31(11) (2005)

    Google Scholar 

  20. Singh, P.K., Lakhotia, A.: Static verification of worm and virus behavior in binary executables using model checking. In: IAW (2003)

    Google Scholar 

  21. Song, F., Touili, T.: Efficient CTL Model-Checking for Pushdown Systems. In: Katoen, J.-P., König, B. (eds.) CONCUR 2011. LNCS, vol. 6901, pp. 434–449. Springer, Heidelberg (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Liafa, CNRS and Univ. Paris Diderot, France

    Fu Song & Tayssir Touili

Authors
  1. Fu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tayssir Touili
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of California at Santa Cruz, 1156 High Street, 95064, Santa Cruz, CA, USA

    Cormac Flanagan

  2. Fakultät für Ingenieurwesen, Abteilung für Informatik und Angewandte Kognitionswissenschaft, Universität Duisburg-Essen, Lotharstraße 65, 47057, Duisburg, Germany

    Barbara König

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Song, F., Touili, T. (2012). Pushdown Model Checking for Malware Detection. In: Flanagan, C., König, B. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2012. Lecture Notes in Computer Science, vol 7214. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28756-5_9

Download citation

  • .RIS
  • .ENW
  • .BIB
  • DOI: https://doi.org/10.1007/978-3-642-28756-5_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-28755-8

  • Online ISBN: 978-3-642-28756-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Search

Navigation

  • Find a journal
  • Publish with us

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support

167.114.118.210

Not affiliated

Springer Nature

© 2023 Springer Nature