Skip to main content
SpringerLink
Log in

Behavioral detection of malware: from a survey towards an established taxonomy

  • Original Paper
  • Published:
Journal in Computer Virology Aims and scope Submit manuscript

Abstract

Behavioral detection differs from appearance detection in that it identifies the actions performed by the malware rather than syntactic markers. Identifying these malicious actions and interpreting their final purpose is a complex reasoning process. This paper draws up a survey of the different reasoning techniques deployed among the behavioral detectors. These detectors have been classified according to a new taxonomy introduced inside the paper. Strongly inspired from the domain of program testing, this taxonomy divides the behavioral detectors into two main families: simulation-based and formal detectors. Inside these families, ramifications are then derived according to the data collection mechanisms the data interpretation, the adopted model and its generation, and the decision support.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cohen, F.: Computer viruses. Ph.D. thesis, University of South California (1986)

  2. Cohen F.B. (1987). Computer viruses: Theory and experiments. Comput. Secur. 6(1): 22–35

    Article  Google Scholar 

  3. Debar H., Dacier M. and Wespi A. (1999). Towards a taxonomy of intrusion-detection systems. Comput. Netw. Spl Issue Comput. Netw. Secur. 31(9): 805–822

    Google Scholar 

  4. Mé, L., Morin, B.: Intrusion detection and virology: an analysis of differences, similarities and complementariness. In: Bonfante, G., Marion, J.-Y. (eds.) J. Comput. Virol., vol. 3, no. 1, WTCV’06 Special Issue, pp. 39–49 (2007)

  5. Anderson, J.: Computer security threat monitoring and surveillance. Tech. rep., James P. Anderson Company (1980)

  6. Denning, D.: An intrusion–detection model. IEEE Trans. Softw. Eng., vol. SE-13 (1987)

  7. Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusion using system calls: Alternative data models, In: Proceedings of IEEE Symposium on Security and Privacy, pp. 133–145 (1999)

  8. Zanero, S.: Behavioral intrusion detection. In: Proceedings of the 19th International Symposium on Computer and Information Sciences (ISCIS), pp. 657–666 (2004)

  9. Filiol, E.: Computer viruses: from theory to applications. Springer, Heidelberg, IRIS Collection (2005). ISBN:2-287-23939-1

  10. Fortinet observatory. http://www.fortinet.com/FortiGuardCenter/

  11. Malware outbreak trend report: Storm-worm, Commtouch Software Ltd (2007). http://www.commtouch.com/downloads/Storm-Worm_MOTR.pdf

  12. Filiol, E.: Malware pattern scanning schemes secure against black-box analysis. In: Broucek, V., Turner, P. (eds.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 35–50 (2006)

  13. Filiol, E. (2007). Techniques Virales Avancées. Springer, Heidelberg, IRIS Collection. ISBN:2-287-33887-8

    Google Scholar 

  14. Ször, P.: The Art of Computer Virus Research and Defense. Addison-Wesley, Reading (2005). ISBN:0-321-30454-3

    Google Scholar 

  15. Spinellis D. (2003). Reliable identification of boundedlength viruses is np-complete. IEEE Trans. Inf. Theory 49: 280–284

    Article  MATH  MathSciNet  Google Scholar 

  16. Filiol, E.: Metamorphism, formal grammars and undecidable code mutation. In: Proceedings of the International Conference on Computational Intelligence (ICCI), Published in the Int. J. Comput. Sci., vol. 2, issue 1, pp. 70–75 (2007)

  17. Christodorescu, M., Jha, S.: Testing malware detectors, In: Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 34–44, ACM Press, New York (2004)

  18. Josse, S.: How to assess the effectiveness of your anti-virus? In: Broucek, V. (ed.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 51–65 (2006)

  19. Filiol, E., Jacob, G., Liard, M.L.: Evaluation methodology and theoretical model for antiviral behavioural detection strategies. In: Bonfante, G., Marion, J.-Y. (eds.) J. Comput. Virol., vol. 3, no. 1, WTCV’06 Special Issue, pp. 23–37 (2007)

  20. Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Proceedings of the European Symposium on Research in Computer Security, pp. 326–343 (2003)

  21. Hoglund, G., Butler, J.: Rootkits, Subverting the Windows Kernel. Addison-Wesley Professional, Reading (2006). ISBN: 0-321-29431-9

  22. Vivanco, A.D.: Comprehensive non-intrusive protection with data-restoration: A proactive approach against malicious mobile code. Master’s thesis, Florida Institute of Technology (2002)

  23. Wagner, M.E.: Behavior oriented detection of malicious code at run-time. Master’s thesis, Florida Institute of Technology (2004)

  24. Norman’s sandbox malware analyzer. Norman ASA. http://www.norman.com/microsites/malwareanalyzer/fr/

  25. Cwsandbox. Sunbelt Software. http://www.cwsandbox.org

  26. Bayer, U., Moser, A., Kruegel, C., Kirda, E.: Dynamic analysis of malicious code. In: Broucek, V., Turner, P., (eds.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 67–77 (2006)

  27. Rutkowska, J.: Red pill... or how to detect vmm using (almost) one cpu instruction (2005). http://invisiblethings.org/papers/redpill.html

  28. Ferrie, P.: Attacks on virtual machine emulators. In: Proceedings of the AVAR Conference (2006)

  29. Debbabi, M.: Dynamic monitoring of malicious activity in software systems. In: Proceedings of the Symposium on Requirements Engineering for Information Security (SREIS) (2001)

  30. Nachenberg, C.: Behavior blocking: The next step in anti-virus protection, SecurityFocus, 2002. http://www.securityfocus.com/infocus/1557

  31. Schmall, M.: Classification and identification of malicious code based on heuristic techniques utilizing meta-languages. Ph.D. thesis, University of Hamburg (2002)

  32. Schmall, M.: Heuristic techniques in av solutions: An overview, SecurityFocus (2002). http://www.securityfocus.com/infocus/1542

  33. Veldman, F.: Heuristic anti-virus technology. In: Proceedings of the International Virus Protection and Information Security Council (1994)

  34. Zwienenberg, R.: Heuristics scanners: Artificial intelligence? In: Proceedings of the Virus Bulletin Conference, pp. 203–210 (1994)

  35. Understanding heuristics: Symantec bloodhound technology. Tech. rep., Symantec White Paper Series, vol. XXXIV (1997)

  36. Glover, F.W., Kochenberger, G.A.: Handbook of Metaheuristics. Springer, Heidelberg (2003). ISBN:1-402-07263-5

  37. Charlier, B.L., Mounji, A., Swimmer, M.: Dynamic detection and classification of computer viruses using general behaviour patterns. In: Proceedings of the Virus Bulletin Conference (1995)

  38. Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based approach for detecting anomalous program behaviors. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 144–155 (2001)

  39. Hopcroft, J., Motwani, R., Ullman, J.: Introduction to Automata Theory, Languages and Computation, 2nd edn. Addison Wesley, Reading (1995). ISBN:0-201-44124-1

  40. Mazeroff, G., Cerqueira, V.D., Gregor, J., Thomason, M.G.: Probabilistic trees and automata for application behavior modeling. In: Proceedings of the 43rd ACM Southeast Conference (2003)

  41. Kaspersky, K.: Hacker Disassembling Uncovered, 2nd edn. A-LIST, LLC (2007). ISBN:1-931-76964-8

  42. Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations. Tech. rep., Technical Report 148, Department of Computer Science, University of Auckland (1997)

  43. Kruegel, C., Robertson, W., Valeur, F., Vigna, G.: Static disassembly of obfuscated binaries. In: SSYM’04: Proceedings of the 13th conference on USENIX Security Symposium, pp. 18–18 (2004)

  44. Josse S. (2007). Secure and advanced unpacking using computer emulation, extended version from the avar conference. J. Comput. Virol. 3(3): 221–236

    Article  Google Scholar 

  45. Preda, M.D., Christodorescu, M., Jha, S., Debray, S.: A semantic-based approach to malware detection. In: Proceedings of the 34th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) (2007)

  46. Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantic-aware malware detection. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 32–46 (2005)

  47. Bruschi, D., Martignoni, L., Monga, M.: Detecting self-mutating malware using control-flow graph matching. In: Proceedings of the Conference on the Detection of Intrusions and Malwares and Vulnerability Assessment (DIMVA), pp. 129–143 (2006)

  48. Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic worm detection using structural information of executables. In: International Symposium on Recent Advances in Intrusion Detection (RAID) (2005)

  49. Periot, F.: Defeating polymorphism through code optimization. In: Proceedings of the Virus Bulletin Conference, pp. 142–159 (2003)

  50. Bruschi, D., Martignoni, L., Monga, M.: Using code normalization for fighting self-mutating malware. In: Proceedings of the International Symposium on Secure Software Engineering, pp. 37–44, IEEE CS Press (2006)

  51. Webster, M.: Algebraic specification of computer viruses and their environments. In: Selected Papers from the First Conference on Algebra and Coalgebra in Computer Science Young Researchers Workshop (CALCO-jnr 2005), University of Wales Swansea Computer Science Report Series (CSR 18-2005), pp. 99–113 (2005)

  52. Webster M. and Malcolm G. (2006). Detection of metamorphic computer viruses using algebraic specification. J. Comput. Virol. 2(3): 149–161

    Article  Google Scholar 

  53. Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M.M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: Proceedings of the Symposium on Requirements Engineering for Information Security (SREIS) (2001)

  54. Singh, P., Lakhotia, A.: Static verification of worm and virus behavior in binary executables using model checking. In: Proceedings of the IEEE Information Assurance Workshop, pp. 298–300 (2003)

  55. Clark, E., Grumberg, O., Long, D.: Model Checking. MIT Press, Cambridge (1999). ISBN:0-262-03270-8

    Google Scholar 

  56. Schnoebelen P. (2003). The complexity of temporal logic model checking. Adv. Modal Logic 4: 393–436

    MathSciNet  Google Scholar 

  57. Kinder J., Katzenbeisser S., Schallhart C. and Veith H. (2005). Detecting malicious code by model checking. Lect. Notes Computer Sci. 3548: 174–187

    Article  Google Scholar 

  58. Perdisci, R., Dagon, D., Fogla, P.W.L., Sharif, M.: Misleading worm signature generators using deliberate noise injection. In: Proceedings of IEEE Symposium on Security and Privacy (2006)

  59. Lee, W., Stolfo, S., Chan, P.: Learning patterns from unix process execution traces for intrusion detection. In: Proceedings of the AAAI97 Workshop on AI Approaches to Fraud Detection and Risk Management, pp. 50–56. Addison Wesley, Reading (1997)

  60. Schultz, M.G., Eskin, E., Zadok, E.: Data mining methods for detection of new malicious executables. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 38–49 (2001)

  61. Wang, J.-H., Deng, P.S., Fan, Y.-S., Jaw, L.-J., Liu, Y.-C.: Virus detection using data mining techniques. In: Proceedings of IEEE on Security Technology, pp. 71–76 (2003)

  62. Kolter, J., Maloof, M.: Learning to detect malicious executables in the wild. In: Proceedings of the 2004 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 470–478. ACM Press, New York (2004)

  63. Lee, T., Mody, J.: Behavioral classification. In: Proceedings of EICAR (2006)

  64. Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.: Behavior-based spyware detection. In: Proceedings of the 15th USENIX Security Symposium (2006)

  65. Frost&Sullivan, Protection en temps réel contre toutes les menaces, Tech. Rep., White Paper Eset

  66. Avg anti-virus. Grisoft. http://www.grisoft.com/doc/39/lng/fr/tpl/tpl01

  67. Viguard. Softed. http://www.viguard.com/detail_163_logiciel_antivirus_viguard-platinium#

  68. Bitdefender antivirus technology, Tech. Rep., BitDefender White Paper

  69. Host and network intrusion prevention, competitors or partners? Tech. rep., Mc Affee White Paper (2004)

  70. Safe′n′sec antivirus. Safen Soft. http://www.safensoft.com/technology/

  71. Truprevent. Panda Software. http://www.pandasoftware.com/products/truprevent_tec.htm?sitepanda=particulares

  72. Virus keeper. AxBa. http://www.viruskeeper.com/fr/faq.htm

Download references

Author information

Authors and Affiliations

  1. France Télécom R&D, Caen, France

    Grégoire Jacob & Hervé Debar

  2. French Army Signals Academy, Virology and Cryptology Lab, Rennes, France

    Grégoire Jacob & Eric Filiol

Authors
  1. Grégoire Jacob
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hervé Debar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Filiol
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Grégoire Jacob.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jacob, G., Debar, H. & Filiol, E. Behavioral detection of malware: from a survey towards an established taxonomy. J Comput Virol 4, 251–266 (2008). https://doi.org/10.1007/s11416-008-0086-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-008-0086-0

Keywords

Search

Navigation