Advertisement

Journal in Computer Virology

, Volume 4, Issue 3, pp 251–266 | Cite as

Behavioral detection of malware: from a survey towards an established taxonomy

  • Grégoire Jacob
  • Hervé Debar
  • Eric Filiol
Original Paper
First Online:
Received:
Revised:
Accepted:

Abstract

Behavioral detection differs from appearance detection in that it identifies the actions performed by the malware rather than syntactic markers. Identifying these malicious actions and interpreting their final purpose is a complex reasoning process. This paper draws up a survey of the different reasoning techniques deployed among the behavioral detectors. These detectors have been classified according to a new taxonomy introduced inside the paper. Strongly inspired from the domain of program testing, this taxonomy divides the behavioral detectors into two main families: simulation-based and formal detectors. Inside these families, ramifications are then derived according to the data collection mechanisms the data interpretation, the adopted model and its generation, and the decision support.

Keywords

Virtual Machine Model Check Intrusion Detection System Call Execution Path 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cohen, F.: Computer viruses. Ph.D. thesis, University of South California (1986)Google Scholar
  2. 2.
    Cohen F.B. (1987). Computer viruses: Theory and experiments. Comput. Secur. 6(1): 22–35 CrossRefGoogle Scholar
  3. 3.
    Debar H., Dacier M. and Wespi A. (1999). Towards a taxonomy of intrusion-detection systems. Comput. Netw. Spl Issue Comput. Netw. Secur. 31(9): 805–822 Google Scholar
  4. 4.
    Mé, L., Morin, B.: Intrusion detection and virology: an analysis of differences, similarities and complementariness. In: Bonfante, G., Marion, J.-Y. (eds.) J. Comput. Virol., vol. 3, no. 1, WTCV’06 Special Issue, pp. 39–49 (2007)Google Scholar
  5. 5.
    Anderson, J.: Computer security threat monitoring and surveillance. Tech. rep., James P. Anderson Company (1980)Google Scholar
  6. 6.
    Denning, D.: An intrusion–detection model. IEEE Trans. Softw. Eng., vol. SE-13 (1987)Google Scholar
  7. 7.
    Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusion using system calls: Alternative data models, In: Proceedings of IEEE Symposium on Security and Privacy, pp. 133–145 (1999)Google Scholar
  8. 8.
    Zanero, S.: Behavioral intrusion detection. In: Proceedings of the 19th International Symposium on Computer and Information Sciences (ISCIS), pp. 657–666 (2004)Google Scholar
  9. 9.
    Filiol, E.: Computer viruses: from theory to applications. Springer, Heidelberg, IRIS Collection (2005). ISBN:2-287-23939-1Google Scholar
  10. 10.
  11. 11.
    Malware outbreak trend report: Storm-worm, Commtouch Software Ltd (2007). http://www.commtouch.com/downloads/Storm-Worm_MOTR.pdf
  12. 12.
    Filiol, E.: Malware pattern scanning schemes secure against black-box analysis. In: Broucek, V., Turner, P. (eds.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 35–50 (2006)Google Scholar
  13. 13.
    Filiol, E. (2007). Techniques Virales Avancées. Springer, Heidelberg, IRIS Collection. ISBN:2-287-33887-8Google Scholar
  14. 14.
    Ször, P.: The Art of Computer Virus Research and Defense. Addison-Wesley, Reading (2005). ISBN:0-321-30454-3Google Scholar
  15. 15.
    Spinellis D. (2003). Reliable identification of boundedlength viruses is np-complete. IEEE Trans. Inf. Theory 49: 280–284 zbMATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Filiol, E.: Metamorphism, formal grammars and undecidable code mutation. In: Proceedings of the International Conference on Computational Intelligence (ICCI), Published in the Int. J. Comput. Sci., vol. 2, issue 1, pp. 70–75 (2007)Google Scholar
  17. 17.
    Christodorescu, M., Jha, S.: Testing malware detectors, In: Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 34–44, ACM Press, New York (2004)Google Scholar
  18. 18.
    Josse, S.: How to assess the effectiveness of your anti-virus? In: Broucek, V. (ed.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 51–65 (2006)Google Scholar
  19. 19.
    Filiol, E., Jacob, G., Liard, M.L.: Evaluation methodology and theoretical model for antiviral behavioural detection strategies. In: Bonfante, G., Marion, J.-Y. (eds.) J. Comput. Virol., vol. 3, no. 1, WTCV’06 Special Issue, pp. 23–37 (2007)Google Scholar
  20. 20.
    Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Proceedings of the European Symposium on Research in Computer Security, pp. 326–343 (2003)Google Scholar
  21. 21.
    Hoglund, G., Butler, J.: Rootkits, Subverting the Windows Kernel. Addison-Wesley Professional, Reading (2006). ISBN: 0-321-29431-9Google Scholar
  22. 22.
    Vivanco, A.D.: Comprehensive non-intrusive protection with data-restoration: A proactive approach against malicious mobile code. Master’s thesis, Florida Institute of Technology (2002)Google Scholar
  23. 23.
    Wagner, M.E.: Behavior oriented detection of malicious code at run-time. Master’s thesis, Florida Institute of Technology (2004)Google Scholar
  24. 24.
    Norman’s sandbox malware analyzer. Norman ASA. http://www.norman.com/microsites/malwareanalyzer/fr/
  25. 25.
    Cwsandbox. Sunbelt Software. http://www.cwsandbox.org
  26. 26.
    Bayer, U., Moser, A., Kruegel, C., Kirda, E.: Dynamic analysis of malicious code. In: Broucek, V., Turner, P., (eds.) J. Comput. Virol., vol. 2, no. 1, EICAR 2006 Special Issue, pp. 67–77 (2006)Google Scholar
  27. 27.
    Rutkowska, J.: Red pill... or how to detect vmm using (almost) one cpu instruction (2005). http://invisiblethings.org/papers/redpill.html
  28. 28.
    Ferrie, P.: Attacks on virtual machine emulators. In: Proceedings of the AVAR Conference (2006)Google Scholar
  29. 29.
    Debbabi, M.: Dynamic monitoring of malicious activity in software systems. In: Proceedings of the Symposium on Requirements Engineering for Information Security (SREIS) (2001)Google Scholar
  30. 30.
    Nachenberg, C.: Behavior blocking: The next step in anti-virus protection, SecurityFocus, 2002. http://www.securityfocus.com/infocus/1557
  31. 31.
    Schmall, M.: Classification and identification of malicious code based on heuristic techniques utilizing meta-languages. Ph.D. thesis, University of Hamburg (2002)Google Scholar
  32. 32.
    Schmall, M.: Heuristic techniques in av solutions: An overview, SecurityFocus (2002). http://www.securityfocus.com/infocus/1542
  33. 33.
    Veldman, F.: Heuristic anti-virus technology. In: Proceedings of the International Virus Protection and Information Security Council (1994)Google Scholar
  34. 34.
    Zwienenberg, R.: Heuristics scanners: Artificial intelligence? In: Proceedings of the Virus Bulletin Conference, pp. 203–210 (1994)Google Scholar
  35. 35.
    Understanding heuristics: Symantec bloodhound technology. Tech. rep., Symantec White Paper Series, vol. XXXIV (1997)Google Scholar
  36. 36.
    Glover, F.W., Kochenberger, G.A.: Handbook of Metaheuristics. Springer, Heidelberg (2003). ISBN:1-402-07263-5Google Scholar
  37. 37.
    Charlier, B.L., Mounji, A., Swimmer, M.: Dynamic detection and classification of computer viruses using general behaviour patterns. In: Proceedings of the Virus Bulletin Conference (1995)Google Scholar
  38. 38.
    Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based approach for detecting anomalous program behaviors. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 144–155 (2001)Google Scholar
  39. 39.
    Hopcroft, J., Motwani, R., Ullman, J.: Introduction to Automata Theory, Languages and Computation, 2nd edn. Addison Wesley, Reading (1995). ISBN:0-201-44124-1Google Scholar
  40. 40.
    Mazeroff, G., Cerqueira, V.D., Gregor, J., Thomason, M.G.: Probabilistic trees and automata for application behavior modeling. In: Proceedings of the 43rd ACM Southeast Conference (2003)Google Scholar
  41. 41.
    Kaspersky, K.: Hacker Disassembling Uncovered, 2nd edn. A-LIST, LLC (2007). ISBN:1-931-76964-8Google Scholar
  42. 42.
    Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations. Tech. rep., Technical Report 148, Department of Computer Science, University of Auckland (1997)Google Scholar
  43. 43.
    Kruegel, C., Robertson, W., Valeur, F., Vigna, G.: Static disassembly of obfuscated binaries. In: SSYM’04: Proceedings of the 13th conference on USENIX Security Symposium, pp. 18–18 (2004)Google Scholar
  44. 44.
    Josse S. (2007). Secure and advanced unpacking using computer emulation, extended version from the avar conference. J. Comput. Virol. 3(3): 221–236 CrossRefGoogle Scholar
  45. 45.
    Preda, M.D., Christodorescu, M., Jha, S., Debray, S.: A semantic-based approach to malware detection. In: Proceedings of the 34th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) (2007)Google Scholar
  46. 46.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantic-aware malware detection. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 32–46 (2005)Google Scholar
  47. 47.
    Bruschi, D., Martignoni, L., Monga, M.: Detecting self-mutating malware using control-flow graph matching. In: Proceedings of the Conference on the Detection of Intrusions and Malwares and Vulnerability Assessment (DIMVA), pp. 129–143 (2006)Google Scholar
  48. 48.
    Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic worm detection using structural information of executables. In: International Symposium on Recent Advances in Intrusion Detection (RAID) (2005)Google Scholar
  49. 49.
    Periot, F.: Defeating polymorphism through code optimization. In: Proceedings of the Virus Bulletin Conference, pp. 142–159 (2003)Google Scholar
  50. 50.
    Bruschi, D., Martignoni, L., Monga, M.: Using code normalization for fighting self-mutating malware. In: Proceedings of the International Symposium on Secure Software Engineering, pp. 37–44, IEEE CS Press (2006)Google Scholar
  51. 51.
    Webster, M.: Algebraic specification of computer viruses and their environments. In: Selected Papers from the First Conference on Algebra and Coalgebra in Computer Science Young Researchers Workshop (CALCO-jnr 2005), University of Wales Swansea Computer Science Report Series (CSR 18-2005), pp. 99–113 (2005)Google Scholar
  52. 52.
    Webster M. and Malcolm G. (2006). Detection of metamorphic computer viruses using algebraic specification. J. Comput. Virol. 2(3): 149–161 CrossRefGoogle Scholar
  53. 53.
    Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M.M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: Proceedings of the Symposium on Requirements Engineering for Information Security (SREIS) (2001)Google Scholar
  54. 54.
    Singh, P., Lakhotia, A.: Static verification of worm and virus behavior in binary executables using model checking. In: Proceedings of the IEEE Information Assurance Workshop, pp. 298–300 (2003)Google Scholar
  55. 55.
    Clark, E., Grumberg, O., Long, D.: Model Checking. MIT Press, Cambridge (1999). ISBN:0-262-03270-8Google Scholar
  56. 56.
    Schnoebelen P. (2003). The complexity of temporal logic model checking. Adv. Modal Logic 4: 393–436 MathSciNetGoogle Scholar
  57. 57.
    Kinder J., Katzenbeisser S., Schallhart C. and Veith H. (2005). Detecting malicious code by model checking. Lect. Notes Computer Sci. 3548: 174–187 CrossRefGoogle Scholar
  58. 58.
    Perdisci, R., Dagon, D., Fogla, P.W.L., Sharif, M.: Misleading worm signature generators using deliberate noise injection. In: Proceedings of IEEE Symposium on Security and Privacy (2006)Google Scholar
  59. 59.
    Lee, W., Stolfo, S., Chan, P.: Learning patterns from unix process execution traces for intrusion detection. In: Proceedings of the AAAI97 Workshop on AI Approaches to Fraud Detection and Risk Management, pp. 50–56. Addison Wesley, Reading (1997)Google Scholar
  60. 60.
    Schultz, M.G., Eskin, E., Zadok, E.: Data mining methods for detection of new malicious executables. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 38–49 (2001)Google Scholar
  61. 61.
    Wang, J.-H., Deng, P.S., Fan, Y.-S., Jaw, L.-J., Liu, Y.-C.: Virus detection using data mining techniques. In: Proceedings of IEEE on Security Technology, pp. 71–76 (2003)Google Scholar
  62. 62.
    Kolter, J., Maloof, M.: Learning to detect malicious executables in the wild. In: Proceedings of the 2004 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 470–478. ACM Press, New York (2004)Google Scholar
  63. 63.
    Lee, T., Mody, J.: Behavioral classification. In: Proceedings of EICAR (2006)Google Scholar
  64. 64.
    Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.: Behavior-based spyware detection. In: Proceedings of the 15th USENIX Security Symposium (2006)Google Scholar
  65. 65.
    Frost&Sullivan, Protection en temps réel contre toutes les menaces, Tech. Rep., White Paper EsetGoogle Scholar
  66. 66.
  67. 67.
  68. 68.
    Bitdefender antivirus technology, Tech. Rep., BitDefender White PaperGoogle Scholar
  69. 69.
    Host and network intrusion prevention, competitors or partners? Tech. rep., Mc Affee White Paper (2004)Google Scholar
  70. 70.
    Safe′n′sec antivirus. Safen Soft. http://www.safensoft.com/technology/
  71. 71.
  72. 72.

Copyright information

© Springer-Verlag France 2008

Authors and Affiliations

  1. 1.France Télécom R&DCaenFrance
  2. 2.French Army Signals Academy, Virology and Cryptology LabRennesFrance

Personalised recommendations

Cite article

Buy options