Tutorial: An Overview of Malware Detection and Evasion Techniques

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11244)


This tutorial presents and motivates various malware detection tools and illustrates their usage on a clear example. We demonstrate how statically-extracted syntactic signatures can be used for quickly detecting simple variants of malware. Since such signatures can easily be obfuscated, we also present dynamically-extracted behavioral signatures which are obtained by running the malware in an isolated environment known as a sandbox. However, some malware can use sandbox detection to detect that they run in such an environment and so avoid exhibiting their malicious behavior. To counteract sandbox detection, we present concolic execution that can explore several paths of a binary. We conclude by showing how opaque predicates and JIT can be used to hinder concolic execution.


  1. 1.
    Agrawal, H., Bahler, L., Micallef, J., Snyder, S., Virodov, A.: Detection of global, metamorphic malware variants using control and data flow analysis. In: 31st IEEE Military Communications Conference, MILCOM 2012, Orlando, October 29 – November 1, 2012, pp. 1–6 (2012).
  2. 2.
    Alazab, M., Venkatraman, S., Watters, P., Alazab, M.: Zero-day malware detection based on supervised learning algorithms of API call signatures. In: Proceedings of the Ninth Australasian Data Mining Conference, vol. 121, pp. 171–182. AusDM 2011, Australian Computer Society Inc., Darlinghurst (2011).
  3. 3.
    Canali, D., Lanzi, A., Balzarotti, D., Kruegel, C., Christodorescu, M., Kirda, E.: A quantitative study of accuracy in system call-based malware detection. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 122–132. ISSTA 2012. ACM, New York (2012).
  4. 4.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: 2005 IEEE Symposium on Security and Privacy (SP 2005), pp. 32–46, May 2005.
  5. 5.
  6. 6.
    ClamAV: Clamav 0.99b meets yara! ClamAV blog.
  7. 7.
    Collberg, C., Martin, S., Myers, J., Nagra, J.: Distributed application tamper detection via continuous software updates. In: Proceedings of the 28th Annual Computer Security Applications Conference. ACSAC 2012, pp. 319–328. ACM, New York (2012).
  8. 8.
    Ehsan, F.: Detecting unknown malware: security analytics & memory forensics. Presentation at RSA 2015 Conference (2015).
  9. 9.
    Godefroid, P., Klarlund, N., Sen, K.: Dart: Directed automated random testing. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 213–223. PLDI 2005, ACM, New York (2005).
  10. 10.
    Goldberg, R.P.: Survey of virtual machine research. Computer 7(6), 34–45 (1974)CrossRefGoogle Scholar
  11. 11.
    Idika, N.C., Mathur, A.P.: A survey of malware detection techniques (2007)Google Scholar
  12. 12.
    Intel: Intel\({\textregistered }\) 64 and ia-32 architectures software developer’s manual combined volumes 2a, 2b, 2c, and 2d: Instruction set reference, a-z. Technical report, May 2018., order Number: 325383–067US
  13. 13.
    Jung, P.: Bypassing sanboxes for fun! Presentation at (2014).
  14. 14.
    Karbalaie, F., Sami, A., Ahmadi, M.: Semantic malware detection by deploying graph mining. Int. J. Comput. Sci. Issues (IJCSI) 9(1), 373 (2012)Google Scholar
  15. 15.
    Kirat, D., Vigna, G., Kruegel, C.: Barecloud: bare-metal analysis-based evasive malware detection. In: USENIX Security Symposium, pp. 287–301 (2014)Google Scholar
  16. 16.
    Kuzurin, N., Shokurov, A., Varnovsky, N., Zakharov, V.: On the concept of software obfuscation in computer security. In: Garay, J.A., Lenstra, A.K., Mambo, M., Peralta, R. (eds.) ISC 2007. LNCS, vol. 4779, pp. 281–298. Springer, Heidelberg (2007). Scholar
  17. 17.
    Ligh, M.H., Case, A., Levy, J., Walters, A.: The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory. Wiley, Indianapolis (2014)Google Scholar
  18. 18.
    MissMalware: Tdsanomalpe - identifying compile time manipulation in pe headers. Miss Malware blog.
  19. 19.
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 421–430, December 2007.
  20. 20.
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: ACSAC, pp. 421–430. IEEE Computer Society (2007).
  21. 21.
    Pietrek, M.: Peering inside the PE: A tour of the win32 portable executable file format. Microsoft Developer Network blog (1994).
  22. 22.
    Preda, M.D., Christodorescu, M., Jha, S., Debray, S.: A semantics-based approach to malware detection. SIGPLAN Not 42(1), 377–388 (2007).,
  23. 23.
    Project, Y.: Yara documentation.
  24. 24.
    Schwartz, M.: Oracle virtualbox multiple guest to host escape vulnerabilities. SecuriTeam Secure Disclosure blog (2018).
  25. 25.
    Sen, K., Marinov, D., Agha, G.: Cute: a concolic unit testing engine for c. In: Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 263–272. ESEC/FSE-13, ACM, New York, (2005).
  26. 26.
    Sharma, A., Sahay, S.K.: Evolution and detection of polymorphic and metamorphic malwares: a survey. Int. J. Comput. Appl. 90(2), 7–11 (2014)Google Scholar
  27. 27.
    Shoshitaishvili, Y., et al.: SoK: (state of) the art of war: offensive techniques in binary analysis. In: IEEE Symposium on Security and Privacy (2016)Google Scholar
  28. 28.
    Sikorski, M., Honig, A.: Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software, 1st edn. No Starch Press, San Francisco (2012)Google Scholar
  29. 29.
    Subwire, l.: throwing a tantrum, part 1: angr internals. Angr blog.
  30. 30.
    Szor, P.: The Art of Computer Virus Research and Defense. Addison-Wesley Professional, Boston (2005)Google Scholar
  31. 31.
    Vasilenko, E., Mamedov, O.: To crypt, or to mine - that is the question. post on Securelist - Kaspersky Lab’s cyberthreat research and reports (2018).
  32. 32.
  33. 33.
    Wojtczuk, R., Rutkowska, J.: Following the White Rabbit: Software attacks against Intel (R) VT-d technology (2011).
  34. 34.
    Yadegari, B., Debray, S.: Symbolic execution of obfuscated code. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 732–744. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.CentraleSupélec/IRISARennesFrance
  2. 2.InriaRocquencourtFrance

Personalised recommendations