A Behavior-Based Approach for Malware Detection

Conference paper
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 511)


Malware is the fastest growing threat to information technology systems. Although a single absolute solution for defeating malware is improbable, a stacked arsenal against malicious software enhances the ability to maintain security and privacy. This research attempts to reinforce the anti-malware arsenal by studying a behavioral activity common to software – the use of handles. The characteristics of handle usage by benign and malicious software are extracted and exploited in an effort to distinguish between the two classes. An automated malware detection mechanism is presented that utilizes memory forensics, information retrieval and machine learning techniques. Experimentation with a malware dataset yields a malware detection rate of 91.4% with precision and recall of 89.8% and 91.1%, respectively.


Malware Memory forensics Machine learning Handles 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Aghaeikheirabady, M., Farshchi, S., Shirazi, H.: A new approach to malware detection by comparative analysis of data structures in a memory image. In: Proceedings of the First International Congress on Technology, Communication and Knowledge (2014)Google Scholar
  2. [2]
    Altman, N.: An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46(3), 175–185 (1992)MathSciNetGoogle Scholar
  3. [3]
    Berlin, K., Slater, D., Saxe, J.: Malicious behavior detection using windows audit logs. In: Proceedings of the Eighth ACM Workshop on Artificial Intelligence and Security, pp. 35–44 (2015)Google Scholar
  4. [4]
    Blunden, B.: The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System. Jones and Bartlett Learning, Burlington (2013)Google Scholar
  5. [5]
    Buitinck, L., Louppe, G., Blondel, M., Pedregosa, F., Mueller, A., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., Layton, R., VanderPlas, J., Joly, A., Holt, B., Varoquaux, G.: API design for machine learning software: experiences from the scikit-learn project. In: Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Workshop: Languages for Data Mining and Machine Learning, pp. 108–122 (2013)Google Scholar
  6. [6]
    Christodorescu, M., Jha, S., Seshia, S., Song, D., Bryant, R.: Semantics-aware malware detection. In: Proceedings of the IEEE Symposium on Security and Privacy, pp. 32–46 (2005)Google Scholar
  7. [7]
    Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995)zbMATHGoogle Scholar
  8. [8]
    Cuckoo Foundation, Cuckoo Sandbox (2016).
  9. [9]
    Dolan-Gavitt, B., Srivastava, A., Traynor, P., Giffin, J.: Robust signatures for kernel data structures. In: Proceedings of the Sixteenth ACM Conference on Computer and Communications Security, pp. 566–577 (2009)Google Scholar
  10. [10]
    Emm, D., Unuchek, R., Garnaeva, M., Ivanov, A., Makrushin, D., Sinitsyn, F.: IT Threat Evolution in Q2 2016. Kaspersky Lab, Moscow (2016)Google Scholar
  11. [11]
    Galal, H., Mahdy, Y., Atiea, M.: Behavior-based features model for malware detection. Journal of Computer Virology and Hacking Techniques 12(2), 59–67 (2016)CrossRefGoogle Scholar
  12. [12]
    Ho, T.: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)CrossRefGoogle Scholar
  13. [13]
    Hoglund, G., Butler, J.: Rootkits: Subverting the Windows Kernel. Pearson Education, Upper Saddle River (2006)Google Scholar
  14. [14]
    Hungenberg, T., Eckert, M.: INetSim: Internet Services Simulation Suite (2007).
  15. [15]
    Klein, B., Peters, R.: Defeating machine learning - What your security vendor is not telling you. Presented at Black Hat USA (2015)Google Scholar
  16. [16]
    Ligh, M., Case, A., Levy, J., Walters, A.: The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux and Mac Memory. John Wiley and Sons, Indianapolis (2014)Google Scholar
  17. [17]
    Lin, Y., Lai, Y., Lu, C., Hsu, P., Lee, C.: Three-phase behavior-based detection and classification of known and unknown malware. Security and Communication Networks 8(11), 2004–2015 (2015)CrossRefGoogle Scholar
  18. [18]
    Luttgens, J., Pepe, M., Mandia, K.: Incident Response and Computer Forensics. McGraw Hill Education, New York (2014)Google Scholar
  19. [19]
    Manning, C., Raghavan, P., Schutze, H.: An Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)CrossRefzbMATHGoogle Scholar
  20. [20]
    Markel, Z., Bilzor, M.: Building a machine learning classifier for malware detection. In: Proceedings of the Second Workshop on Anti-Malware Testing Research (2014)Google Scholar
  21. [21]
    Masud, M., Sahib, S., Abdollah, M., Selamat, S., Yusof, R.: Analysis of features selection and machine learning classifier in Android malware detection. In: Proceedings of the International Conference on Information Science and Applications (2014)Google Scholar
  22. [22]
    Mohaisen, A., Alrawi, O., Mohaisen, M.: AMAL: High-fidelity, behavior-based automated malware analysis and classification. Computers and Security 52, 251–266 (2015)CrossRefGoogle Scholar
  23. [23]
    Mosli, R., Li, R., Yuan, B., Pan, Y.: Automated malware detection using artifacts in forensic memory images. In: Proceedings of the IEEE Symposium on Technologies for Homeland Security (2016)Google Scholar
  24. [24]
    Nath, H., Mehtre, B.: Static malware analysis using machine learning methods. In: Proceedings of the Second International Conference on Recent Trends in Computer Networks and Distributed Systems Security, pp. 440–450 (2014)Google Scholar
  25. [25]
    Naval, S., Laxmi, V., Rajarajan, M., Gaur, M., Conti, M.: Employing program semantics for malware detection. IEEE Transactions on Information Forensics and Security 10(12), 2591–2604 (2015)CrossRefGoogle Scholar
  26. [26]
    Park, Y., Reeves, D., Stamp, M.: Deriving common malware behavior through graph clustering. Computers and Security 39(B), 419–430 (2013)CrossRefGoogle Scholar
  27. [27]
    Pirscoveanu, R., Hansen, S., Larsen, T., Stevanovic, M., Pedersen, J., Czech, A.: Analysis of malware behavior: type classification using machine learning. In: Proceedings of the International Conference on Cyber Situational Awareness, Data Analytics and Assessment (2015)Google Scholar
  28. [28]
    Roberts, J.: VirusShare Project (2017).
  29. [29]
    Russinovich, M.: Pushing the limits of Windows: Handles, Mark’s Blog, September 29, 2009.
  30. [30]
    Russinovich, M.: Sysinternals Suite, Microsoft TechNet, Redmond, Washington (2017).
  31. [31]
    Russinovich, M., Solomon, D., Ionescu, A.: Windows Internals. Microsoft Press, Redmond (2012)Google Scholar
  32. [32]
    Santos, I., Brezo, F., Ugarte-Pedrero, X., Bringas, P.: Opcode sequences as representation of executables for data-mining-based unknown malware detection. Information Sciences 231, 64–82 (2013)MathSciNetCrossRefGoogle Scholar
  33. [33]
    Saxe, J., Berlin, K.: Deep neural network based malware detection using two dimensional binary program features. In: Proceedings of the Tenth International Conference on Malicious and Unwanted Software, pp. 11–20 (2015)Google Scholar
  34. [34]
    Schuster, A.: Enumerate Object Types. Computer Forensic Blog, April 7, 2009.
  35. [35]
    Stuttgen, J., Cohen, M.: Anti-forensic resilient memory acquisition. Digital Investigation 10(S), S105–S115 (2013)CrossRefGoogle Scholar
  36. [36]
    Teller, T., Hayon, A.: Enhancing automated malware analysis machines with memory analysis. Presented at Black Hat USA (2014)Google Scholar
  37. [37]
    Zaki, A., Humphrey, B.: Unveiling the kernel: Rootkit discovery using selective automated kernel memory differencing. Presented at the Virus Bulletin Conference (2014)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2017

Authors and Affiliations

  1. 1.Rochester Institute of TechnologyRochesterUSA

Personalised recommendations