Skip to main content
SpringerLink
Log in

A hierarchical layer of atomic behavior for malicious behaviors prediction

  • Original Paper
  • Published:
Journal of Computer Virology and Hacking Techniques Aims and scope Submit manuscript

Abstract

Early prediction of malicious activity can help prevent irreparable damage caused by rogue actions. A malware analysis tool can anticipate malicious activity and stop the execution of the instance based on API calls to avoid the damage caused by the malware. The anticipation operation examines signatures as behaviors defined in a hierarchical model based on atomic behaviors and API calls. We define atomic behaviors as benign behaviors commonly observed in legitimate and malicious activities. Moreover, the hierarchical model is a reconstruction of the sequence of API calls, called activity or behavior, with atomic behaviors. A sequence of API calls is modeled as a behavior graph, and we use graph mining techniques to derive atomic behaviors and behavior signatures. Behavioral signatures are explored as the most common sequences of API calls observed in malicious behavior graphs. To this end, we used our formula \(H_{test}\) for ranking malicious behaviors. We then compared our signatures with those generated by the two well-known behavior signature generation techniques, CDG and CMQDG. The comparison results provide a reasonable indication of the superiority of our proposed method with an accuracy of 95.86% and an F1 of 95.72%. It also yields a micro F1 of 95.8% in early detection and prevention of malicious activities before any harm can occur.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alaeiyan, M., Dehghantanha, A., Dargahi, T., Conti, M., Parsa, S.: A multilabel fuzzy relevance clustering system for malware attack attribution in the edge layer of cyber-physical networks. ACM Trans. Cyber-Phys. Syst. 4(3), 1–22 (2020)

    Article  Google Scholar 

  2. Alaeiyan, M., Parsa, S., Conti, M.: Analysis and classification of context-based malware behavior. Comput. Commun. 136, 76–90 (2019)

    Article  Google Scholar 

  3. Alaeiyan, M.H., Parsa, S.: Automatic loop detection in the sequence of system calls. In 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI), pp. 720–723. IEEE (2015)

  4. Alazab, M., Venkataraman, S., Watters, P.: Towards understanding malware behaviour by the extraction of api calls. In Cybercrime and Trustworthy Computing Workshop (CTC), 2010 Second, pp. 52–59. IEEE (2010)

  5. Amer, E., Zelinka, I., El-Sappagh, S.: A multi-perspective malware detection approach through behavioral fusion of api call sequence. Comput. Secur. (2021). https://doi.org/10.1016/j.cose.2021.102449

    Article  Google Scholar 

  6. Ashik, M., Jyothish, A., Anandaram, S., Vinod, P., Mercaldo, F., Martinelli, F., Santone, A.: Detection of malicious software by analyzing distinct artifacts using machine learning and deep learning algorithms. Electronics 10(14), 1694 (2021)

    Article  Google Scholar 

  7. avtest: Malware statistics and trends report | av-test. https://www.av-test.org/en/statistics/malware/. Date: 01/18/2021

  8. Chen, Z.G., Kang, H.S., Yin, S.N., Kim, S.R.: Automatic ransomware detection and analysis based on dynamic api calls flow graph. In Proceedings of the International Conference on Research in Adaptive and Convergent Systems, pp. 196–201. ACM (2017)

  9. D’Angelo, G., Ficco, M., Palmieri, F.: Association rule-based malware classification using common subsequences of api calls. Appl. Soft Comput. 105, 107234 (2021)

    Article  Google Scholar 

  10. Ding, Y., Xia, X., Chen, S., Li, Y.: A malware detection method based on family behavior graph. Comput. Secur. 73, 73–86 (2018)

    Article  Google Scholar 

  11. Elhadi, A..A..E., Maarof, M..A., Barry, B.I., Hamza, H.: Enhancing the detection of metamorphic malware using call graphs. Comput. Secur. 46, 62–78 (2014)

    Article  Google Scholar 

  12. Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In Proceedings 1996 IEEE Symposium on Security and Privacy, pp. 120–128. IEEE (1996)

  13. Galal, H.S., Mahdy, Y.B., Atiea, M.A.: Behavior-based features model for malware detection. J. Comput. Virol. Hack. Tech. 12(2), 59–67 (2016)

    Article  Google Scholar 

  14. Gao, Y., Lu, Z., Luo, Y.: Survey on malware anti-analysis. In Fifth International Conference on Intelligent Control and Information Processing (ICICIP), pp. 270–275. IEEE (2014)

  15. Han, W., Xue, J., Wang, Y., Huang, L., Kong, Z., Mao, L.: Maldae: Detecting and explaining malware based on correlation and fusion of static and dynamic characteristics. Comput. Secur. 83, 208 (2019)

    Article  Google Scholar 

  16. Karbalaie, F., Sami, A., Ahmadi, M.: Semantic malware detection by deploying graph mining. Int. J. Comput. Sci. Issues (IJCSI) 9(1), 373–379 (2012)

    Google Scholar 

  17. Ki, Y., Kim, E., Kim, H.K.: A novel approach to detect malware based on api call sequence analysis. Int. J. Distrib. Sens. Netw. 11(6), 1–9 (2015)

    Article  Google Scholar 

  18. Kumar, A.V., Vishnani, K., Kumar, K.V.: Split personality malware detection and defeating in popular virtual machines. In Proceedings of the Fifth International Conference on Security of Information and Networks, pp. 20–26. ACM (2012)

  19. Liu, S., Feng, P., Wang, S., Sun, K., Cao, J.: Enhancing malware analysis sandboxes with emulated user behavior. Comput. Secur. p. 102613 (2022)

  20. Luh, R., Schramm, G., Wagner, M., Janicke, H., Schrittwieser, S.: Sequin: a grammar inference framework for analyzing malicious system behavior. J. Comput. Virol. Hack. Tech. pp. 1–21 (2018)

  21. Lundsgård, G., Nedström, V.: Bypassing modern sandbox technologies (2016). Student Paper

  22. Mahindru, A., Sangal, A.: Semidroid: a behavioral malware detector based on unsupervised machine learning techniques using feature selection approaches. Int. J. Mach. Learn. Cybern. 12(5), 1369–1411 (2021)

    Article  Google Scholar 

  23. MalwareBazaar: Malwarebazaar. https://bazaar.abuse.ch. Date: 2/22/2022

  24. Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In International Workshop on Recent Advances in Intrusion Detection, pp. 78–97. Springer (2008)

  25. Miao, Q., Liu, J., Cao, Y., Song, J.: Malware detection using bilayer behavior abstraction and improved one-class support vector machines. Int. J. Inf. Secur. 15(4), 361–379 (2016)

    Article  Google Scholar 

  26. Ming, J., Xin, Z., Lan, P., Wu, D., Liu, P., Mao, B.: Impeding behavior-based malware analysis via replacement attacks to malware specifications. J. Comput. Virol. Hack. Tech. 13(3), 193–207 (2017)

    Article  Google Scholar 

  27. Mpanti, A., Nikolopoulos, S.D., Polenakis, I.: A graph-based model for malicious software detection exploiting domination relations between system-call groups. In 9th International Conference on Computer Systems and Technologies, pp. 20–26. ACM (2018)

  28. Naderi, H., Vinod, P., Conti, M., Parsa, S., Alaeiyan, M.H.: Malware signature generation using locality sensitive hashing. In International Conference on Security and Privacy, pp. 115–124. Springer (2019)

  29. Péchoux, R., Ta, T.D.: A categorical treatment of malicious behavioral obfuscation. In International Conference on Theory and Applications of Models of Computation, pp. 280–299. Springer (2014)

  30. portableapps: portableapps. www.portableapps.com. Date: 2/22/2022

  31. portablefreeware: portablefreeware. www.portablefreeware.com. Date: 2/22/2022

  32. Ranu, S., Singh, A.K.: Graphsig: A scalable approach to mining significant subgraphs in large graph databases. In Data Engineering, 2009. ICDE’09. IEEE 25th International Conference on, pp. 844–855. IEEE (2009)

  33. Singh, J., Singh, J.: A survey on machine learning-based malware detection in executable files. J. Syst. Arch. 112, 101861 (2021). https://doi.org/10.1016/j.sysarc.2020.101861

    Article  Google Scholar 

  34. Souri, A., Hosseini, R.: A state-of-the-art survey of malware detection approaches using data mining techniques. HCIS 8(1), 3 (2018)

    Google Scholar 

  35. Ucci, D., Aniello, L., Baldoni, R.: Survey of machine learning techniques for malware analysis. Comput. Secur. 81, 123-147 (2019)

    Article  Google Scholar 

  36. Usman, N., Usman, S., Khan, F., Jan, M.A., Sajid, A., Alazab, M., Watters, P.: Intelligent dynamic malware detection using machine learning in ip reputation for forensics data analytics. Futur. Gener. Comput. Syst. 118, 124–141 (2021)

    Article  Google Scholar 

  37. virusshare: virusshare. www.virusshare.com. Date: 9/1/2021

  38. Wüchner, T., Cisłak, A., Ochoa, M., Pretschner, A.: Leveraging compression-based graph mining for behavior-based malware detection. IEEE Trans. Depend. Secure Comput. 16(1), 99–112 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Engineering, K. N. Toosi University of Technology, Seyed Khandan, Shariati Ave, 16317-14191, Tehran, Iran

    Mohammadhadi Alaeiyan

  2. Department of Computer Engineering, Iran University of Science and Technology, Narmak, 16844, Tehran, Iran

    Saeed Parsa

Authors
  1. Mohammadhadi Alaeiyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saeed Parsa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammadhadi Alaeiyan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alaeiyan, M., Parsa, S. A hierarchical layer of atomic behavior for malicious behaviors prediction. J Comput Virol Hack Tech 18, 367–382 (2022). https://doi.org/10.1007/s11416-022-00422-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-022-00422-5

Keywords

Search

Navigation