Abstract
The battle between malware analysts and malware authors is a never-ending challenge with the advent of complex malware such as polymorphic, metamorphic, and packed malware. A malware packer uses various techniques combined with file encryption to harden against reverse engineering of the program and hinder the analysis of program behaviors. In any case, substantial elements have emerged after more than a decade of continuous research in malware packer detection, such as multi-packing. Newly modified packers have this persistent problem, which demands new concepts and techniques. This study aims to provide a systematic and comprehensive review of run-time packers’ mitigation techniques. We provide different types of packers and propose a malware packer handling life cycle for AV engines. Furthermore, we deliver a modern malware packers classification features set by examining the feature engineering in the packing handling life-cycle, such as feature extraction techniques in machine learning approaches. Also, we present extensive related works and discuss each work’s benefits and weaknesses to address this problem, with a particular emphasis on packers identification techniques, to aid in unpacking malware. Finally, we identify the current gaps in knowledge and provide ideas about future work.
Similar content being viewed by others
Data Availability Statement
The authors thank VirusTotal for providing the malware dataset used in this research. The malware dataset can be requested at https://www.virustotal.com. Additionally, the authors thank Softpedia for providing various benign downloadable software, which can be found at https://www.softpedia.com.
References
A portable reversing framework. Radare2 (2021). https://rada.re/r/
Aaraj, N., Raghunathan, A., Jha, N.K.: Dynamic binary instrumentation-based framework for malware defense. In: International Conference on Detection of Intrusions and Malware and Vulnerability Assessment, pp. 64–87. Springer, Berlin (2008)
Alkhateeb, E.M., Stamp, M.: A dynamic heuristic method for detecting packed malware using Naive Bayes. In: 2019 International Conference on Electrical and Computing Technologies and Applications (ICECTA), pp. 1–6. IEEE (2019)
Alkhateeb, E.M.S.: Dynamic malware detection using API similarity. In: 2017 IEEE International Conference on Computer and Information Technology (CIT), pp. 297–301. IEEE (2017)
Amer, E., Zelinka, I.: A dynamic windows malware detection and prediction method based on contextual understanding of API call sequence. Comput. Secur. 92, 101760 (2020)
Anderson, H.S., Roth, P.: Ember: an open dataset for training static PE malware machine learning models. arXiv preprint arXiv:1804.04637 (2018)
Bai, J., Shi, Q., Mu, S.: A malware and variant detection method using function call graph isomorphism. Security and Communication Networks (2019)
Bania, P.: Generic unpacking of self-modifying, aggressive, packed binary programs. arXiv preprint arXiv:0905.4581 (2009)
Bat-Erdene, M., Park, H., Li, H., Lee, H., Choi, M.-S.: Entropy analysis to classify unknown packing algorithms for malware detection. Int. J. Inf. Secur. 16(3), 227–248 (2017)
Bat-Erdene, M., Kim, T., Park, H., Lee, H.: Packer detection for multi-layer executables using entropy analysis. Entropy 19(3), 125 (2017)
Bergenholtz, E., Casalicchio, E., Ilie, D., Moss, A.: Detection of metamorphic malware packers using multilayered LSTM networks. In: International Conference on Information and Communications Security, pp. 36–53. Springer, Berlin (2020)
Biondi, F., Enescu, M.A., Given-Wilson, T., Legay, A., Noureddine, L., Verma, V.: Effective, efficient, and robust packing detection and classification. Comput. Secur. 85, 436–451 (2019)
Biryukov, A., Nakahara, J., Jr., Yıldırım, H.M.: Differential entropy analysis of the idea block cipher. J. Comput. Appl. Math. 259, 561–570 (2014)
Blazytko, T., Contag, M., Aschermann, C., Holz, T.: Syntia: synthesizing the semantics of obfuscated code. In: 26th \(\{\)USENIX\(\}\) Security Symposium (\(\{\)USENIX\(\}\) Security 17), pp. 643–659 (2017)
Bonfante, G., Fernandez, J., Marion, J.-Y., Rouxel, B., Sabatier, F., Thierry, A.: Codisasm: medium scale concatic disassembly of self-modifying binaries with overlapping instructions. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 745–756 (2015)
Branco, R.R., Barbosa, G.N., Neto, P.D.: Scientific but not academical overview of malware anti-debugging, anti-disassembly and anti-VM technologies. Black. Hat. 1, 1–27 (2012)
BROADCOM: Critical system protection. 2010. https://community.broadcom.com/symantecenterprise/communities/community-home/librarydocuments/viewdocument?DocumentKey=3265611c-0bbb-4232-ac08-9ebfbd89870d &CommunityKey=3f8a53f1-00c7-4411-8203-ee040b59e575 &tab=librarydocuments
Carvey, H.: Windows Forensic Analysis Toolkit: Advanced Analysis Techniques for Windows 8. Elsevier, Amsterdam (2014)
Cesare, S., Xiang, Y., Zhou, W.: MALWISE—an effective and efficient classification system for packed and polymorphic malware. IEEE Trans. Comput. 62(6), 1193–1206 (2012)
Cheng, B., Ming, J., Fu, J., Peng, G., Chen, T., Zhang, X., Marion, J.-Y.: Towards paving the way for large-scale windows malware analysis: Generic binary unpacking with orders-of-magnitude performance boost. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 395–411 (2018)
Choi, M.-J., Bang, J., Kim, J., Kim, H., Moon, Y.-S.: All-in-one framework for detection, unpacking, and verification for malware analysis. Secur. Commun. Netw. (2019)
Choi, Y.-S., Kim, I.-K., Oh, J.-T., Ryou, J.-C.: Pe file header analysis-based packed pe file detection technique (phad). In: International Symposium on Computer Science and its Applications, pp. 28–31. IEEE, (2008)
Chubachi, Y., Aiko, K.: Tentacle: environment-sensitive malware palpation. PacSec2014 (2014)
Cozzi, E., Graziano, M., Fratantonio, Y., Balzarotti, D.: Understanding linux malware. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 161–175. IEEE (2018)
Dam, K.H.T., Given-Wilson, T., Legay, A., Veroneze, R.: Packer classification based on association rule mining. Appl. Soft Comput. 127, 109373 (2022)
D’Elia, D.C., Nicchi, S., Mariani, M., Marini, M., Palmaro, F.: Designing robust API monitoring solutions. arXiv preprint arXiv:2005.00323 (2020)
Devi, D., Nandi, S.: PE file features in detection of packed executables. Int. J. Comput. Theory Eng. 4(3), 476 (2012)
Dolan-Gavitt, B.F., Hodosh, J., Hulin, P., Leek, T., Whelan, R.: Repeatable reverse engineering for the greater good with panda (2014)
Structural entropy and metamorphic malware: Donabelle, B., Richard, M.L., Mark. S. J. Comput. Virol. Hack. Tech. 9, 179–192 (2013)
DynamicRIO: Library call tracer. 2021. https://dynamorio.org/page_drltrace.html
Eagle, C.: The IDA pro book. No starch press (2011)
Ebringer, T., Sun, L., Boztas, S.: A fast randomness test that preserves local detail. In: Proceedings of the 18th Virus Bulletin International Conference, pp. 34–42. Virus Bulletin Ltd (2008)
Fang, Y., Zeng, Y.: Deepdetectnet vs Rlattacknet: an adversarial method to improve deep learning-based static malware detection model. PLoS ONE 15(4), e0231626 (2020)
Farinholt, B., Rezaeirad, M., Pearce, P., Dharmdasani, H., Yin, H., Le Blond, S., McCoy, D., Levchenko, K.: To catch a ratter: monitoring the behavior of amateur darkcomet rat operators in the wild. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 770–787. IEEE (2017)
Gao, X., Changzhen, H., Shan, C., Han, W.: Malicage: a packed malware family classification framework based on DNN and GAN. J. Inf. Secur. Appl. 68, 103267 (2022)
Guide, P.: Intel® 64 and IA-32 architectures software developer’s manual. Volume 3B: System programming Guide, Part, 2(11), 1–64 (2011)
HaddadPajouh, H., Dehghantanha, A., Khayami, R., Choo, K.-K.R.: A deep recurrent neural network based approach for internet of things malware threat hunting. Fut. Gener. Comput. Syst. 85, 88–96 (2018)
Hai, N.M., Ogawa, M., Tho, Q.T.: Packer identification based on metadata signature. In: Proceedings of the 7th Software Security, Protection, and Reverse Engineering/Software Security and Protection Workshop, pp. 1–11 (2017)
Herrmann, D.: Cyber Espionage and Cyber Defence, pp. 83–106. Springer Fachmedien Wiesbaden, Wiesbaden (2019)
Homeland Security Today: Increased use of a Delphi packer to evade malware classification (2018). https://www.hstoday.us/subject-matter-areas/cybersecurity/increased-use-of-a-delphi-packer-to-evade-malware-classification/
Hors: Program for determining types of files (2021). https://github.com/horsicq/Detect-It-Easy
Hotz, G.: The ultimate disassembler (2021). https://www.capstone-engine.org
Hsiao, S.-C., Kao, D.-Y., Tso, R.: Malware-detection model using learning-based discovery of static features. In: 2018 IEEE Conference on Application, Information and Network Security (AINS), pp. 54–59. IEEE (2018)
Jacob, G., Comparetti, P.M., Neugschwandtner, M., Kruegel, C., Vigna, G.: A static, packer-agnostic filter to detect similar malware samples. In: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pp. 102–122. Springer, Berlin (2012)
Jajodia, S., Shakarian, P., Subrahmanian, V.S., Swarup, V., Wang, C.: Cyber Warfare: Building the Scientific Foundation, vol. 56. Springer, Berlin (2015)
Jeong, G., Choo, E., Lee, J., Bat-Erdene, M., Lee, H.: Generic unpacking using entropy analysis. In: 2010 5th International Conference on Malicious and Unwanted Software, pp. 98–105. IEEE (2010)
Jin, Q., Duan, J., Vasudevan, S., Bailey, M.: Packer classifier based on PE header information. In: Proceedings of the 2015 Symposium and Bootcamp on the Science of Security, pp. 1–2 (2015)
Jung, B.H., Bae, S.I., Choi, C., Im, E.G.: Packer identification method based on byte sequences. Concurr. Comput.: Pract. Exp. 32(8), e5082 (2020)
Kancherla, K., Donahue, J., Mukkamala, S.: Packer identification using byte plot and Markov plot. J. Comput. Virol. Hack. Tech. 12(2), 101–111 (2016)
Kerrisk, M.: Objdump (2021). https://sourceware.org/binutils/docs/binutils/objdump.html
Kim, J.-W., Moon, Y.-S., Choi, M.-J: An efficient multi-step framework for malware packing identification. arXiv preprint arXiv:2208.08071 (2022)
Korczynski, D.: Precise system-wide concatic malware unpacking. arXiv preprint arXiv:1908.09204 (2019)
Kwiatkowski, I.: A static analyzer for PE executables (2021). https://github.com/JusticeRage/Manalyze
Lab, K.: Multipacked (2021). https://encyclopedia.kaspersky.com/knowledge/multipacked/
Lau, B., Svajcer, V.: Measuring virtual machine detection in malware using DSD tracer. J. Comput. Virol. 6(3), 181–195 (2010)
Lawton, K.: The cross platform ia-32 emulator (2021). https://bochs.sourceforge.io/
Laxmi, V., Gaur, M.S., Faruki, P., Naval, S.: Peal-packed executable analysis. In: International Conference on Advanced Computing, Networking and Security, pp. 237–243. Springer, Berlin (2011)
Lee, Y.B., Suk, J.H., Lee, D.H.: Bypassing anti-analysis of commercial protector methods using DBI tools. IEEE Access 9, 7655–7673 (2021)
Li, X., Shan, Z., Liu, F., Chen, Y., Hou, Y.: A consistently-executing graph-based approach for malware packer identification. IEEE Access 7, 51620–51629 (2019)
Lim, C., Ramli, K., Kotualubun, Y.S., et al.: Mal-flux: rendering hidden code of packed binary executable. Digit. Investig. 28, 83–95 (2019)
Liţă, C.V., Cosovan, D., Gavriluţ, D.: Anti-emulation trends in modern packers: a survey on the evolution of anti-emulation techniques in UPA packers. J. Comput. Virol. Hack. Tech. 14(2), 107–126 (2018)
Liu, H, Guo, C., Cui, Y., Shen, G., Ping, Y.: 2-spiff: a 2-stage packer identification method based on function call graph and file attributes. Appl. Intell. pp. 1–16 (2021)
Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. ACM Sigplan Notices 40(6), 190–200 (2005)
Lyda, R., Hamrock, J.: Using entropy analysis to find encrypted and packed malware. IEEE Secur. Priv. 5(2), 40–45 (2007)
Lyu, F., Lin, Y., Yang, J.: An efficient and packing-resilient two-phase android cloned application detection approach. Mobile Inf. Syst. 2017, Art. no. 6958698, (2017)
Malin, C.H., Casey, E., Aquilina, J.M.: Malware forensics field guide for Linux systems: digital forensics field guides. Syngress, an imprint of Elsevier (2013)
Aqulina, J.M., Casey, E., Malin, C.H.: Malware forensics: investigating and analyzing Malicious Code. Syngress, an imprint of Elsevier (2008)
Mantovani, A., Aonzo, S., Ugarte-Pedrero, X., Merlo, A., Balzarotti, D.: Prevalence and impact of low-entropy packing schemes in the malware ecosystem. In: Network and Distributed System Security (NDSS) Symposium, NDSS, vol. 20 (2020)
Martignoni, L., Christodorescu, M., Jha, S.: Omniunpack: fast, generic, and safe unpacking of malware. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 431–441. IEEE (2007)
McAfee: The good, the bad, and the unknown (2017). http://www.techdata.com/mcafee/files/MCAFEE_wp_appcontrol-good-bad-unknown.pdf
Menéndez, H.D., Llorente, J.L.: Mimicking anti-viruses with machine learning and entropy profiles. Entropy 21(5), 513 (2019)
Menéndez, H.D., Bhattacharya, S., Clark, D., Barr, E.T.: The arms race. Adversarial search defeats entropy used to detect malware. Expert Syst. Appl. 118, 246–260 (2019)
Menéndez, H.D., Clark, D., Barr, E.T.: Getting ahead of the arms race: hothousing the coevolution of virustotal with a packer. Entropy 23(4), 395 (2021)
Munkhbayar, B.-E., Kim, T., Li, H., Lee, H.: Dynamic classification of packing algorithms for inspecting executables using entropy analysis. In: 2013 8th International Conference on Malicious and Unwanted Software: “The Americas”(MALWARE), pp. 19–26. IEEE (2013)
Naval, S., Laxmi, V., Gaurm M.S., Vinod, P.: Escape: entropy score analysis of packed executable. In: Proceedings of the Fifth International Conference on Security of Information and Networks, pp. 197–200 (2012)
Naval, S., Laxmi, V., Gaur, M.S., Vinod, P.: Spade: signature based packer detection. In: Proceedings of the First International Conference on Security of Internet of Things, pp. 96–101 (2012)
Naval, S., Laxmi, V., Gaur, M.S., et al.: An efficient block-discriminant identification of packed malware. Sadhana 40(5), 1435–1456 (2015)
networkworld. Chapter 2: Discover what your boss is looking at. 2008. https://www.networkworld.com/article/2271108/chapter-2--discover-what-your-boss-is-looking-at.html?page=2
Noureddine, L., Heuser, A., Puodzius, C., Zendra, O.: SE-PAC: a self-evolving packer classifier against rapid packers evolution. In: Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pp. 281–292 (2021)
NSA’s research directorate. Ghidra (2021). https://ghidra-sre.org/
Okane, P., Sezer, S., McLaughlinm, K.: Detecting obfuscated malware using reduced opcode set and optimised runtime trace. Secur. Inform. 5(1), 1–12 (2016)
Oleh Yuschuk. Ollydbg (2021). https://www.ollydbg.de/
Omachi, R., Murakami, Y.: Packer identification method for multi-layer executables with k-nearest neighbor of entropies. In: 2020 International Symposium on Information Theory and Its Applications (ISITA), pp. 504–508. IEEE (2020)
Oreans: Software protectors (2018). https://www.oreans.com/Themida.php
Oriyano, S.-P.: CEH v9: Certified Ethical Hacker Version 9 Study Guide. Wiley, Hoboken (2016)
Park, L.H., Yu, J., Kang, H.-K., Lee, T., Kwon, T.: Birds of a feature: intrafamily clustering for version identification of packed malware. IEEE Syst. J. 14(3), 4545–4556 (2020)
PEiD. Peid detects most common packers, cryptors and compilers for PE files. (2021) https://github.com/wolfram77web/app-peid
Perdisci, R., Lanzi, A., Lee, W.: Classification of packed executables for accurate computer virus detection. Pattern Recognit. Lett. 29(14), 1941–1946 (2008)
PINdemonium. An unpacker for windows executables exploiting the capabilities of pin. (2021). https://github.com/Phat3/PINdemonium
Raju, A.D., AbuAlhaol, I., Giagone, R.S., Zhou, Y., Shengqiang, H.: A survey on cross-architectural IoT malware threat hunting, IEEE Access (2021)
rays Hex. Ida pro. 2021. https://hex-rays.com/ida-pro/
RDG Soft. Rdg packer detector (2021). http://www.rdgsoft.net/
reversinglabs. Dynamic analysis (2021). https://blog.reversinglabs.com/definitions/dynamic-analysis
Rohleder, R.: Hands-on Ghidra—a tutorial about the software reverse engineering framework. In: Proceedings of the 3rd ACM Workshop on Software Protection, pp. 77–78 (2019)
Saleh, M., Ratazzi, E.P., Xu, S.: A control flow graph-based signature for packer identification. In: MILCOM 2017-2017 IEEE Military Communications Conference (MILCOM), pp. 683–688. IEEE (2017)
Saleh, M., Ratazzi, E.P., Xu, S.: Instructions-based detection of sophisticated obfuscation and packing. In: 2014 IEEE Military Communications Conference, pp. 1–6. IEEE (2014)
Santos, I., Ugarte-Pedrero, X., Sanz, B., Laorden, C., Bringas, P.G.: Collective classification for packed executable identification. In: Proceedings of the 8th Annual Collaboration, Electronic Messaging, Anti-Abuse and Spam Conference, pp. 23–30 (2011)
Shafiq, M.Z., Tabish, S., Farooq, M.: Pe-probe: leveraging packer detection and structural information to detect malicious portable executables. In: Proceedings of the Virus Bulletin Conference (VB), vol. 8 (2009)
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)
Sharif, M., Yegneswaran, V., Saidi, H., Porras, P., Lee, W.: Eureka: a framework for enabling static malware analysis. In: European Symposium on Research in Computer Security, pp. 481–500. Springer, Berlin (2008)
Siglidis, G., Nikolentzos, G., Limnios, S., Giatsidis, C., Skianis, K., Vazirgiannis, M.: Grakel: a graph kernel library in python. J. Mach. Learn. Res. 21, 54–1 (2020)
Singh, A., Arora, R., Pareek, H.: Malware analysis using multiple API sequence mining control flow graph. arXiv preprint arXiv:1707.02691 (2017)
StatistaL Annual number of malware attacks worldwide from 2015 to 2020. 2021. https://www.statista.com/statistics/873097/malware-attacks-per-year-worldwide/
Suk, J.H., Lee, J.-Y., Jin, H., Kim, I.S., Lee, D.H.: Unthemida: commercial obfuscation technique analysis with a fully obfuscated program. Software: Pract Exp. 48(12), 2331–2349 (2018)
Sun, L., Versteeg, S., Boztaş, S., Yann, T.: Pattern recognition techniques for the classification of malware packers. In: Australasian Conference on Information Security and Privacy, pp. 370–390. Springer, Berlin (2010)
Trend Micro: Crypter (2023). https://www.trendmicro.com/vinfo/us/security/definition/crypter
Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: Sok: deep packer inspection: a longitudinal study of the complexity of run-time packers. In: 2015 IEEE Symposium on Security and Privacy, pp. 659–673. IEEE (2015)
Ugarte-Pedrero, X., Santos, I., Bringas, P.G., Gastesi, M., Esparza, J.M.: Semi-supervised learning for packed executable detection. In: 2011 5th International Conference on Network and System Security, pp. 342–346. IEEE (2011)
Ugarte-Pedrero, X., Santos, I., García-Ferreira, I., Huerta, S., Sanz, B., Bringas, P.G.: On the adoption of anomaly detection for packed executable filtering. Comput. Secur. 43, 126–144 (2014)
Ullah, S., Jin, W., Heekuck, O.: Efficient features for function matching in multi-architecture binary executables. IEEE Access 9, 104950–104968 (2021)
Usaphapanus, P., Piromsopa, K.: Classification of computer viruses from binary code using ensemble classifier and recursive feature elimination. In: 2017 Twelfth International Conference on Digital Information Management (ICDIM), pp. 27–31 (2017)
Van Ouytsel, C.-H.B., Given-Wilson, T., Minet, J., Roussieau, J., Legay, A.: Analysis of machine learning approaches to packing detection. arXiv preprint arXiv:2105.00473, 2021
Vidyarthi, D., Damri, G., Rakshit, S., Suthikshn Kumar, C.R., Chansarkar, S.: Classification of malicious process using high-level activity based dynamic analysis. Secur. Priv. 2(6), e86 (2019)
VirusTotal. Yara ( 2021). https://virustotal.github.io/yara/
Zakeri, M., Faraji Daneshga, F., Abbaspour, M.: A static heuristic approach to detecting malware targets. Secur. Commun. Netw. 8(17), 3015–3027 (2015)
Author information
Authors and Affiliations
Contributions
EA wrote and prepared the main manuscript text, and conceptualization. AG contributed to supervision, conceptualization, writing—review, and editing. AHL contributed to supervision, conceptualization, writing—review, and editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could appear to influence the work reported in this paper.
Ethical approval
This article does not contain any study with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Alkhateeb, E., Ghorbani, A. & Habibi Lashkari, A. A survey on run-time packers and mitigation techniques. Int. J. Inf. Secur. 23, 887–913 (2024). https://doi.org/10.1007/s10207-023-00759-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10207-023-00759-y