Abstract
Malware authors do their best to conceal their malicious software to increase its probability of spreading and to slow down analysis. One method used to conceal malware is packing, in which the original malware is completely hidden through compression or encryption, only to be reconstructed at run-time. In addition, packers can be metamorphic, meaning that the output of the packer will never be exactly the same, even if the same file is packed again. As the use of known off-the-shelf malware packers is declining, it is becoming increasingly more important to implement methods of detecting packed executables without having any known samples of a given packer. In this study, we evaluate the use of recurrent neural networks as a means to classify whether or not a file is packed by a metamorphic packer. We show that even with quite simple networks, it is possible to correctly distinguish packed executables from non-packed executables with an accuracy of up to \(89.36\%\) when trained on a single packer, even for samples packed by previously unseen packers. Training the network on more packer raises this number to up to \(99.69\%\).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An epoch is a pass over all training and validation data exactly once.
- 2.
- 3.
Windows 10 Education 32-bit, build 17763.316.
- 4.
\(\texttt {objdump -d <FILENAME>}\).
- 5.
\(\texttt {objdump -Mintel -D --start-address <ENTRY POINT>}\).
References
Objdump. https://sourceware.org/binutils/docs/binutils/objdump.html. Accessed 16 Jan 2020
Radare2. https://www.radare.org/r/. Accessed 16 Jan 2020
Retargetable decompiler. https://retdec.com/. Accessed 8 May 2019
Intel® 64 and IA-32 ArchitecturesSoftware Developer’s Manual, May 2019
Ban, T., Isawa, R., Guo, S., Inoue, D., Nakao, K.: Application of string kernel based support vector machine for malware packer identification. In: The 2013 International Joint Conference on Neural Networks, IJCNN 2013, Dallas, TX, USA, 4–9 August 2013, pp. 1–8. IEEE (2013). https://doi.org/10.1109/IJCNN.2013.6707043
Ban, T., Isawa, R., Guo, S., Inoue, D., Nakao, K.: Efficient malware packer identification using support vector machines with spectrum kernel. In: Eighth Asia Joint Conference on Information Security, AsiaJCIS 2013, Seoul, Korea, 25–26 July 2013, pp. 69–76. IEEE (2013). https://doi.org/10.1109/ASIAJCIS.2013.18
Bat-Erdene, M., Kim, T., Li, H., Lee, H.: Dynamic classification of packing algorithms for inspecting executables using entropy analysis. In: 8th International Conference on Malicious and Unwanted Software: “The Americas”, MALWARE 2013, Fajardo, PR, USA, 22–24 October 2013, pp. 19–26. IEEE Computer Society (2013). https://doi.org/10.1109/MALWARE.2013.6703681
Bat-Erdene, M., Kim, T., Park, H., Lee, H.: Packer detection for multi-layer executables using entropy analysis. Entropy 19(3), 125 (2017). https://doi.org/10.3390/e19030125
Bat-Erdene, M., Park, H., Li, H., Lee, H., Choi, M.-S.: Entropy analysis to classify unknown packing algorithms for malware detection. Int. J. Inf. Secur. 16(3), 227–248 (2017). https://doi.org/10.1007/s10207-016-0330-4
Bergenholtz, E., Casalicchio, E., Ilie, D., Moss, A.: Appendices for: detection of metamorphic malware packers using multilayered LSTM networks (2020). https://github.com/erikbergenholtz/appendix_metamorphic_packers/blob/master/appendix.pdf. Accessed 14 Apr 2020
Blazytko, T., Contag, M., Aschermann, C., Holz, T.: Syntia: syntesizing the semantics of obfuscated code. In: Proceedings of 26 USENIX Security Symposium. Vancouver, BC, Canada, August 2017
Brosch, T., Morgenstern, M.: Runtime packers: the hidden problem. Black Hat USA (2006)
Burgess, C.J., Kurugollu, F., Sezer, S., McLaughlin, K.: Detecting packed executables using steganalysis. In: 5th European Workshop on Visual Information Processing, EUVIP 2014, Villetaneuse, Paris, France, 10–12 December 2014, pp. 1–5. IEEE (2014). https://doi.org/10.1109/EUVIP.2014.7018361
Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations, January 1997. http://www.cs.auckland.ac.nz/staff-cgi-bin/mjd/csTRcgi.pl?serial
The Mental Driller: Metamorphism in practice or “How I made MetaPHOR and what I’ve learnt”, February 2002. https://web.archive.org/web/20070602061547/http://vx.netlux.org/lib/vmd01.html. Accessed 10 Dec 2019
Gupta, N., Naval, S., Laxmi, V., Gaur, M.S., Rajarajan, M.: P-SPADE: GPU accelerated malware packer detection. In: Miri, A., Hengartner, U., Huang, N., Jøsang, A., García-Alfaro, J. (eds.) 2014 Twelfth Annual International Conference on Privacy, Security and Trust, Toronto, ON, Canada, 23–24 July 2014, pp. 257–263. IEEE Computer Society (2014). https://doi.org/10.1109/PST.2014.6890947
holy\_father: Morphine v2.7 (2004). https://github.com/bowlofstew/rootkit.com/tree/master/hf/Morphine27. Accessed 24 Oct 2018
Hubballi, N., Dogra, H.: Detecting packed executable file: supervised or anomaly detection method? In: 11th International Conference on Availability, Reliability and Security, ARES 2016, Salzburg, Austria, 31 August–2 September 2016, pp. 638–643. IEEE Computer Society (2016). https://doi.org/10.1109/ARES.2016.18
Jeong, G., Choo, E., Lee, J., Bat-Erdene, M., Lee, H.: Generic unpacking using entropy analysis. In: 5th International Conference on Malicious and Unwanted Software, MALWARE 2010, Nancy, France, 19–20 October 2010, pp. 98–105. IEEE Computer Society (2010). https://doi.org/10.1109/MALWARE.2010.5665789
Kancherla, K., Donahue, J., Mukkamala, S.: Packer identification using byte plot and markov plot. J. Comput. Virol. Hacking Tech. 12(2), 101–111 (2016). https://doi.org/10.1007/s11416-015-0249-8
Křoustek, J., Matula, P., Kolár, D., Zavoral, M.: Advanced preprocessing of binary executable files and its usage in retargetable decompilation. Int. J. Adv. Softw. 7(1), 112–122 (2014)
Lee, W.Y., Saxe, J., Harang, R.: SeqDroid: obfuscated android malware detection using stacked convolutional and recurrent neural networks. In: Alazab, M., Tang, M.J. (eds.) Deep Learning Applications for Cyber Security. ASTSA, pp. 197–210. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-13057-2_9
Morgenstern, M., Pilz, H.: Useful and useless statistics about viruses and anti-virus programs. In: Proceedings of the CARO Workshop (2010)
Naval, S., Laxmi, V., Gaur, M.S., Vinod, P.: SPADE: Signature based PAcker DEtection. In: Chandrasekhar, R., Tanenbaum, A.S., Rangan, P.V. (eds.) First International Conference on Security of Internet of Things, SECURIT 2012, Kollam, India, 17–19 August 2012. pp. 96–101. ACM (2012). https://doi.org/10.1145/2490428.2490442
Rolles, R.: Unpacking virtualization obfuscators. In: Proceedings of USENIX WOOT. Montreal, Canada, August 2009
Sun, L., Versteeg, S., Boztaş, S., Yann, T.: Pattern recognition techniques for the classification of malware packers. In: Steinfeld, R., Hawkes, P. (eds.) ACISP 2010. LNCS, vol. 6168, pp. 370–390. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14081-5_23
Szor, P.: The Art of Computer Virus Research and Defense. Addison-Wesley, Boston, Massachusetts (2005)
Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: SoK: Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers. In: 2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, 17–21 May 2015, pp. 659–673. IEEE Computer Society (2015). https://doi.org/10.1109/SP.2015.46
Xie, P., Liu, X., Yin, J., Wang, Y.: Absent extreme learning machine algorithm with application to packed executable identification. Neural Comput. Appl. 27(1), 93–100 (2014). https://doi.org/10.1007/s00521-014-1558-4
Yan, W., Zhang, Z., Ansari, N.: Revealing packed malware. IEEE Secur. Priv. 6(5), 65–69 (2008). https://doi.org/10.1109/MSP.2008.126
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix
Recall and Precision
Recall (top) and precision (bottom) of model trained on the individual packers, when evaluated against only the packer included in the training set (Self), and all packers included in the study (All).
Recall (top) and precision (bottom) of model trained on N-1 packers, when evaluated against only the packer excluded in the training set (Excluded), and all packers included in the training set (All).
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Bergenholtz, E., Casalicchio, E., Ilie, D., Moss, A. (2020). Detection of Metamorphic Malware Packers Using Multilayered LSTM Networks. In: Meng, W., Gollmann, D., Jensen, C.D., Zhou, J. (eds) Information and Communications Security. ICICS 2020. Lecture Notes in Computer Science(), vol 12282. Springer, Cham. https://doi.org/10.1007/978-3-030-61078-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-61078-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61077-7
Online ISBN: 978-3-030-61078-4
eBook Packages: Computer ScienceComputer Science (R0)