Abstract
In this paper, we aim to accomplish malware classification using word embeddings. Specifically, we trained machine learning models using word embeddings generated by BERT. We extract the “words” directly from the malware samples to achieve multi-class classification. In fact, the attention mechanism of a pre-trained BERT model can be used in malware classification by capturing information about the relation between each opcode and every other opcode belonging to a specific malware family. As means of comparison, we repeat the same experiments with Word2Vec. Differently than BERT, Word2Vec generates word embeddings where words with similar context are considered closer, being able to classify malware samples based on similarity. As classification algorithms, we used and compared Support Vector Machines (SVM), Logistic Regression, Random Forests, and Multi-Layer Perceptron (MLP). We found that the classification accuracy obtained by the word embeddings generated by BERT is effective in detecting malware samples, and superior in accuracy when compared to the ones created by Word2Vec.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yara Awad, Mohamed Nassar, and Haidar Safa. Modeling malware as a language. In 2018 IEEE International Conference on Communications (ICC), pages 1–6, 2018.
P. Baldi and Y. Chavin. Smooth on-line learning algorithms for hidden markov models. Neural Computation, 6:307–318, 1994.
S. Banerjee. Word2vec — a baby step in deep learning but a giant leap towards natural language processing. https://laptrinhx.com/word2vec-a-baby-step-in-deep-learning-but-a-giant-leap-towards-natural-language-processing-3998188269/, 2018.
S. Basole, F. Di Troia, and M. Stamp. Multifamily malware models. Journal of Computer Virology and Hacking Techniques, 16:79–92, 2020.
D. Bilar. Opcodes as predictor for malware. Int. J. Electron. Secur. Digit. Forensic, 1(2):156–168, January 2007.
L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
Aniket Chandak, Wendy Lee, and Mark Stamp. A comparison of word2vec, hmm2vec, and pca2vec for malware classification. https://arxiv.org/abs/2103.05763, 2021.
K. Clark, U. Khandelwal, O. Levy, and C. Manning. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy, August 2019. Association for Computational Linguistics.
HuggingFace. Distilbert. https://huggingface.co/transformers/model_doc/distilbert.html.
Microsoft Security Intelligence. Renos. https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=TrojanDownloader:Win32/Renos&threatId=16054, 2006.
Microsoft Security Intelligence. Ceeinject. https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=VirTool%3AWin32%2FCeeInject, 2007.
Microsoft Security Intelligence. Onlinegames. https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=PWS%3AWin32%2FOnLineGames, 2008.
Microsoft Security Intelligence. Winwebsec. https://www.microsoft.com/security/portal/threat/encyclopedia/entry.aspx?Name=Win32%2fWinwebsec, 2010.
Microsoft Security Intelligence. Fakerean. https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=Win32/FakeRean, 2011.
Samuel Kim. Pe header analysis for malware detection. Master’s thesis, San Jose State University, Department of Computer Science, 2018.
C. McCormick. Word2vec tutorial - the skip-gram model. http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model, 2016.
W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943.
C. Mihai and J. Somesh. Testing malware detectors. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’04, page 34–44, New York, NY, USA, 2004. Association for Computing Machinery.
Fred C. Pampel. Logistic Regression: A Primer. SAGE Publications, Inc., 2000.
H. Ramchoun, M. A. J. Idrissi, Y. Ghanou, and M. Ettaouil. Multilayer perceptron: Architecture optimization and training. Int. J. Interact. Multim. Artif. Intell., 4(1):26–30, 2016.
N. Ranjan, K. Mundada, K. Phaltane, and S. Ahmad. A survey on techniques in nlp. International Journal of Computer Applications, 134(8):6–9, 2016.
sklearn. Gridsearchcv. https://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.
SophosLabs. Sophos 2021 threat report. https://www.sophos.com/en-us/medialibrary/pdfs/technical-papers/sophos-2021-threat-report.pdf, 2021.
Mark Stamp. Introduction to Machine Learning with Applications in Information Security. Chapman and Hall/CRC, 2020.
Symantec. Internet security threat report: Malware. https://interactive.symantec.com/istr24-web, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. https://arxiv.org/abs/1706.03762, 2017.
S. Vemparala, F. Di Troia, C. Visaggio, T. Austin, and M. Stamp. Malware detection using dynamic birthmarks. In Proceedings of the 2016 ACM on International Workshop on Security And Privacy Analytics, IWSPA ’16, page 41–46, New York, NY, USA, 2016. Association for Computing Machinery.
P. Vinod, R. Jaipur, R. Laxmi, and M. Gaur. Survey on malware detection methods. In Proceedings of the 3rd Hackers Workshop on Computer and Internet Security, pages 74–79, 2009.
M. Wadkar, F. Di Troia, and M. Stamp. Detecting malware evolution using support vector machines. Expert Systems with Applications, 143:113022, 2020.
W. Wong and M. Stamp. Hunting for metamorphic engines. Journal of Computer Virology and Hacking Techniques, 2:211–229, 2017.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Alvares, J., Troia, F.D. (2022). BERT for Malware Classification. In: Stamp, M., Aaron Visaggio, C., Mercaldo, F., Di Troia, F. (eds) Artificial Intelligence for Cybersecurity. Advances in Information Security, vol 54. Springer, Cham. https://doi.org/10.1007/978-3-030-97087-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-97087-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-97086-4
Online ISBN: 978-3-030-97087-1
eBook Packages: Computer ScienceComputer Science (R0)