Joint detection of malicious domains and infected clients
Detection of malware-infected computers and detection of malicious web domains based on their encrypted HTTPS traffic are challenging problems, because only addresses, timestamps, and data volumes are observable. The detection problems are coupled, because infected clients tend to interact with malicious domains. Traffic data can be collected at a large scale, and antivirus tools can be used to identify infected clients in retrospect. Domains, by contrast, have to be labeled individually after forensic analysis. We explore transfer learning based on sluice networks; this allows the detection models to bootstrap each other. In a large-scale experimental study, we find that the model outperforms known reference models and detects previously unknown malware, previously unknown malware families, and previously unknown malicious domains.
KeywordsMachine learning Neural networks Computer security Traffic data Https traffic
The work of Tomáš Pevný has been partially funded by Czech Ministry of education under the GACR project 18-21409S. We would like to thank Virustotal.com for their kind support.
Funding was provided by Cisco R&D.
- Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., & Zheng, X. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from https://www.tensorflow.org/. Accessed 6 Sept 2018.
- Argyriou, A., Evgeniou, T., & Pontil, M. (2007). Multi-task feature learning. In B. Schölkopf, J. C. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems 19 (pp. 41–48). MIT Press.Google Scholar
- Arora, A., Garg, S., & Peddoju, S. K. (2014). Malware detection using network traffic analysis in android based mobile devices. In International conference on next generation mobile apps, services and technologies (pp. 66–71).Google Scholar
- Bartos, K., & Sofka, M. (2015). Robust representation for domain adaptation in network security. In European conference on machine learning and principles and practice of knowledge discovery in databases (pp. 116–132). Springer.Google Scholar
- Bartos, K., Sofka, M., & Franc, V. (2016). Optimized invariant representation of network traffic for detecting unseen malware variants. In USENIX security symposium (pp. 807–822).Google Scholar
- Bickel, S., Bogojeska, J., Lengauer, T., & Scheffer, T. (2008). Multi-task learning for hiv therapy screening. In Proceedings of the international conference on machine learning (pp. 56–63). ACM.Google Scholar
- Blum, S. B., & Lueker, J. (2001). Transparent proxy server, January 30. US Patent 6,182,141.Google Scholar
- Caruana, R. (1993) Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the international conference on machine learning.Google Scholar
- Chollet, F., et al. (2015). Keras. https://keras.io. Accessed 6 Sept 2018.
- Demontis, A., Melis, M., Biggio, B., Maiorca, D., Arp, D., Rieck, K., et al. (2018). Yes, machine learning can be more secure! a case study on android malware detection. IEEE Transactions on Dependable and Secure Computing. https://doi.org/10.1109/TDSC.2017.2700270.
- Duong, L., Cohn, T., Bird, S., & Cook, P. (2015). A neural network model for low-resource universal dependency parsing. In Proceedings of the conference on empirical methods in natural language processing (pp. 339–348).Google Scholar
- Finkel, J. R., & Manning, C. D. (2009). Hierarchical bayesian domain adaptation. In Proceedings of ACL human language technologies (pp. 602–610).Google Scholar
- Finley, K. (2017). Half the web is now encrypted. That makes everyone safer. Wired. https://www.wired.com/2017/01/half-web-now-encrypted-makes-everyone-safer/.
- Franc, V., Sofka, M., & Bartos, K. (2015). Learning detector of malicious network traffic from weak labels. In A. Bifet, M. May, B. Zadrozny, R. Gavalda, D. Pedreschi, F. Bonchi, J. Cardoso, & M. Spiliopoulou (Eds.), Machine learning and knowledge discovery in databases (pp. 85–99). Springer.Google Scholar
- Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017). Convolutional sequence to sequence learning. arXiv:1705.03122.
- Kogan, R. (2015). Bedep trojan malware spread by the angler exploit kit gets political. Spider Labs Blog. https://www.trustwave.com/Resources/SpiderLabs-Blog/Bedep-trojan-malware-spread-by-the-Angler-exploit-kit-gets-political/. Accessed 6 Sept 2018.
- Kohout, J., & Pevny, T. (2015a) Automatic discovery of web servers hosting similar applications. In Proceedings of the IFIP/IEEE international symposium on integrated network management.Google Scholar
- Kohout, J., & Pevny, T. (2015b). Unsupervised detection of malware in persistent web traffic. In Proceedings of the IEEE international conference on acoustics, speech and signal processing.Google Scholar
- Lashkari, A., Kadir, A., Gonzalez, H., Mbah, K., & Ghorbani, A. (2015). Towards a network-based framework for android malware detection and characterization. In Proceedings international conference on privacy, security, and trust.Google Scholar
- Li, L., Jamieson, K. G., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016). Efficient hyperparameter optimization and infinitely many armed bandits. CoRR. arXiv:1603.06560.
- Lokoč, J., Kohout, J., Čech, P., Skopal, T., & Pevnỳ, T. (2016). k-NN classification of malware in HTTPS traffic using the metric space approach. In M. Chau, G. A. Wang, & H. Chen (Eds.), Intelligence and security informatics (pp. 131–145). Springer.Google Scholar
- Long, M., & Wang, J. (2015). Learning multiple tasks with deep relationship networks. In arXiv:1506.02117.
- Malik, J., & Kaushal, R. (2016). CREDROID: Android malware detection by network traffic analysis. In Proceedings of the first ACM workshop on privacy-aware mobile computing (pp. 28–36). ACM.Google Scholar
- Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems 26 (pp. 3111–3119). Curran Associates, Inc.Google Scholar
- Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. (2016). Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3994–4003).Google Scholar
- Nelms, T., Perdisci, R., & Ahamad, M. (2013). Execscent: Mining for new C&C domains in live networks with adaptive control protocol templates. In Proceedings of the USENIX security symposium.Google Scholar
- Pascanu, R., Stokes, J. W., Sanossian, H., Marinescu, M., & Thomas, A. (2015). Malware classification with recurrent networks. In Proceedings of the IEEE international conference on acoustics, speech and signal processing (pp. 1916–1920). IEEE.Google Scholar
- Pevny, T., & Somol, P. (2016). Discriminative models for multi-instance problems with tree structure. In Proceedings of the international workshop on artificial intelligence for computer security.Google Scholar
- Prasse, P., Machlica, L., Pevný, T., Havelka, J., & Scheffer, T. (2017). Malware detection by analysing network traffic with neural networks. In Proceedings of the European conference on machine learning.Google Scholar
- Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. (2017). Sluice networks: Learning what to share between loosely related tasks. arXiv:1705.08142v1 [stat.ML]
- Swinnen, A., & Mesbahi, A. (2014). One packer to rule them all: Empirical identification, comparison and circumvention of current antivirus detection techniques. BlackHat USA. https://www.blackhat.com/docs/us-14/materials/us-14-Mesbahi-One-Packer-To-Rule-Them-All-WP.pdf.
- Yang, Y., & Hospedales, T. M. (2016). Trace norm regularised deep multi-task learning. arXiv:1606.04038.