MORTON: Detection of Malicious Routines in Large-Scale DNS Traffic

Abstract

We present MORTON, a method that identifies compromised devices in enterprise networks based on the existence of routine DNS communication between devices and disreputable host names. With its compact representation of the input data and use of efficient signal processing and a neural network for classification, MORTON is designed to be accurate, robust, and scalable. We evaluate MORTON using a large dataset of corporate DNS logs and compare it with two recently proposed beaconing detection methods aimed at detecting malware communication. The results demonstrate that while MORTON ’s accuracy in a synthetic experiment is comparable to that of the other methods, it outperforms those methods in terms of its ability to detect sophisticated bot communication techniques, such as multistage channels. Additionally, MORTON was the most efficient method, running at least 13 times faster than the other methods on large-scale datasets, thus reducing the time to detection. In a real-world evaluation, which includes previously unreported threats, MORTON and the two compared methods were deployed to monitor the (unlabeled) DNS traffic of two global enterprises for a week-long period; this evaluation demonstrates the effectiveness of MORTON in real-world scenarios where it achieved the highest F1-score.

Appendices

Appendix A Detecting Multiple Host Names

The primary drawback of bot communication techniques that use a single host name (e.g., malware beaconing technique with single host name C&C communication) is their lack of robustness. The single host name is effectively a single point of failure, and if the host name is unavailable for any reason, the attackers cannot control their bots. An additional drawback is that communication with a single host name may be less covert. For instance, in DNS data exfiltration, every exfiltration message is sent to an attacker’s host name. A single host name that receives a large volume of exfiltration messages is more detectable by security systems [28]. Therefore, the activities of bots that split their DNS exfiltration messages and send them to multiple host names are less suspicious.

The most well-known use of multiple host names for botnet communication is through domain generation algorithms [26] (DGAs), which are used by over 40 known botnets [29]. Most bots that use DGAs generate new domain names on a daily basis [29], thus pointing to the importance of detecting bot communication that uses multiple domain names.

Multistage channels (MSC) are another bot communication technique in which multiple host names are used. The initial installation of the bot on a compromised device is referred to as the first stage of the infection. Throughout the first stage, the bot communicates with its C&C through either a single host name or multiple host names. However, the host names will change when the first stage bot requires an upgrade. A bot upgrade typically involves communicating with a new host name to download a module that enhances the bot’s capabilities. The process of upgrading the bot is referred to as the second stage of the infection. The MSC bot communication technique often involves several stages, where multiple host names are gradually upgrading the bot. The use of MSC improves the robustness of a botnet’s infrastructure, because security researchers cannot easily identify the different host names that will be used by a botnet in order to shut down its operation (i.e., prevent bots from upgrading).

Other cases of bot communication techniques in which multiple host names are used include fallback channels and multihop proxies [26]. In fallback channels, a bot that fails to communicate with its C&C host name attempts to communicate to the host name next in line, based on a prioritized list of host names. Multihop proxies is a bot communication technique in which the C&C channel is established through a series of proxy servers that are associated with different host names. The series of proxy servers between bots and their C&C servers prevents security researchers from easily matching a bot communicating with its C&C server based on network logs. MORTON is designed to detect every multiple host communication technique mentioned, as long as it is used in a periodic manner.

Appendix B Neural Network Parameters

The architecture and the learning rate were selected, because they performed best with regard to the area-under-curve metric when compared against more than 25 alternative architectures originating from an ablation study and the use of AutoML for structured data [19], as can be seen in Table 4. (note that all of the settings were trained and evaluated on a smaller subset of the data to reduce training time).

Table 4. Neural network parameters