Mac-A-Mal: macOS malware analysis framework resistant to anti evasion techniques

Abstract

With macOS increasing popularity, the number, and variety of macOS malware are rising as well. Yet, very few tools exist for dynamic analysis of macOS malware. In this paper, we propose a macOS malware analysis framework called Mac-A-Mal. We develop a kernel extension to monitor malware behavior and mitigate several anti-evasion techniques used in the wild. Our framework exploits the macOS features of XPC service invocation that typically escape traditional mechanisms for detection of children processes. Performance benchmarks show that our system is comparable with professional tools and able to withstand VM detection. By using Mac-A-Mal, we discovered 71 unknown adware samples (8 of them using valid distribution certificates), 2 keyloggers, and 1 previously unseen trojan involved in the APT32 OceanLotus.

1 Introduction

Contrary to popular belief, the Mac ecosystem is not unaffected by malware. In 2014, the first known ransomware appeared, and other ransomware has been discovered as Software-as-a-Service (SaSS), where malware is available as requests. Mac devices saw more malware attacks in 2015 than the past five years combined, according to a cyber-security report from the Bit9 and Carbon Black Threat Research team.¹ In 2016, Mac malware grew 744% with around 460,000 instances detected, says McAfee report [1] and increases 270% between 2016 and 2017 (Table 1).²

There exist tools which support malware analysis of Windows, Linux or Android applications, while, investigation of macOS malware and development of tools supporting monitoring their behavior is still limited in functionalities or anti-analysis resistance, or both. For example, the open source Mac-sandbox [2] is vulnerable to anti-analysis techniques such as Dylib name verification. Cuckoo sandbox [3] does not support anti-analysis mitigation and human interaction under the macOS environment. The closed source FireEye monitor³ use a kernel extension which is resistant to anti-analysis techniques, but requires human intervention. VirusTotal Box of Apples sandbox⁴ executes malware to show screenshots of what an analyst would see, also reports network traffic and file operations but the underlying technology itself is enclosed.

Our goal is to design and implement a malware analysis framework, which can automatically capture malware behavior in an adversary environment, called Mac-A-Mal. It consists of two main modules implemented at user-space and kernel-space. At the kernel level, we implement system call hooking and process tracing techniques to capture system calls and their arguments. During sample execution, the analysis engines are customized to prevent analysis traces left on the system while maximizing malware behavior exposure using memory patching and virtual machine hardening techniques. At the user level, we make specific handlers to deal with various file types.

Using Mac-A-Mal, we discovered 71 unknown Adware under 8 legitimate certificates, 2 keyloggers, and 1 trojan involved in the APT32 OceanLotus, which are unknown to many Anti-virus vendors for a long time. Only Ikarus and ESET-NOD32 products can recognize two of these unknown samples at the time of finding. Further, we observed that many of them are aimed with sophisticated anti-analysis techniques like one avoiding the bash execution to stay under the radar. After a few days since our analysis reports have published on social media, may anti-virus vendors had updated their engines to be able to detect these unknown samples (Table 2).

Table 1 Supported files of macOS analysis sandboxes

Table 2 Comparative analysis of competing solutions

In Sect. 2, we provide some background on macOS security. Then we review related literature on macOS malware analysis and detection in Sect. 3. In Sect. 4 we describe details of our approach. The anti-evasion techniques analysis and mitigation are presented in Sect. 5. Comparison of Mac-A-Mal and other dynamic analysis tools are conducted in Sect. 6. In Sect. 7, we show case studies of analyzing malware family based on our Mac-A-Mal. In Sect. 8 We presents a summary of our approach and highlight future directions in Sect. 9.

2 Background on OSX kernel and application security

The basic principles of OSX Security follow the Common Data Security architecture, which consists of three layers: firmware, kernel-space, and user-space security [4]. Each layer contains specific features to secure data. In this paper, we only discuss the security properties of kernel-space and user-space layers. Some relevant firmware attacks are: Mac EFI rootkits [5] with boot device replacement techniques or ThunderStrike [6] permanently modifies firmware modifications. For other security aspects of components in these aforementioned layers, we refer [7].

Mac kernel is built from BSD and Mach. There are two main features in Kernel-space security: Portable Operating System Interface (POSIX) and Apple security features. The BSD itself define access policies to system artifacts (e.g., files, computing resources) based on user and group IDs at various levels. Virtual memory pages are encrypted to prevent memory allocation exploitation attacks. These attacks are widely used by malware to force the processor to execute arbitrary code from another process’ memory area. Address Space Layout Randomization (ASLR) is adopted to prevent exploitation by randomly offsetting location of modules and specific in-memory structures.

System Integrity Protection (SIP) is a built-in feature introduced in OS X El Captian to protect system files and directories against the modifications caused by non-“entitlement” processes. It also keeps a process from code injection and runtime attachment attacks. A runtime debugging (e.g., DTrace) cannot interact with a protected process because SIP denies any attempt to load the unsigned kernel extensions (KEXT). Using KEXT as a device driver is a common technique of rootkit.

Sandboxing techniques restrict untrusted applications under the controlled environment such as files they can access or the ability to connect the internet. The built-in Mac Sandbox system consists of user-space library functions to initialize and configure environments for different processes, a Mach server which is responsible for handling logs from the kernel, a kernel extension that enforces security policies, and a kernel extension which provides regular expression matching for policy enforcement [8].

macOS uses its built-in XPCProxy tool to spawn new services. The child processes of Apple applications are often spawned via XPC services, which do not inherit their parent’s environment.⁵ Security monitoring tools should keep track of “XPC children.” /usr/libexec/xpcproxy is a “trampoline” that’s used by launchd to launch XPC services using posix_spawnp() system call. It uses xpc_pipe_routine() to communicate back to launchd to get the information needed to spawn the XPC service – notably the path to the XPC service executable and the values that should be set in its environment.

The Code signing mechanism uses a digital signature used to verify the software’s origin and integrity in case binaries were subverted. Built-in Anti-malware XProtect scan suspicious files by matching their signatures with malware definitions in the Xprotect.plist file. Gatekeeper prevents applications that are downloaded and installed from malicious websites. It suffers from the same restriction of XProtect [9]. In particular, Malware that signed with a valid Apple certificate can be able to bypass Gatekeeper. Any modification that invalidates the signature will cause the binary crash. However, malware authors can unsign an application, infect it with malicious code, and then execute it without crashing [8]. In contrast, all kernel extensions are required to be signed to prevent malicious code in kernel space. Moreover, the verification step is performed in user mode, and therefore malware author can turn on user-mode checking (on their signature) and load their kernel extensions.

Table 3 Survey of binary analysis frameworks for macOS executable

Table 4 Survey of sandboxes

Table 5 Mac-A-Mal system calls hook

3 Related work

Static analysis tools demonstrated in Table 3 are common commercial and opensource disassemblers used by analysts to extract static information of malware such as file header, code disassembly. The commercial IDA Pro, Hopper and Radare2 support more architectures than the others. They also provide bindings for scripting languages such as Python. Static features can be feed to machine learning algorithms to automatically classify malware samples in [4, 10]. They extracted binary header, load commands, and segments features from the dataset of 2300 benign and 760 malicious samples and achieved 96.62% accuracy.

Dynamic analysis Existing tools shown in Table 2 proposed to monitor dynamic artifacts in the system such as processes in user-space and kernel-space. FireEye monitor collects information about processes, file and network activities, TTY events from the kernel-space. It, however, lacks any support of analysis automation for large-scale analysis. DTrace logs program execution trace (e.g., system calls, CPU performance) using probes callbacks from the user-space. Lindorfer [11] has used the tool for their analysis of honeypot captured samples. Moreover, it has three limitations: (i) requiring root privilege to run samples, (ii) inability to monitor official Apple binaries without placing them in a temporary folder and (iii) fail to follow a new process or fork triggered by launchd. The limitation (i) has been addressed by the opensource Cuckoo sandbox [3] via two running modes: executing DTrace as root permission or dropping privileges to the user with sudo command and alter the local sudoers file to reflect the change. This solution, however, appears to be trivial to detect a process executing under root privilege, analysts, therefore, must issue an additional sudo to drop the privilege. As a result, the analysis trace might unwillingly include benign sudo activities which confuse analysts. Only filtering them out would lead to significant information missing. Our observation in Table 6 shows that Mac-A-Mal found 31.03% samples of the dataset exhibits super-user permission (sudo) requests.Mac-Sandbox is an extension of Cuckoo sandbox, which uses Dylib hijacking [12] to suspend process and hook system calls of particular interest. Compared to Cuckoo, it extended the static analysis and human interaction simulation. It, however, seems to leave analysis traces on the system caused by its suspending and resuming operations, and thereby highly be vulnerable to anti-analysis techniques. Moreover, relying on the parent-child process tracing mechanism at user-space, it likely neglects malicious processes spawned by system services.

Memory analysis using Volatility plugin [13] involves inspection of relevant macOS subsystems and volatile artifacts lie in the memory. J. Rentzsch [14] uses function overriding and code injection to execute the hooking code. They patch the target function to jump to a dynamically allocated executable code area (Table 4).

4 General system architecture

Figure 1 illustrates our Mac-A-Mal system design. The analysis machine submits samples to the monitor analysis machine. To prevent analysis information leak and infect to the analyst’s host machine, we turn off data sharing features on the guest machine. To run applications from the Internet, we disable Gatekeeper feature and remove built-in XProtect malware detection as well as disable default Apple protection mechanisms to capture malware behavior explicitly. The Analyzer component of the analysis machine is responsible for distributing samples to one or more monitor machines and display analysis results to users. The monitor machine executes the examples and collect system call, process, file operation logs, and send them to the analysis machine. After the execution ends, the monitor machine is restored to a clean state by using snapshot mechanisms.

Using default Mac handler - /usr/bin/open and XPCProxy, so we can open and trace up to at least 15 file types as shown in Table 1. It supports more file types than other sandboxes by executing samples via default open handler mechanism, which is similar to how users open a file by double click. The challenge is that open spawns the new process which goes through the mac_syscall interface that prevents any tracing attempts regard to sandbox policy. Apparently, by tracing in kernel space, we can execute any samples in which DTrace and MacSandbox failed to. It means any types of files can be opened with the capability of process forks tracing (e.g., Office documents) without using other file handers. We also classify file types for different ways of handling, for example, archives such as Apple disk image, we firstly mount via hdiutil, and then mac-A-Mal executes all applications inside the attached volume. For cross-platform malware variants using scripting application such as Java, Perl, Python, we use their own interpreters to execute (e.g. CrossRAT malware).

Then System call logging module will capture system calls using hooks at the kernel-space. By doing so, we monitor raw malware behaviors. Our module finds system call location in the sysent table by three steps: first we un-slide KASLR address to obtain kernel base image address, then walk through the kernel base image to locate kernel’s load command segment, and finally perform system table lookups. Once the location of the sysent table is known, we replace its entries with our defined callback functions. As a result, when a malware invokes a system call, it will query the address of system call in the BSD kernel syscall table which matches our callback function address. The callback function then performs execution logging, malware evasion detection, and mitigation, and decide proper post-processing for the calls (forward or drop). Table 5 shows a list of system calls that are hooked by callbacks by categories, which can be easily extended by writing custom callback functions. With SIP enabled, the compiled KEXTs of our module must be signed with a developer ID for signing KEXTs certificate.

Fig. 1

Mac-A-Mal framework design

We further trace processes to give insights into their characteristics, the interaction between them and exclude irrelevant processes. We capture system calls, their arguments, and program path by making a tunnel to transfer data between user and kernel side as depicted in Fig. 2. In particular, the challenge is to transfer arguments which are typical arrays.⁶ Mac-A-Mal hooks system calls at kernel space and therefore can trace process spawns by open, launchd, and xpcproxy processes accurately, while DTrace, Cuckoo sandbox, and Macsandbox cannot trace. Information about process creation is collected by filtering MAC framework Kernel Authorization (KAuth)’s scope KAUTH_FILEOP_EXEC for process notifications. In particular, KAuths calls a defined callback function which logs a Vnode reference of the running program. We collect its full path, process ID (PID), and parent ID (PPID) in each process. The processes are visualized the tree structure that displays child and parent relationships. By Analyzing process tree, we can be able to identify suspicious processes such as orphaned processes.

Fig. 2

Process argument logging

Mac-A-Mal captures and automatically dissects arguments of shipped XPC-Proxy component and posix_spawn() function described in Fig. 3. To capture parent-child relationship among processes (see 2 on XPCProxy) The framework can keep track of the information of XPC children by using XPCproxy. In particular, xpc_pipe_routine() communicate back to launchd to get the information of the spawn XPC service such as the path to the service executable and environmental values. Then XPCproxy calls posix_spawn() function with the POSIX_SPAWN_SETEXEC flag to execute the XPC service. The general logging scheme optimizes the logger via internal sockets which is thread-safety and synchronous. Also, during malware execution, the extension does not leave any explicit traces on the system in comparison to previous approaches ([15, 16]).

Fig. 3

Mac-A-Mal XPCProxy tracing

Automation The communication protocol between the analysis and monitoring machine is designed to be reliable and straightforward. In particular, Mac-A-Mal sends data to user-space via a socket, which is thread-safe, synchronous, and fast. A thread is created to handle messages over the socket and wait for system call events. Once a system call is issued, the socket waked and started sending data to the daemon. The daemon then receives data and selects the next action for the process, such as doing a start or stop. Following steps describe the workflow of Mac-A-Mal sandbox: (i) Restore snapshots, (ii) Download sample(s), (iii) Static analysis, (iv) Sample execution, (v) Suspension, (vi) Produce reports.

Malware persistence mechanisms analysis and detection Malware employ persistence mechanisms to be hidden in the system for a long time. An identification of persistence indicators can be useful to fingerprint malware if it is unique enough. Simple malware can modify Unix startup files, install malware as a launchd daemon [17]. Advanced malware can leverage cron jobs, dynamic libraries hijacking, MachO binary infection, and application-specific installation (e.g., Safari, Google Chrome, iTunes,etc.) [9].

Human interaction simulation: The framework constantly captures changes in the analysis screen. The Quartz graphics layer library is used to detect interaction windows and react keyboard and mouse events. First, screenshot images are analyzed to seek for login forms or confirmation buttons. Mostly, Mac applications ask for superuser credentials, user agreement, installation location, etc. Mac-A-Mal passes username, password, and mouse clicks using Quartz’s KeyboardEvents and MouseEvents.

5 Robust anti-evasion techniques

Mitigating anti-debugging techniques Apple introduces two standard methods to detect and prevent applications from being attached by debuggers. Sysctl() retrieves information about a process and determine its debugging status. Furthermore, PT_DENY_ATTACH protects a process from debuggers by using non-standard ptrace requests. Any attempts to trace the process will receive the exit status or a segmentation violation. We place a monitor module in the kernel-space to capture and analyze both sysctl() and ptrace anti-debug calls and their arguments.

Mitigating Anti-Virtual machine techniques Analysts usually disable SIP to enable kernel debugging, load unsigned kernel extension, or remove DTrace restrictions. While these deactivation techniques allow bypassing constraints by Apple, their footprints can be leveraged by malware to detect the analysis machine. Malware can exploit the difference in cycles time between real and virtual processor to identify a virtual environment (Locky and KeRanger ransomware). We detect anti-virtual machine techniques by hooking and replacing csrctl() function call which is used to verify SIP. We catch attempts to gather environment artifacts such as configuration files, processor specifications, running processes and randomly change machine settings (e.g., MAC address, BIOS ID) to make malware harder to detect. Figure 4 shows how our memory patching technique is implemented to adjust CPU and SIP configuration information for mitigating anti-analysis malware⁷

Fig. 4

Mac-A-Mal anti-environment evasion module using hook and memory patching. (1) A system call is issued in the user space, (2) Hook the system call, (3) Return the control to the original system call, (4) Returns value to the user space and manipulate system call arguments if anti-evasion attempts are detection, (5) The return value is logged for further analysis using Logger component, (6) returns values from the kernel space is also to the analyzer in user-space, (7) the exit() system call, the is not hooked and forward back to the user-space

6 Performance evaluation

We first analyze the performance of Mac-A-Mal as it needs to be close to bare performance as much as possible on the other hand. For the comparison, we select the opensource BYTE’s Unix Benchmark Suite due to its controllability, repeatability, and durability. To make the suite compatible with macOS platform, we rebuilt it with some modifications. Read Timestamp Counter (RDTSC) instruction is used to determine how quickly the processor executes program’s instructions. It returns the number of ticks counted since the last reboot. Slowness in the processor’s execution might indicate the presence of malware analysis tools, such as a debugger. Hence, the faster an analysis tool runs, the less chance it has of getting detected by malware. We run Mac-A-Mal on the benchmark of 1000 system calls for 100 times and calculate the overall execution time. Figure 6 demonstrates that Mac-A-Mal performs faster than other competitors. In particular, Mac-A-Mal completes the benchmark with only 9.934 and 20,363 number of cycles on stdout writing and localhost connection operations, respectively.

Figure 5 depicts that Mac-A-Mal’s tracing time on the same benchmark, our tool takes only 1.1 seconds to complete the benchmark which is approximately 16 times faster than Macsandbox and 15 times than Cuckoo sandbox (Fig. 6).

Fig. 5

We benchmark every system tracting time against a Macbook Pro running samples naturally without extra hooking/monitoring techniques

Fig. 6

We benchmark every system against a MacBook Pro running samples naturally without any extra hooking/monitoring techniques. Hence that is the best performance that one can achieve. Their baseline took 19.013 and 5.686 number of cycles to write to stdout and connect to the localhost, respectively. Note that anti-debugging is not required for Mac-A-Mal and FireEye Monitor

Table 6 Analysis of malware behavior and techniques used by examples in the dataset

Fig. 7

Behavior heatmap of macOS malware in 2017. The behaviors can be grouped as following categories: (i) persistence, file write, browser modification (ii) root request, bash execution (iii) anti analysis (iv) network, Tor, open port (v) detection rate by AVs on the x-axis. ChromePatch and Mughthesec are classified as Adware, Macransom is ransonware and the rest of variants is trojan/backdoor on the y-axis. We clustered operations as they are often needed one another. For example: persistence needs to write file locations, modify startup locations. Browser modification (like adware installing malicious plugins on the browser), also make file write operation

7 Malware family analysis

Our Dataset consists of 2,000 unique macOS malware examples collected from VirusTotal Malware Intelligence Services (VTMIS),⁸ Macmalware,⁹ and Objective-see¹⁰ between January 2017 and July 2017. macOS malware evolution is investigated using heat map from 2016 to 2017. Figure 7 shows that only 2 samples use anti-analysis techniques while 5 samples do in 2017. We observed that some samples are an improvement of existing malware appeared in 2016. In 2017, we have also observed that the increasing trend of using malware to affect victim internet browser, to redirect them to fraud web traffic or unwanted advertising for profit.¹¹ In 2017, we have also observed that the trend of using malware to affect victim internet browser to redirect them to fraud web traffic or unwanted advertising for profit.

Table 6 shows suspicious behaviors captured by Mac-A-Mal from malware samples in the dataset. Many of them execute network operations, writing data to the victim machine, spawn processes or threads, and perform persistence techniques. Among anti-analysis techniques, we observed common system calls such as ptrace(), sysctl(), sleep()

We write analysis rules to identify suspicious characteristics or behaviors such as network activity, anti-debug techniques, persistence mechanisms, etc. We then apply the rules to analyze malware in the dataset. Figure 7 shows our analysis result for malware families. Family OSX/Fake File Opener appears to do something benignly with neither network activity nor persistence mechanisms. However, further analysis on the samples we found a large embedded Info.plist file which registers 232 file extensions to the operating system. Such behavior makes the malware look potentially like an unwanted application that masquerades as a document handler. macOS/Mughthesec tries to hijack victim’s browsers, install the SafeFinder adware, and use query victim’s Mac address to detect virtual machine, download additional malicious applications such as a scam utility Advanced Mac Cleaner. OSX/Dok employs anti-analysis technique, uses Tor to communicate with its command and control server or try to tamper victim network traffic, modify browser settings, spaw shell, and request superuser permission to disable system update as well as XProtect signature.

More importantly, Mac-A-Mal shows significant approaches in detecting and analyzing unknown malware using VTMIS. We provide an analysis of two undetected malware campaign. The first one is OSX/Mughthesec adware campaign (founded in August 2017) where Mac-A-Mal discovered 71 signed archives, 10 Apple legitimate developer certificates (only 2 of them revoked later). Another one is the APT32-OceanLotus campaign where we catch the second generation of Mac APT32. More details our finding and malware analysis artifacts, refer to [18, 19].

8 Threats to validity

Although we implement our logging mechanism in the user-space, which might be detected by malware, its counterpart kernel module, however, is armed with advanced rootkit functionalities [20] such as KEXT, process and file hider can minimize the traces. Besides, our framework could also not capture all anti-analysis techniques by malware authors as they are becoming more and more complex. But with its nature open-source¹² enables analysts to add their own handlers for a specific technique.

9 Conclusion and future work

In this paper, we presented a hybrid malware analysis framework where static and dynamic analysis are combined to support the analysts. Its key feature is a kernel hooking module, which implements various analysis techniques including system calls and services invocations that are macOS specific. We provide many hooks for arguments monitoring, process tracing, anti-analysis detection, and mitigation. Mac-A-Mal performs best among other monitor tools on the same benchmark both regarding tracing process execution time and many CPU counts. With Mac-A-Mal, we discovered 71 unknown Mac malware variants currently using new evasion techniques. Our behavior classification rules allow us to filter out behaviors that are unrelated to malware. While 85% of the collected samples are adware, most of the classified variants (49%) belong to the trojan/backdoor family. We discovered a significant improvement of Mac evasion methods such as sleep probing, detect MAC address, etc.

An interesting future development is to use Mac-A-Mal combined with machine learning techniques to detect and classify malware automatically. To this extent, Mac-A-Mal provides a rich set of traces (e.g., system call, network activity, persistence mechanisms, anti-debug techniques) that machine learning tool can exploit.