A Graph-Based Model for Discovering Host-Based Hook Attacks

Abstract

Though computer malicious software can be referred with different names such as virus, worm, Trojan, spam, and botnet, their ultimate goal is to causing damage to the end-computer or end-user. The progression in computer technology allows a malware writer to integrate obfuscation technique to evade detection specifically API hooking in Windows. Unfortunately, signature-based detection approach such as anti-virus software at the end-computer is not effective against system call reordering. To overcome this shortcoming, many different behavior-based approaches have been offered. However, these approaches bear limitations such as false positive, detecting zero-day attacks, and improving detection accuracy rate from past experience. In this article, an application programming interface (API)-based call graph model is put forward which captures API system call during malicious rootkit execution in Windows platform. As graph model can be effectively applied to replica complicated relation between entities, we opt it to visualize malicious rootkit behavior activities by monitoring system API calls. This will help the defender to optimally find malicious system calls from benign calls. Our simulated experiment analysis proves that our method achieves higher detection rate and accuracy with less false positive compared to existing techniques.