A Host Based Kernel Level Rootkit Detection Mechanism Using Clustering Technique

  • Jestin Joy
  • Anita John
Part of the Communications in Computer and Information Science book series (CCIS, volume 204)


Rootkits are a set of software tools used by an attacker to gain unauthorized access into a system, thereby providing him with privilege to access sensitive data, conceal its own existence and allowing him to install other malicious software. They are difficult to detect due to their elusive nature. Modern rootkit attacks mainly focus on modifying operating system kernel. Existing techniques for detection rely mainly on saving the system state before detection and comparing it with the infected state. Efficient detection is possible by properly differentiating malicious and non malicious activities taking place in a kernel. In this paper we present a novel anomaly detection method for kernel level rootkits based on k-means clustering algorithm.


System Call Memory Address Malicious Code Guest Operating System Kernel Level 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Arthur, C.: More than 50 android apps found infected with rootkit malware. Guardian Technology Blog (2011),
  3. 3.
    Baliga, A., Ganapathy, V., Iftode, L.: Detecting kernel-level rootkits using data structure invariants. IEEE Transactions on Dependable and Secure Computing 99(PrePrints) (2010)Google Scholar
  4. 4.
    Bickford, J., O’Hare, R., Baliga, A., Ganapathy, V., Iftode, L.: Rootkits on smart phones: attacks, implications and opportunities. In: Proceedings of the Eleventh Workshop on Mobile Computing Systems & Applications, pp. 49–54. ACM, New York (2010)CrossRefGoogle Scholar
  5. 5.
    Bunten, A.: Unix and linux based rootkits techniques and countermeasures (2004)Google Scholar
  6. 6.
    Clemens, J.: Intrusion Detection FAQ: Knark: Linux Kernel Subversion (2001)Google Scholar
  7. 7.
    Desnos, A.: Detecting (and creating!) a hvm rootkit (aka bluepill-like). Journal in Computer Virology, 1–27 (2009),
  8. 8.
    Garfinkel, T., Rosenblum, M.: A virtual machine introspection based architecture for intrusion detection. In: Proc. Network and Distributed Systems Security Symposium, vol. 1, pp. 253–285. Citeseer (2003)Google Scholar
  9. 9.
    Kroah-Hartman, G.: Signed kernel modules. Linux Journal (2004)Google Scholar
  10. 10.
    Levine, J., Grizzard, J., Owen, H.: A methodology to detect and characterize kernel level rootkit exploits involving redirection of the system call table. In: Proceedings Second IEEE International Information Assurance Workshop 2004, pp. 107–125. IEEE, Los Alamitos (2005)Google Scholar
  11. 11.
    Levine, J.G., Grizzard, J.B., Owen, H.L.: Detecting and categorizing kernel-level rootkits to aid future detection. IEEE Security and Privacy 4, 24 (2006), CrossRefGoogle Scholar
  12. 12.
  13. 13.
    Miller, T.: Analysis of the KNARK Rootkit (2004)Google Scholar
  14. 14.
    Petroni Jr., N.L., Fraser, T., Molina, J., Arbaugh, W.A.: Copilot - a coprocessor-based kernel runtime integrity monitor. In: Proceedings of the 13th Conference on USENIX Security Symposium, SSYM 2004, vol. 13, p. 13. USENIX Association, Berkeley (2004), Google Scholar
  15. 15.
    Wang, Z., Jiang, X., Cui, W., Ning, P.: Countering kernel rootkits with lightweight hook protection. In: Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS 2009, pp. 545–554. ACM, New York (2009), Google Scholar
  16. 16.
    Yan, Q., Li, Y., Li, T., Deng, R.: Insights into Malware Detection and Prevention on Mobile Phones. Security Technology, 242–249 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Jestin Joy
    • 1
  • Anita John
    • 1
  1. 1.Rajagiri School of Engineering & TechnologyKochiIndia

Personalised recommendations