Abstract
Installing various hooks into the victim system is an important attacking strategy employed by malware, including spyware, rootkits, stealth backdoors, and others. In order to defeat existing hook detectors, malware writers keep exploring new hooking mechanisms. However, the current malware analysis procedure is painstaking, mostly manual and error-prone. In this chapter, we propose the first systematic approach for automatically identifying hooks and extracting hooking mechanisms. We propose a unified approach, fine-grained impact analysis, to identify malware hooking behaviors. Our approach does not rely on any prior knowledge of hooking mechanisms, and thus can identify novel hooking mechanisms. Moreover, we devise a method using semantics-aware impact dependency analysis to provide a succinct and intuitive graph representation to illustrate hooking mechanisms. We have developed a prototype, HookFinder, and conducted extensive experiments using representative malware samples from various categories. We have demonstrated that HookFinder can correctly identify the hooking behaviors of all samples, and provide accurate insights about their hooking mechanisms.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afxrootkit. http://www.rootkit.com/project.php?id=23
Butler, J., Hoglund, G.: VICE–catch the hookers! In: Black Hat USA (2004). http://www.blackhat.com/presentations/bh-usa-04/bh-us-04-butler/bh-us-04-butler.pdf
Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: Automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of ACM Conference on Computer and Communication Security (2007)
Clandestine file system driver. http://www.rootkit.com/vault/merlvingian/cfsd.zip
Chow, J., Pfaff, B., Garfinkel, T., Christopher, K., Rosenblum, M.: Understanding data lifetime via whole system simulation. In: Proceedings of the 13th USENIX Security Symposium (Security’03) (2004)
Cost, M., Crowcroft, J., Castro, M., Rowstron, A., Zhou, L., Zhang, L., Barham, P.: Vigilante: End-to-end containment of internet worms. In: 20th ACM Symposium on Operating System Principles (SOSP 2005) (2005)
Crandall, J.R., Chong, F.T.: Minos: Control data attack prevention orthogonal to memory model. In: Proceedings of the 37th International Symposium on Microarchitecture (MICRO’04) (2004)
Egele, M., Kruegel, C., Kirda, E., Yin, H., Song, D.: Dynamic Spyware Analysis. In: Proceedings of the 2007 Usenix Annual Conference (Usenix’07) (2007)
Hacker defender. http://www.rootkit.com/project.php?id=5
Newsome, J., Song, D.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS) (2005)
Portokalidis, G., Slowinska, A., Bos, H.: Argos: an emulator for fingerprinting zero-day attacks. In: EuroSys 2006 (2006)
Rutkowska, J.: System virginity verifier: Defining the roadmap for malware detection on windows systems. In: Hack In The Box Security Conference (2005). http://www.invisiblethings.org/papers/hitb05_virginity_verifier.ppt
Rutkowska, J.: Rootkit hunting vs. compromise detection. In: Black Hat Federal (2006). http://www.invisiblethings.org/papers/rutkowska_bhfederal2006.ppt
Sony’s DRM Rootkit: The Real Story. http://www.schneier.com/blog/archives/2005/11/sonys_drm_rootk.html
Suh, G.E., Lee, J.W., Zhang, D., Devadas, S.: Secure program execution via dynamic information flow tracking. In: Proceedings of the 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’04) (2004)
UAY kernel-mode backdoor. http://uty.512j.com/uay.rar
Vanquish. https://www.rootkit.com/vault/xshadow/vanquish-0.2.1.zip
Vogt, P., Nentwich, F., Jovanovic, N., Kirda, E., Kruegel, C., Vigna, G.: Cross-Site Scripting Prevention with Dynamic Data Tainting and Static Analysis. In: Proceeding of the Network and Distributed System Security Symposium (NDSS’07) (2007)
Yin, H., Song, D., Egele, M., Kruegel, C., Kirda, E.: Panorama: Capturing system-wide information flow for malware detection and analysis. In: Proceedings of ACM Conference on Computer and Communication Security (2007)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2013 The Author(s)
About this chapter
Cite this chapter
Yin, H., Song, D. (2013). Hooking Behavior Analysis. In: Automatic Malware Analysis. SpringerBriefs in Computer Science. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-5523-3_5
Download citation
DOI: https://doi.org/10.1007/978-1-4614-5523-3_5
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-5522-6
Online ISBN: 978-1-4614-5523-3
eBook Packages: Computer ScienceComputer Science (R0)