Abstract
Malware often encrypts its malicious code and sensitive data to avoid static pattern detection, thus detecting encryption functions and extracting the encryption keys in a malware can be very useful in security analysis. However, it’s a complicated process to automatically detect encryption functions among huge amount of binary code, and the main challenge is to keep high efficiency and accuracy at the same time. In this paper we propose an enhanced detection approach. First we designed a novel process level emulation technique to efficiently analyze binary code, which is less resource-consuming compared with full system emulation. Further, we conduct program partitioning and assembly-to-IL(intermediate language) translation on binary code to simplify the analysis. We applied our approach to sample programs using cryptographic libraries and custom implemented version of typical encryption algorithms, and showed that these routines can be detected efficiently. It is convenient for analysts to use our approach to deal with the encrypted data within malware automatically. Our approach also provides an extensible interface for analysts to add extra templates to detect other forms of functions besides encryption routines.
Keywords
Supported by the National Science and Technology Major Projects 2012ZX03002011-002.
Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Nettle: a low-level crypto library (last visited, 2012), http://www.lysator.liu.se/~nisse/nettle/
Bellard, F.: Qemu, a fast and portable dynamic translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005)
Caballero, J.: Binary code extraction and interface identification for security applications. Technical report, DTIC Document (2009)
Gröbert, F., Willems, C., Holz, T.: Automated Identification of Cryptographic Primitives in Binary Programs. In: Sommer, R., Balzarotti, D., Maier, G. (eds.) RAID 2011. LNCS, vol. 6961, pp. 41–60. Springer, Heidelberg (2011)
Intel. Intel 64 and ia-32 architectures software developers manual. intel, http://www.intel.com/products/processor/manuals 64
Jhi, Y.C., Wang, X., Jia, X., Zhu, S., Liu, P., Wu, D.: Value-based program characterization and its application to software plagiarism detection. In: Proceeding of the 33rd International Conference on Software Engineering, pp. 756–765. ACM (2011)
Lawton, K.P.: Bochs: A portable pc emulator for unix/x. Linux Journal 29es, 7 (1996)
Luk, C.K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: ACM SIGPLAN Notices, vol. 40, pp. 190–200. ACM (2005)
Oksanen, K.: Detecting algorithms using dynamic analysis. In: Proceedings of the Ninth International Workshop on Dynamic Analysis, pp. 1–6. ACM (2011)
Sæbjørnsen, A., Willcock, J., Panas, T., Quinlan, D., Su, Z.: Detecting code clones in binary executables. In: Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, pp. 117–128. ACM (2009)
OpenSSL: The Open Source toolkit for SSL/TLS (last visited, 2012), http://www.openssl.org/
Zhang, F., Jhi, Y.C., Wu, D., Liu, P., Zhu, S.: A first step towards algorithm plagiarism detection (2011)
Zhao, R., Gu, D., Li, J., Yu, R.: Detection and analysis of cryptographic data inside software. Information Security, 182–196 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhao, R., Gu, D., Li, J., Liu, H. (2012). Detecting Encryption Functions via Process Emulation and IL-Based Program Analysis. In: Chim, T.W., Yuen, T.H. (eds) Information and Communications Security. ICICS 2012. Lecture Notes in Computer Science, vol 7618. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34129-8_22
Download citation
DOI: https://doi.org/10.1007/978-3-642-34129-8_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34128-1
Online ISBN: 978-3-642-34129-8
eBook Packages: Computer ScienceComputer Science (R0)