Abstract
Malware is substantial security threat today and most likely in the foreseeable future. The analysis of malware is a key activity in the fight against the threat. Since manual analysis is time consuming and given the extent of the malware threat, malware analysis needs to be automated. Malware analysis sandboxes offer such automation and play already an important role in practice. Yet, they only uncover certain aspects of malware behavior, and still require manual analysis in many cases. This is not a viable way to go, and thus the automation and quality of automated analysis needs to be pushed further. A promising technique towards this goal is instruction tracing combined with analyzes algorithms that uncover malware behavior from an instruction trace.
In this position paper, we shall argue that instruction tracing is still in its infancy and point out challenges and open problems of instruction tracing in general. In particular, we shall describe Helios, which is our new instruction tracer that offers a better balance of tracing speed and transparency than existing techniques.
Chapter PDF
Similar content being viewed by others
References
Bania, P.: Generic unpacking of self-modifying, aggressive, packed binary programs (2009)
Buehlmann, S., Kropp, M.: Extending joebox - a scriptable malware analysis system. In: University of Applied Science Northwestern of Switzerland, Bachelor Thesis (2008)
Bayer, U., Kruegel, C., Kirda, E.: Ttanalyze: A tool for analyzing malware. In: 15th European Institute for Computer Antivirus Research, EICAR 2006 (2006)
Caballero, J., Poosankam, P., Song, D., Kreibich, C.: Dispatcher: Enabling active botnet infiltration using automatic protocol reverse-engineering. In: The 16th ACM Conference on Computer and Communications Security, CCS 2009, pp. 621–634. ACM (2009)
Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: Automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of ACM Conference on Computer and Communication Security (2007)
Dinaburg, A., Royal, P., Sharif, M.I., Lee, W.: Ether: malware analysis via hardware virtualization extensions. In: ACM Conference on Computer and Communications Security (2008)
Egele, M., Kruegel, C., Kirda, E., Yin, H., Song, D.: Dynamic spyware analysis. In: Proceedings of USENIX Annual Technical Conference (2007)
Groebert, F., Willems, C., Holz, T.: Automated identification of cryptographic primitives in binary programs. In: The 14th International Symposium on Recent Advances in Intrusion Detection, RAID (2011)
Hex-Rays. Hex-rays decompiler, http://www.hex-rays.com/decompiler.shtml
Intel. Intel 64 and ia-32 architectures software developer’s manual. Basic architecture, ch. 5, 5.1.7, vol. 1, pp. 142–143, (2010)
Kruegel, C., Kirda, E., Comparetti, P.M., Wondracek, G.: Automatic network protocol analysis. In: 15th Annual Network and Distributed System Security Symposium, NDSS 2008 (2008)
Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2005, pp. 190–200. ACM, New York (2005), http://doi.acm.org/10.1145/1065010.1065034
Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic protocol format reverse engineering through conectect-aware monitored execution. In: 15th Symposium on Network and Distributed System Security, NDSS (2008)
Lutz, N.: Towards revealing attackers intent by automatically decrypting network traffic. Master’s thesis, ETH Zuerich (2008)
Leder, F., Werner, T.: Know your enemy: Containing conficker - to tame a malware. In: Know Your Enemy Series of the Honeynet Project (2009)
Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: 23rd Annual Computer Security Applications Conference, ACSAC (2007)
Martignoni, L., Paleari, R., Roglia, G.F., Bruschi, D.: Testing cpu emulators. In: ISSTA (2009)
Newsome, J., Song, D.: Dynamic taint analysis: Automatic detection, analysis, and signature generation of exploit attacks on commodity software. In: Proceedings of the Network and Distributed Systems Security Symposium (2005)
Porras, P., Saidi, H., Yegneswaran, V.: A foray into conficker’s logic and rendezvous points. In: Proceedings of the 2nd USENIX Conference on Large-Scale Exploits and Emergent Threats, LEET 2009 (2009)
Quist, D.A., Liebrock, L.M.: Visualizing compiled executables for malware analysis. In: 6th International Workshop on Visualization for Cyber Security, VizSec 2009 (2009)
Quist, D., Liebrock, L., Neil, J.: Visualizing compiled executables for malware analysis. Journal in Computer Virology (2009)
Song, D., Brumley, D., Yin, H., Caballero, J., Jager, I., Kang, M.G., Liang, Z., Newsome, J., Poosankam, P., Saxena, P.: BitBlaze: A New Approach to Computer Security via Binary Analysis. In: Sekar, R., Pujari, A.K. (eds.) ICISS 2008. LNCS, vol. 5352, pp. 1–25. Springer, Heidelberg (2008)
Skaletsky, A., Devor, T., Chachmon, N., Cohn, R.S., Hazelwood, K.M., Vladimirov, V., Bach, M.: Dynamic program analysis of microsoft windows applications. In: ISPASS (2010)
Vasudevan, A., Yerraballi, R.: Cobra: Fine-grained malware analysis using stealth localized-executions. In: IEEE Symposium on Security and Privacy (2006)
Willems, C., Holz, T., Freiling, F.: Toward automated dynamic malware analysis using cwsandbox (2007)
Yin, H., Song, D.: Temu: Binary code analysis via whole-system layered annotative execution (2010)
Jiang, X., Wang, Z., Cui, W., Wang, X.: Reformat: Automatic reverse engineering of encrypted messages. In: Technical report, NC State University (2008)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bangerter, E., Bühlmann, S., Kirda, E. (2012). Efficient and Stealthy Instruction Tracing and Its Applications in Automated Malware Analysis: Open Problems and Challenges. In: Camenisch, J., Kesdogan, D. (eds) Open Problems in Network Security. iNetSec 2011. Lecture Notes in Computer Science, vol 7039. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27585-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-27585-2_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-27584-5
Online ISBN: 978-3-642-27585-2
eBook Packages: Computer ScienceComputer Science (R0)