Efficient and Stealthy Instruction Tracing and Its Applications in Automated Malware Analysis: Open Problems and Challenges

  • Endre Bangerter
  • Stefan Bühlmann
  • Engin Kirda
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7039)


Malware is substantial security threat today and most likely in the foreseeable future. The analysis of malware is a key activity in the fight against the threat. Since manual analysis is time consuming and given the extent of the malware threat, malware analysis needs to be automated. Malware analysis sandboxes offer such automation and play already an important role in practice. Yet, they only uncover certain aspects of malware behavior, and still require manual analysis in many cases. This is not a viable way to go, and thus the automation and quality of automated analysis needs to be pushed further. A promising technique towards this goal is instruction tracing combined with analyzes algorithms that uncover malware behavior from an instruction trace.

In this position paper, we shall argue that instruction tracing is still in its infancy and point out challenges and open problems of instruction tracing in general. In particular, we shall describe Helios, which is our new instruction tracer that offers a better balance of tracing speed and transparency than existing techniques.


Basic Block Code Coverage Rendezvous Point Application Thread Distribute System Security Symposium 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. [Ban09]
    Bania, P.: Generic unpacking of self-modifying, aggressive, packed binary programs (2009)Google Scholar
  2. [BK08]
    Buehlmann, S., Kropp, M.: Extending joebox - a scriptable malware analysis system. In: University of Applied Science Northwestern of Switzerland, Bachelor Thesis (2008)Google Scholar
  3. [BKK06]
    Bayer, U., Kruegel, C., Kirda, E.: Ttanalyze: A tool for analyzing malware. In: 15th European Institute for Computer Antivirus Research, EICAR 2006 (2006)Google Scholar
  4. [CPSK09]
    Caballero, J., Poosankam, P., Song, D., Kreibich, C.: Dispatcher: Enabling active botnet infiltration using automatic protocol reverse-engineering. In: The 16th ACM Conference on Computer and Communications Security, CCS 2009, pp. 621–634. ACM (2009)Google Scholar
  5. [CYLS07]
    Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: Automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of ACM Conference on Computer and Communication Security (2007)Google Scholar
  6. [DRSL08]
    Dinaburg, A., Royal, P., Sharif, M.I., Lee, W.: Ether: malware analysis via hardware virtualization extensions. In: ACM Conference on Computer and Communications Security (2008)Google Scholar
  7. [EKK+07]
    Egele, M., Kruegel, C., Kirda, E., Yin, H., Song, D.: Dynamic spyware analysis. In: Proceedings of USENIX Annual Technical Conference (2007)Google Scholar
  8. [GWH11]
    Groebert, F., Willems, C., Holz, T.: Automated identification of cryptographic primitives in binary programs. In: The 14th International Symposium on Recent Advances in Intrusion Detection, RAID (2011)Google Scholar
  9. [HR]
    Hex-Rays. Hex-rays decompiler,
  10. [Int10]
    Intel. Intel 64 and ia-32 architectures software developer’s manual. Basic architecture, ch. 5, 5.1.7, vol. 1, pp. 142–143, (2010)Google Scholar
  11. [KKCW08]
    Kruegel, C., Kirda, E., Comparetti, P.M., Wondracek, G.: Automatic network protocol analysis. In: 15th Annual Network and Distributed System Security Symposium, NDSS 2008 (2008)Google Scholar
  12. [LCM+05]
    Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2005, pp. 190–200. ACM, New York (2005), CrossRefGoogle Scholar
  13. [LJXZ08]
    Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic protocol format reverse engineering through conectect-aware monitored execution. In: 15th Symposium on Network and Distributed System Security, NDSS (2008)Google Scholar
  14. [Lut08]
    Lutz, N.: Towards revealing attackers intent by automatically decrypting network traffic. Master’s thesis, ETH Zuerich (2008)Google Scholar
  15. [LW09]
    Leder, F., Werner, T.: Know your enemy: Containing conficker - to tame a malware. In: Know Your Enemy Series of the Honeynet Project (2009)Google Scholar
  16. [MKK07]
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: 23rd Annual Computer Security Applications Conference, ACSAC (2007)Google Scholar
  17. [MPRB09]
    Martignoni, L., Paleari, R., Roglia, G.F., Bruschi, D.: Testing cpu emulators. In: ISSTA (2009)Google Scholar
  18. [NS05]
    Newsome, J., Song, D.: Dynamic taint analysis: Automatic detection, analysis, and signature generation of exploit attacks on commodity software. In: Proceedings of the Network and Distributed Systems Security Symposium (2005)Google Scholar
  19. [PSY09]
    Porras, P., Saidi, H., Yegneswaran, V.: A foray into conficker’s logic and rendezvous points. In: Proceedings of the 2nd USENIX Conference on Large-Scale Exploits and Emergent Threats, LEET 2009 (2009)Google Scholar
  20. [QL09]
    Quist, D.A., Liebrock, L.M.: Visualizing compiled executables for malware analysis. In: 6th International Workshop on Visualization for Cyber Security, VizSec 2009 (2009)Google Scholar
  21. [QLN09]
    Quist, D., Liebrock, L., Neil, J.: Visualizing compiled executables for malware analysis. Journal in Computer Virology (2009)Google Scholar
  22. [SBY+08]
    Song, D., Brumley, D., Yin, H., Caballero, J., Jager, I., Kang, M.G., Liang, Z., Newsome, J., Poosankam, P., Saxena, P.: BitBlaze: A New Approach to Computer Security via Binary Analysis. In: Sekar, R., Pujari, A.K. (eds.) ICISS 2008. LNCS, vol. 5352, pp. 1–25. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  23. [SDC+10]
    Skaletsky, A., Devor, T., Chachmon, N., Cohn, R.S., Hazelwood, K.M., Vladimirov, V., Bach, M.: Dynamic program analysis of microsoft windows applications. In: ISPASS (2010)Google Scholar
  24. [VY06]
    Vasudevan, A., Yerraballi, R.: Cobra: Fine-grained malware analysis using stealth localized-executions. In: IEEE Symposium on Security and Privacy (2006)Google Scholar
  25. [WHF07]
    Willems, C., Holz, T., Freiling, F.: Toward automated dynamic malware analysis using cwsandbox (2007)Google Scholar
  26. [YS10]
    Yin, H., Song, D.: Temu: Binary code analysis via whole-system layered annotative execution (2010)Google Scholar
  27. [ZWCW08]
    Jiang, X., Wang, Z., Cui, W., Wang, X.: Reformat: Automatic reverse engineering of encrypted messages. In: Technical report, NC State University (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Endre Bangerter
    • 1
  • Stefan Bühlmann
    • 2
  • Engin Kirda
    • 3
  1. 1.Bern University of Applied SciencesSwitzerland
  2. 2.Bern University of Applied Sciences and, Joe SecuritySwitzerland
  3. 3.Northeastern UniversityUSA

Personalised recommendations