Behavior Abstraction in Malware Analysis

  • Philippe Beaucamps
  • Isabelle Gnaedig
  • Jean-Yves Marion
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6418)

Abstract

We present an approach for proactive malware detection working by abstraction of program behaviors. Our technique consists in abstracting program traces, by rewriting given subtraces into abstract symbols representing their functionality. Traces are captured dynamically by code instrumentation, which allows us to handle packed or self-modifying malware. Suspicious behaviors are detected by comparing trace abstractions to reference malicious behaviors. The expressive power of abstraction allows us to handle general suspicious behaviors rather than specific malware code and then, to detect malware mutations. We present and discuss an implementation validating our approach.

Keywords

Malware behavioral detection behavior abstraction trace string rewriting finite state automaton formal language dynamic binary instrumentation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
    Apel, M., Bockermann, C., Meier, M.: Measuring similarity of malware behavior. In: IEEE Conference on Local Computer Networks, pp. 891–898. IEEE, Los Alamitos (October 2009)CrossRefGoogle Scholar
  5. 5.
    Beaucamps, P., Gnaedig, I., Marion, J.-Y.: Behavior Abstraction in Malware Analysis - Extended Version. HAL-INRIA Open Archive Number inria-00509486Google Scholar
  6. 6.
    Bergeron, J., Debbabi, M., Desharnais, J., Erhioui, M., Lavoie, Y., Tawbi, N.: Static detection of malicious code in executable programs. In: Symposium on Requirements Engineering for Information Security (2001)Google Scholar
  7. 7.
    Bonfante, G., Kaczmarek, M., Marion, J.-Y.: Architecture of a morphological malware detector. Journal in Computer Virology (2008)Google Scholar
  8. 8.
    Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., Yin, H.: Automatically identifying trigger-based behavior in malware. Botnet Detection 36, 65–88 (2008)CrossRefGoogle Scholar
  9. 9.
    Bruschi, D., Martignoni, L., Monga, M.: Detecting self-mutating malware using control-flow graph matching. In: Büschkes, R., Laskov, P. (eds.) DIMVA 2006. LNCS, vol. 4064, pp. 129–143. Springer, Heidelberg (2006)Google Scholar
  10. 10.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: IEEE Symposium on Security and Privacy, pp. 32–46. IEEE Computer Society, Los Alamitos (2005)Google Scholar
  11. 11.
    Cohen, F.: Computer viruses: Theory and experiments. Computers and Security 6(1), 22–35 (1987)CrossRefGoogle Scholar
  12. 12.
    Dullien, T., Rolles, R.: Graph-based comparison of executable objects. In: Symposium sur la Sécurité des Technologies de l’Information et des Télécommunications (2005)Google Scholar
  13. 13.
    Esparza, J., Rossmanith, P., Schwoon, S.: A uniform framework for problems on context-free grammars. Bulletin of the EATCS 72, 169–177 (2000)MATHMathSciNetGoogle Scholar
  14. 14.
    Godefroid, P., Levin, M.Y., Molnar, D.: Automated whitebox fuzz testing. In: Network Distributed Security Symposium, Internet Society (2008)Google Scholar
  15. 15.
    Gunter, C.A.: Semantics of Programming Languages: Structures and Techniques. MIT Press, Cambridge (1992)MATHGoogle Scholar
  16. 16.
    Jacob, G., Debar, H., Filiol, E.: Malware behavioral detection by attribute-automata using abstraction from platform and language. In: Balzarotti, D. (ed.) RAID 2009. LNCS, vol. 5758, pp. 81–100. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Kinder, J., Katzenbeisser, S., Schallhart, C., Veith, H.: Detecting malicious code by model checking. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 174–187. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  18. 18.
    Charlier, B.L., Mounji, A., Swimmer, M.: Dynamic detection and classification of computer viruses using general behaviour patterns. In: International Virus Bulletin Conference, pp. 1–22 (1995)Google Scholar
  19. 19.
    Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 78–97. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Moser, A., Kruegel, C., Kirda, E.: Exploring multiple execution paths for malware analysis. In: IEEE Symposium on Security and Privacy, pp. 231–245. IEEE Computer Society, Los Alamitos (2007)Google Scholar
  21. 21.
    Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: IEEE Symposium on Security and Privacy, pp. 144–155. IEEE Computer Society, Los Alamitos (2001)Google Scholar
  22. 22.
    Singh, P.K., Lakhotia, A.: Static verification of worm and virus behavior in binary executables using model checking. In: Information Assurance Workshop, pp. 298–300. IEEE Press, Los Alamitos (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Philippe Beaucamps
    • 1
  • Isabelle Gnaedig
    • 1
  • Jean-Yves Marion
    • 1
  1. 1.INPL - INRIA Nancy Grand Est Nancy-Université - LORIAVandoeuvre-lès-Nancy CedexFrance

Personalised recommendations