Detecting Traditional Packers, Decisively

  • Denis Bueno
  • Kevin J. Compton
  • Karem A. Sakallah
  • Michael Bailey
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8145)


Many of the important decidability results in malware analysis are based Turing machine models of computation. We exhibit computational models which use more realistic assumptions about machine and attacker resources. While seminal results such as [1–5] remain true for Turing machines, we show under more realistic assumptions, important tasks are decidable instead of undecidable. Specifically, we show that detecting traditional malware unpacking behavior – in which a payload is decompressed or decrypted and subsequently executed – is decidable under our assumptions. We then examine the issue of dealing with complex but decidable problems. We look for lessons from the hardware verification community, which has been striving to meet the challenge of intractable problems for the past three decades.


Model Check Turing Machine Computer Virus Malicious Behavior Symbolic Model Check 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Royal, P., Halpin, M., Dagon, D., Edmonds, R., Lee, W.: PolyUnpack: Automating the hidden-code extraction of unpack-executing malware. In: Annual Computer Security Applications Conference, pp. 289–300. IEEE Computer Society (2006)Google Scholar
  2. 2.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D.X., Bryant, R.E.: Semantics-aware malware detection. In: Security and Privacy, pp. 32–46. IEEE Computer Society Press (2005)Google Scholar
  3. 3.
    Oppliger, R., Rytz, R.: Does trusted computing remedy computer security problems? IEEE Security Privacy 3(2), 16–19 (2005)CrossRefGoogle Scholar
  4. 4.
    Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D.X., Yin, H.: Automatically identifying trigger-based behavior in malware. In: Botnet Detection, pp. 65–88. Springer (2008)Google Scholar
  5. 5.
    Newsome, J., Brumley, D., Franklin, J., Song, D.: Replayer: automatic protocol replay by binary analysis. In: ACM Conference on Computer and Communications Security, CCS 2006, pp. 311–321. ACM, New York (2006)Google Scholar
  6. 6.
    Bayer, U., Kirda, E., Kruegel, C.: Improving the efficiency of dynamic malware analysis. In: Proceedings of the 2010 ACM Symposium on Applied Computing, pp. 1871–1878. ACM (2010)Google Scholar
  7. 7.
    Guo, F., Ferrie, P., Chiueh, T.-c.: A study of the packer problem and its solutions. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 98–115. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  8. 8.
    Cook, S.A., Reckhow, R.A.: Time bounded random access machines. J. Comput. Syst. Sci. 7(4), 354–375 (1973)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Jang, J., Brumley, D., Venkataraman, S.: Bitshred: feature hashing malware for scalable triage and semantic analysis. In: ACM Conference on Computer and Communications Security, CCS 2011, pp. 309–320. ACM, New York (2011)Google Scholar
  10. 10.
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: Computer Security Applications Conference, pp. 421–430 (2007)Google Scholar
  11. 11.
    Zhang, Q., Reeves, D.S.: MetaAware: Identifying metamorphic malware. In: Annual Computer Security Applications Conference, pp. 411–420. IEEE Computer Society Press (2007)Google Scholar
  12. 12.
    Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Automatic reverse engineering of malware emulators. In: Security and Privacy, pp. 94–109. IEEE Computer Society (2009)Google Scholar
  13. 13.
    Kang, M.G., Poosankam, P., Yin, H.: Renovo: A hidden code extractor for packed executables. In: WORM. ACM (November 2007)Google Scholar
  14. 14.
    Martignoni, L., Christodorescu, M., Jha, S.: OmniUnpack: Fast, generic, and safe unpacking of malware. In: Annual Computer Security Applications Conference, pp. 431–441. IEEE Computer Society Press (2007)Google Scholar
  15. 15.
    Yin, H., Song, D.: Hidden code extraction. In: Automatic Malware Analysis. SpringerBriefs in Computer Science, pp. 17–26. Springer, New York (2013)CrossRefGoogle Scholar
  16. 16.
    Liu, L., Ming, J., Wang, Z., Gao, D., Jia, C.: Denial-of-service attacks on host-based generic unpackers. In: Qing, S., Mitchell, C.J., Wang, G. (eds.) ICICS 2009. LNCS, vol. 5927, pp. 241–253. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Xie, P.D., Li, M.J., Wang, Y.J., Su, J.S., Lu, X.C.: Unpacking techniques and tools in malware analysis. Applied Mechanics and Materials 198–199, 343–350 (2012)CrossRefGoogle Scholar
  18. 18.
    Perdisci, R., Lanzi, A., Lee, W.: Classification of packed executables for accurate computer virus detection. Pattern Recognition Letters 29(14), 1941–1946 (2008)CrossRefGoogle Scholar
  19. 19.
    Spinellis, D.: Reliable identification of bounded-length viruses is NP-complete. IEEE Transactions on Information Theory 49(1), 280–284 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Borello, J.M., Mé, L.: Code obfuscation techniques for metamorphic viruses. Journal in Computer Virology 4(3), 211–220 (2008)CrossRefGoogle Scholar
  21. 21.
    Katz, J., Lindell, Y.: Introduction to Modern Cryptography. Chapman & Hall (2008)Google Scholar
  22. 22.
    Elgot, C.C., Robinson, A.: Random-access stored-program machines, an approach to programming languages. J. ACM 11(4), 365–399 (1964)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Hartmanis, J.: Computational complexity of random access stored program machines. Mathematical Systems Theory 5(3), 232–245 (1971)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Aho, A.V., Hopcroft, J.E., Ullman, J.D.: The Design and Analysis of Computer Algorithms. Addison-Wesley (1974)Google Scholar
  25. 25.
    Cohen, F.: Computer Viruses. PhD thesis, University of Southern California (1986)Google Scholar
  26. 26.
    Cohen, F.: Computational aspects of computer viruses. Computers & Security 8(4), 297–298 (1989)CrossRefGoogle Scholar
  27. 27.
    Adleman, L.M.: An abstract theory of computer viruses. In: Goldwasser, S. (ed.) CRYPTO 1988. LNCS, vol. 403, pp. 354–374. Springer, Heidelberg (1990)CrossRefGoogle Scholar
  28. 28.
    Thimbleby, H., Anderson, S., Cairns, P.: A framework for modelling trojans and computer virus infection. The Computer Journal 41(7), 444–458 (1998)CrossRefzbMATHGoogle Scholar
  29. 29.
    Chess, D.M., White, S.R.: An undetectable computer virus. In: Proceedings of Virus Bulletin Conference, vol. 5 (2000)Google Scholar
  30. 30.
    Filiol, E., Josse, S.: A statistical model for undecidable viral detection. Journal in Computer Virology 3(2), 65–74 (2007)CrossRefGoogle Scholar
  31. 31.
    Oreans Technologies,
  32. 32.
    Sipser, M.: Introduction to the Theory of Computation, vol. 27. Thomson Course Technology, Boston (2006)zbMATHGoogle Scholar
  33. 33.
    Shaw, D.E., Deneroff, M.M., Dror, R.O., Kuskin, J.S., Larson, R.H., Salmon, J.K., Young, C., Batson, B., Bowers, K.J., Chao, J.C., et al.: Anton, a special-purpose machine for molecular dynamics simulation. In: ACM SIGARCH Computer Architecture News, vol. 35, pp. 1–12. ACM (2007)Google Scholar
  34. 34.
    Stevens, M., Lenstra, A.K., de Weger, B.: Chosen-prefix collisions for MD5 and colliding X.509 certificates for different identities. In: Naor, M. (ed.) EUROCRYPT 2007. LNCS, vol. 4515, pp. 1–22. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  35. 35.
    Garey, M.R., Johnson, D.S.: Computers and Intractability, vol. 174. Freeman, New York (1979)zbMATHGoogle Scholar
  36. 36.
    Clarke, E.M., Emerson, E.A.: Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic. In: Engeler, E. (ed.) Logic of Programs 1979. LNCS, vol. 125, pp. 52–71. Springer, Heidelberg (1981)Google Scholar
  37. 37.
    Queille, J.P., Sifakis, J.: Specification and Verification of Concurrent Systems in CESAR. In: Dezani-Ciancaglini, M., Montanari, U. (eds.) Programming 1982. LNCS, vol. 137, pp. 337–351. Springer, Heidelberg (1982)CrossRefGoogle Scholar
  38. 38.
    Clarke, E.M.: The Birth of Model Checking. In: Grumberg, O., Veith, H. (eds.) 25MC Festschrift. LNCS, vol. 5000, pp. 1–26. Springer, Heidelberg (2008)Google Scholar
  39. 39.
    Sistla, A.P., Clarke, E.M.: The complexity of propositional linear temporal logics. J. ACM 32(3), 733–749 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Clarke, E.M., Emerson, E.A., Sistla, A.P.: Automatic Verification of Finite State Concurrent Systems Using Temporal Logic Specifications: A Practical Approach. In: POPL, pp. 117–126 (1983)Google Scholar
  41. 41.
    Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. In: LICS, pp. 428–439 (1990)Google Scholar
  42. 42.
    Marques-Silva, J.A.P., Sakallah, K.A.: GRASP-A New Search Algorithm for Satisfiability. In: Digest of IEEE International Conference on Computer-Aided Design, ICCAD, San Jose, California, pp. 220–227 (November 1996)Google Scholar
  43. 43.
    Marques-Silva, J.A.P., Sakallah, K.A.: GRASP: A Search Algorithm for Propositional Satisfiability. IEEE Transactions on Computers 48(5), 506–521 (1999)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Moskewicz, M.W., Madigan, C.F., Zhao, Y., Zhang, L., Malik, S.: Chaff: Engineering an Efficient SAT Solver. In: DAC, pp. 530–535 (2001)Google Scholar
  45. 45.
    Biere, A., Cimatti, A., Clarke, E.M., Zhu, Y.: Symbolic Model Checking without BDDs. In: Cleaveland, W.R. (ed.) TACAS 1999. LNCS, vol. 1579, pp. 193–207. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  46. 46.
    Clarke, E., Biere, A., Raimi, R., Zhu, Y.: Bounded Model Checking Using Satisfiability Solving. Form. Methods Syst. Des. 19, 7–34 (2001)CrossRefzbMATHGoogle Scholar
  47. 47.
    Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-Guided Abstraction Refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154–169. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  48. 48.
    Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-Guided Abstraction Refinement for Symbolic Model Checking. J. ACM 50, 752–794 (2003)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Bradley, A.R., Manna, Z.: Checking Safety by Inductive Generalization of Counterexamples to Induction. In: Formal Methods in Computer Aided Design, FMCAD 2007, pp. 173–180 (November 2007)Google Scholar
  50. 50.
    Bradley, A.R.: SAT-Based Model Checking without Unrolling. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 70–87. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  51. 51.
    Knuth, D.E.: Art of Computer Programming, 3rd edn. Fundamental Algorithms, vol. 1. Addison-Wesley Professional (July 1997)Google Scholar
  52. 52.
    Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the Spring Joint Computer Conference, AFIPS 1967 (Spring), April 18-20, pp. 483–485. ACM, New York (1967)CrossRefGoogle Scholar
  53. 53.
    Downey, R.G., Fellows, M.R., Stege, U.: Computational tractability: The view from mars. In: Bulletin of the European Association of Theoretical Computer Science, pp. 73–97Google Scholar
  54. 54.
    VMProtect Software,

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Denis Bueno
    • 1
  • Kevin J. Compton
    • 1
  • Karem A. Sakallah
    • 1
  • Michael Bailey
    • 1
  1. 1.Electrical Engineering and Computer Science DepartmentUniversity of MichiganUSA

Personalised recommendations