Denial-of-Service Attacks on Host-Based Generic Unpackers

  • Limin Liu
  • Jiang Ming
  • Zhi Wang
  • Debin Gao
  • Chunfu Jia
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5927)

Abstract

With the advance of packing techniques, a few generic and automatic unpackers have been proposed. These unpackers are designed to automatically unpack packed binaries without specific knowledge of the packing techniques used. In this paper, we present an automatic packer with which packed malware forges spurious unpacking behaviors that lead to a denial-of-service attack on host-based generic unpackers. We present the design, implementation, and evaluation of the proposed packer and malware produced using the proposed packer, and show the success of denial-of-service attacks on host-based generic unpackers.

Keywords

generic unpacker denial-of-service attack spurious unpacking behavior 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bohne, L.: Pandora’s Bochs: Automatic Unpacking of Malware. PhD thesis, University of Mannheim (2008)Google Scholar
  2. 2.
    Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., Veith, H.: Malware normalization. Technical report 1539, University of Wisconsin, Madison, WI, USA (2005)Google Scholar
  3. 3.
    Ferrie, P.: Attacks on virtual machine emulation. In: AVAR Conference. Symantec Advanced Threat Research (2006)Google Scholar
  4. 4.
    Ferrie, P.: Anti-unpacker tricks. In: Proceedings of the 2nd International CARO Workshop (2008)Google Scholar
  5. 5.
    Guo, F., Ferrie, P., Chiueh, T.C.: A study of the packer problem and its solutions. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 98–115. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Hawes, J.: Comparative review. Virus Bulletin, 14–27 (2009)Google Scholar
  7. 7.
    Intel Corporation. Using the RDTSC Instruction for Performance Monitoring (1997)Google Scholar
  8. 8.
    Kang, M.G., Poosankam, P., Yin, H.: Renovo: A hidden code extractor for packed executables. In: Proceedings of the 2007 ACM workshop on Recurring malcode (WORM), pp. 46–53 (2007)Google Scholar
  9. 9.
    Martignoni, L., Christodorescu, M., Jha, S.: Omniunpack: Fast, generic, and safe unpacking of malware. In: Proceedings of the 2007 Annual Computer Security Applications Conference(ACSAC), pp. 431–441. IEEE Computer Society, Los Alamitos (2007)Google Scholar
  10. 10.
    Morgenstern, M., Marx, A.: Runtime packer testing experiences. In: Proceedings of the 2nd International CARO Workshop (2008)Google Scholar
  11. 11.
    Quist, D.: Covert debugging: Circumventing software armoring techniques. In: Black Hat (2007)Google Scholar
  12. 12.
    Raffetseder, T., Krügel, C., Kirda, E.: Detecting system emulators. In: Proceedings of 10th International Conference on Information Security (ISC), pp. 1–18 (2007)Google Scholar
  13. 13.
    Royal, P.: Alternative medicine: The malware analyst’s blue pill. In: Black Hat (2008)Google Scholar
  14. 14.
    Royal, P., Halpin, M., Dagon, D., Edmonds, R., Lee, W.: Polyunpack: Automating the hidden-code extraction of unpack-executing malware. In: Proceedings of 2006 Annual Computer Security Applications Conference (ACSAC), pp. 289–300. IEEE Computer Society, Los Alamitos (2006)CrossRefGoogle Scholar
  15. 15.
    Sharif, M., Yegneswaran, V., Saidi, H., Porras, P., Lee, W.: Eureka: A framework for enabling static malware analysis. In: Proceedings of 13th European Symposium on Research in Computer Security (ESORICS), pp. 481–500 (2008)Google Scholar
  16. 16.
    Skape: Using dual-mappings to evade automated unpackers. Uninformed Journal (2008)Google Scholar
  17. 17.
    Stepan, A.: Improving proactive detection of packed malware. Virus Bulletin, 11–13 (2006)Google Scholar
  18. 18.
    Sun, L., Ebringer, T., Boztas, S.: Hump-and-dump: Efficient generic unpacking using an ordered address execution histogram. In: Second International CARO Workshop, Department of Computer Science and Software Engineering, The University of Melbourne, Australia (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Limin Liu
    • 1
  • Jiang Ming
    • 2
  • Zhi Wang
    • 2
    • 3
  • Debin Gao
    • 2
  • Chunfu Jia
    • 3
  1. 1.State Key Lab of Information SecurityGraduate University of CASChina
  2. 2.School of Information SystemsSingapore Management UniversitySingapore
  3. 3.College of Information Technology and ScienceNankai UniversityChina

Personalised recommendations