Abstract
With the advance of packing techniques, a few generic and automatic unpackers have been proposed. These unpackers are designed to automatically unpack packed binaries without specific knowledge of the packing techniques used. In this paper, we present an automatic packer with which packed malware forges spurious unpacking behaviors that lead to a denial-of-service attack on host-based generic unpackers. We present the design, implementation, and evaluation of the proposed packer and malware produced using the proposed packer, and show the success of denial-of-service attacks on host-based generic unpackers.
This research was mostly done when the first three authors, Limin Liu, Jiang Ming, and Zhi Wang, were researchers working in Singapore Management University. It was partially supported by National Science Foundation (NSF) China under the agreements 90718005, 70890084/G021102, and 60573015.
Chapter PDF
Similar content being viewed by others
References
Bohne, L.: Pandora’s Bochs: Automatic Unpacking of Malware. PhD thesis, University of Mannheim (2008)
Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., Veith, H.: Malware normalization. Technical report 1539, University of Wisconsin, Madison, WI, USA (2005)
Ferrie, P.: Attacks on virtual machine emulation. In: AVAR Conference. Symantec Advanced Threat Research (2006)
Ferrie, P.: Anti-unpacker tricks. In: Proceedings of the 2nd International CARO Workshop (2008)
Guo, F., Ferrie, P., Chiueh, T.C.: A study of the packer problem and its solutions. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 98–115. Springer, Heidelberg (2008)
Hawes, J.: Comparative review. Virus Bulletin, 14–27 (2009)
Intel Corporation. Using the RDTSC Instruction for Performance Monitoring (1997)
Kang, M.G., Poosankam, P., Yin, H.: Renovo: A hidden code extractor for packed executables. In: Proceedings of the 2007 ACM workshop on Recurring malcode (WORM), pp. 46–53 (2007)
Martignoni, L., Christodorescu, M., Jha, S.: Omniunpack: Fast, generic, and safe unpacking of malware. In: Proceedings of the 2007 Annual Computer Security Applications Conference(ACSAC), pp. 431–441. IEEE Computer Society, Los Alamitos (2007)
Morgenstern, M., Marx, A.: Runtime packer testing experiences. In: Proceedings of the 2nd International CARO Workshop (2008)
Quist, D.: Covert debugging: Circumventing software armoring techniques. In: Black Hat (2007)
Raffetseder, T., Krügel, C., Kirda, E.: Detecting system emulators. In: Proceedings of 10th International Conference on Information Security (ISC), pp. 1–18 (2007)
Royal, P.: Alternative medicine: The malware analyst’s blue pill. In: Black Hat (2008)
Royal, P., Halpin, M., Dagon, D., Edmonds, R., Lee, W.: Polyunpack: Automating the hidden-code extraction of unpack-executing malware. In: Proceedings of 2006 Annual Computer Security Applications Conference (ACSAC), pp. 289–300. IEEE Computer Society, Los Alamitos (2006)
Sharif, M., Yegneswaran, V., Saidi, H., Porras, P., Lee, W.: Eureka: A framework for enabling static malware analysis. In: Proceedings of 13th European Symposium on Research in Computer Security (ESORICS), pp. 481–500 (2008)
Skape: Using dual-mappings to evade automated unpackers. Uninformed Journal (2008)
Stepan, A.: Improving proactive detection of packed malware. Virus Bulletin, 11–13 (2006)
Sun, L., Ebringer, T., Boztas, S.: Hump-and-dump: Efficient generic unpacking using an ordered address execution histogram. In: Second International CARO Workshop, Department of Computer Science and Software Engineering, The University of Melbourne, Australia (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Liu, L., Ming, J., Wang, Z., Gao, D., Jia, C. (2009). Denial-of-Service Attacks on Host-Based Generic Unpackers. In: Qing, S., Mitchell, C.J., Wang, G. (eds) Information and Communications Security. ICICS 2009. Lecture Notes in Computer Science, vol 5927. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11145-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-642-11145-7_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11144-0
Online ISBN: 978-3-642-11145-7
eBook Packages: Computer ScienceComputer Science (R0)