Abstract
Automatic protocol reverse engineering has recently received significant attention due to its importance to many security applications. However, previous methods are all limited in analyzing only plain-text communications wherein the exchanged messages are not encrypted. In this paper, we propose ReFormat, a system that aims at deriving the message format even when the message is encrypted. Our approach is based on the observation that an encrypted input message will typically go through two phases: message decryption and normal protocol processing. These two phases can be differentiated because the corresponding instructions are significantly different. Further, with the help of data lifetime analysis of run-time buffers, we can pinpoint the memory locations that contain the decrypted message generated from the first phase and are later accessed in the second phase. We have developed a prototype and evaluated it with several real-world protocols. Our experiments show that ReFormat can accurately identify decrypted message buffers and then reveal the associated message structure.
Chapter PDF
Similar content being viewed by others
References
Paxson, V.: Bro: A System for Detecting Network Intruders in Real-Time. Computer Networks 31(23-24), 2345–2463 (1999)
Wang, H.J., Guo, C., Simon, D.R., Zugenmaier, A.: Shield: Vulnerability-Driven Network Filters for Preventing Known Vulnerability Exploits. In: Proceedings of ACM SIGCOMM 2004, pp. 193–204 (2004)
Cui, W., Peinado, M., Wang, H.J., Locasto, M.: Shieldgen: Automatic Data Patch Generation for Unknown Vulnerabilities with Informed Probing. In: Proceedings of 2007 IEEE Symposium on Security and Privacy, Oakland, CA (May 2007)
The Protocol Informatics Project, http://www.baselineresearch.net/PI/
Caballero, J., Song, D.: Polyglot: Automatic Extraction of Protocol Format using Dynamic Binary Analysis. In: Proceedings of the 14th ACM Conference on Computer and and Communications Security, CCS 2007 (2007)
Cui, W., Kannan, J., Wang, H.J.: Discoverer: Automatic Protocol Reverse Engineering from Network Traces. In: Proceedings of the 16th USENIX Security Symposium (Security 2007), Boston, MA (August 2007)
Cui, W., Peinado, M., Chen, K., Wang, H.J., Irun-Briz, L.: Tupni: Automatic Reverse Engineering of Input Formats. In: Proceedings of the 15th ACM Conferences on Computer and Communication Security, CCS 2008 (October 2008)
Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic Protocol Format Reverse Engineering Through Context-Aware Monitored Execution. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008) (February 2008)
Wondracek, G., Comparetti, P.M., Kruegel, C., Kirda, E.: Automatic Network Protocol Analysis. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008) (February 2008)
Comparetti, P.M., Wondracek, G., Kruegel, C., Kirda, E.: Prospex: Protocol Specification Extraction. In: Proceedings of 2009 IEEE Symposium on Security and Privacy, Oakland, CA (May 2009)
Chow, J., Pfaff, B., Christopher, K., Rosenblum, M.: Understanding Data Lifetime via Whole-System Simulation. In: Proceedings of the 13th USENIX Security Symposium, San Diego, CA (2004)
Newsome, J., Song, D.: Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. In: Proceedings of the 14th Annual Network and Distributed System Security Symposium (NDSS 2005), San Diego, CA (February 2005)
SHTTP: An Embeddable Web Server, http://shttpd.sourceforge.net/
Know your Enemy: Tracking Botnets - Bot-Commands, http://www.honeynet.org/papers/bots/botnet-commands.html
Wang, Z., Jiang, X., Cui, W., Wang, X.: Reformat: Automatic Reverse Engineering of Encrypted Messages (Department of Computer Science Technical Report, North Carolina State University, TR-2008-26) (2008)
Ircd-hybrid – High Performance Internet Relay Chat, http://ircd-hybrid.com/
Cui, W., Paxson, V., Weaver, N., Katz, R.H.: Protocol-Independent Adaptive Replay of Application Dialog. In: Proceedings of the 13th Annual Network and Distributed System Security Symposium (NDSS 2006), San Diego, CA (February 2006)
Leita, C., Mermoud, K., Dacier, M.: ScriptGen: An Automated Script Generation Tool for Honeyd. In: Srikanthan, T., Xue, J., Chang, C.-H. (eds.) ACSAC 2005. LNCS, vol. 3740, pp. 203–214. Springer, Heidelberg (2005)
Chow, J., Pfaff, B., Garfinkel, T., Rosenblum, M.: Shredding Your Garbage: Reducing Data Lifetime through Secure Deallocation. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, Maryland (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, Z., Jiang, X., Cui, W., Wang, X., Grace, M. (2009). ReFormat: Automatic Reverse Engineering of Encrypted Messages. In: Backes, M., Ning, P. (eds) Computer Security – ESORICS 2009. ESORICS 2009. Lecture Notes in Computer Science, vol 5789. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04444-1_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-04444-1_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04443-4
Online ISBN: 978-3-642-04444-1
eBook Packages: Computer ScienceComputer Science (R0)