Protocol Reverse Engineering: Challenges and Obfuscation

  • J. Duchêne
  • C. Le Guernic
  • E. Alata
  • V. Nicomette
  • M. Kaâniche
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10158)


Reverse engineering of communication protocols is aimed at providing methods and tools allowing to infer a model of these protocols. It is very relevant for many application domains, such as interoperability or security audits. Recently, several tools have been developed in order to automate, entirely or partially, the protocol inference process. These tools rely on several techniques, that are usually tuned and adapted according to the final goal of the reverse engineering task. The aim of this paper is (1) to present an overview of the main challenges related to reverse engineering, and (2) to introduce the use of obfuscation techniques to make the reverse engineering process more complex and difficult in particular to malicious users.


  1. 1.
    Antunes, J., Neves, N., Verissimo, P.: Reverse engineering of protocols from network traces. In: 2011 18th Working Conference on Reverse Engineering (WCRE), pp. 169–178. IEEE, New York (2011)Google Scholar
  2. 2.
    Bossert, G., Hiet, G., Henin, T.: Modelling to simulate botnet command and control protocols for the evaluation of network intrusion detection systems. In: 2011 Conference on Network and Information Systems Security (SAR-SSI), pp. 1–8. IEEE, La Rochelle (2011)Google Scholar
  3. 3.
    Bossert, G.: Exploiting semantic for the automatic reverse engineering of communication protocols. Ph.D. Thesis, Suplec December 2014Google Scholar
  4. 4.
    Bridger, H., Rishab, N., Phillipa, G., Rob, J.: Games without frontiers: investigating video games as a covert channel. In: Proceedings of the 2016 IEEE European Symposium on Security and Privacy, IEEE European Symposium on Security and Privacy. IEEE (2015)Google Scholar
  5. 5.
    Caballero, J., Poosankam, P., Kreibich, C., Song, D.: Dispatcher: enabling active botnet infiltration using automatic protocol reverse-engineering. In: Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS 2009, pp. 621–634. ACM, New York (2009)Google Scholar
  6. 6.
    Caballero, J., Song, D.: Rosetta: extracting protocol semantics using binary analysis with applications to protocol replay and NAT rewriting. Technical Report CMU-CyLab-07-014, Carnegie Mellon University, Pittsburgh, USA (2007)Google Scholar
  7. 7.
    Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, CCS 2007, pp. 317–329. ACM, New York (2007)Google Scholar
  8. 8.
    Caballero Bayerri, J.: Grammar and model extraction for security applications using dynamic program binary analysis. Ph.D. Thesis, Carnegie Mellon University, Pittsburgh, PA, USA (2010)Google Scholar
  9. 9.
    Cui, W., Paxson, V., Weaver, N., Katz, R.H.: Protocol-independent adaptive replay of application dialog. In: Proceedings of the 13th Annual Network and Distributed System Security Symposium (NDSS). Internet Society, San Diego, USA, February 2006.
  10. 10.
    Cui, W., Peinado, M., Chen, K., Wang, H.J., Irun-Briz, L.: Tupni: automatic reverse engineering of input formats. In: Proceedings of the 15th ACM Conference on Computer and Communications Security, CCS 2008, pp. 391–402. ACM, New York (2008)Google Scholar
  11. 11.
    de la Higuera, C.: Grammatical Inference: Learning Automata and Grammars. Cambridge University Press, New York (2010)Google Scholar
  12. 12.
    Hjelmvik, E., John, W.: Breaking and Improving Protocol Obfuscation. Technical Report 2010–05, Chalmers University of Technology, Gothenburg, Sweden (2010).
  13. 13.
    Leita, C., Mermoud, K., Dacier, M.: ScriptGen: an automated script generation tool for Honeyd. In: 21st Annual Computer Security Applications Conference, p. 214. IEEE, Tucson (2005)Google Scholar
  14. 14.
    Li, X., Chen, L.: A survey on methods of automatic protocol reverse engineering. In: 2011 Seventh International Conference on Computational Intelligence and Security (CIS), pp. 685–689. IEEE, Hainan (2011)Google Scholar
  15. 15.
    Newsome, J., Brumley, D., Franklin, J., Song, D.: Replayer: automatic protocol replay by binary analysis. In: Proceedings of the 13th ACM Conference on Computer and Communications Security, CCS 2006, pp. 311–321. ACM, New York (2006)Google Scholar
  16. 16.
    Samba Team: Opening windows to a wider world.

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • J. Duchêne
    • 1
    • 3
  • C. Le Guernic
    • 1
    • 2
  • E. Alata
    • 3
  • V. Nicomette
    • 3
  • M. Kaâniche
    • 3
  1. 1.DGA Maîtrise de l’informationRennesFrance
  2. 2.Laboratory High SecurityINRIA Team TAMISRennesFrance
  3. 3.LAAS-CNRSUniv. de Toulouse, CNRS, INSAToulouseFrance

Personalised recommendations