Network Protocol Discovery and Analysis via Live Interaction

  • Patrick LaRoche
  • A. Nur Zincir-Heywood
  • Malcolm I. Heywood
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7248)


In this work, we explore the use of evolutionary computing toward protocol analysis. The ability to discover, analyse, and experiment with unknown protocols is paramount within the realm of network security; our approach to this crucial analysis is to interact with a network service, discovering sequences of commands that do not result in error messages. In so doing, our work investigates the real-life responses of a service, allowing for exploration and analysis of the protocol in question. Our system initiates sequences of commands randomly, interacts with and learns from the responses, and modifies its next set of sequences accordingly. Such an exploration results in a set of command sequences that reflect correct uses of the service in testing. These discovered sequences can then be used to identify the service, unforeseen uses of the service, and, most importantly, potential weaknesses.


Network Protocol Error Message Evolutionary Computing Linear Genetic Programming Genetic Programming System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: Automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, p. 329. ACM (2007)Google Scholar
  2. 2.
    Doucette, J., Heywood, M.I.: Novelty-Based Fitness: An Evaluation under the Santa Fe Trail. In: Esparcia-Alcázar, A.I., Ekárt, A., Silva, S., Dignum, S., Uyar, A.Ş. (eds.) EuroGP 2010. LNCS, vol. 6021, pp. 50–61. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Froese, T.: Steps toward the evolution of communication in a multi-agent system. In: Symposium for Cybernetics Annual Research Projects, SCARP 2003. Citeseer (2003)Google Scholar
  4. 4.
    Heywood, M.I., Nur Zincir-Heywood, A.: Dynamic page based crossover in linear genetic programming. IEEE Transactions on Systems, Man, and Cybernetics: Part B - Cybernetics 32(3), 380–388 (2002)CrossRefGoogle Scholar
  5. 5.
    Huelsbergen, L.: Toward simulated evolution of machine language iteration. In: Koza, J.R., Goldberg, D.E., Fogel, D.B., Riolo, R.L. (eds.) Proceedings of the First Annual Conference on Genetic Programming 1996, July 28-31, pp. 315–320. Stanford University, MIT Press, CA, USA (1996)Google Scholar
  6. 6.
    Kaksonen, R., Laasko, M., Takanen, A.: Vulnerability analysis of software through syntax testing. University of Oulu, Finland, Tech. Rep. (2000)Google Scholar
  7. 7.
    Gunes Kayacik, H., Heywood, M.I., Nur Zincir-Heywood, A.: Evolving Buffer Overflow Attacks with Detector Feedback. In: Giacobini, M. (ed.) EvoWorkshops 2007. LNCS, vol. 4448, pp. 11–20. Springer, Heidelberg (2007)Google Scholar
  8. 8.
    Gunes Kayacyk, H., Nur Zincir-Heywood, A., Heywood, M.: Evolving successful stack overflow attacks for vulnerability testing. In: 21st Annual Computer Security Applications Conference, ACSAC 2005, pp. 225–234. IEEE Computer Society (December 2005)Google Scholar
  9. 9.
    Khasteh, S.H., Shouraki, S.B., Halavati, R., Khameneh, E.: Evolution of a communication protocol between a group of intelligent agents. In: World Automation Congress, WAC 2006, pp. 1–6. Citeseer (2006)Google Scholar
  10. 10.
    Khasteh, S.H., Shouraki, S.B., Halavati, R., Lesani, M.: Communication Protocol Evolution by Natural Selection. In: 2006 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, Computational Intelligence for Modelling, Control and Automation, p. 152 (2006)Google Scholar
  11. 11.
    LaRoche, P., Nur Zincir-Heywood, A., Heywood, M.I.: Evolving tcp/ip packets: A case study of port scans. In: CDROM: IEEE Symposium on Computational Intelligence for Security and Defense Applications (2009)Google Scholar
  12. 12.
    LaRoche, P., Nur Zincir-Heywood, A., Heywood, M.I.: Using Code Bloat to Obfuscate Evolved Network Traffic. In: Di Chio, C., Brabazon, A., Di Caro, G.A., Ebner, M., Farooq, M., Fink, A., Grahl, J., Greenfield, G., Machado, P., O’Neill, M., Tarantino, E., Urquhart, N. (eds.) EvoApplications 2010. LNCS, vol. 6025, pp. 101–110. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    LaRoche, P., Nur Zincir-Heywood, A., Heywood, M.I.: Exploring the state space of an application protocol: A case study of smtp. In: 2011 IEEE Symposium on Computational Intelligence in Cyber Security (CICS 2011), pp. 152–159 (April 2011)Google Scholar
  14. 14.
    Marquis, S., Dean, T.R., Knight, S.: Scl: a language for security testing of network applications. In: CASCON 2005: Proceedings of the 2005 Conference of the Centre for Advanced Studies on Collaborative Research, pp. 155–164. IBM Press (2005)Google Scholar
  15. 15.
    Nordin, P.: A compiling genetic programming system that directly manipulates the machine code. In: Kinnear Jr., K.E. (ed.) Advances in Genetic Programming, ch. 14, pp. 311–331. MIT Press (1994)Google Scholar
  16. 16.
    Postel, J., Reynolds, J.: File Transfer Protocol. RFC 959 (Standard), Updated by RFCs 2228, 2640, 2773, 3659, 5797 (October 1985)Google Scholar
  17. 17.
    Tal, O., Knight, S., Dean, T.: Syntax-based vulnerability testing of frame-based network protocols. In: Proc. 2nd Annual Conference on Privacy, Security and Trust (2004)Google Scholar
  18. 18.
    Wondracek, G., Comparetti, P.M., Kruegel, C., Kirda, E., Anna, S.S.S.: Automatic network protocol analysis. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium, NDSS 2008. Citeseer (2008)Google Scholar
  19. 19.
    Xiao, S., Deng, L., Li, S., Wang, X.: Integrated tcp/ip protocol software testing for vulnerability detection. In: 2003 International Conference on Computer Networks and Mobile Computing, ICCNMC 2003, pp. 311–319. IEEE (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Patrick LaRoche
    • 1
  • A. Nur Zincir-Heywood
    • 1
  • Malcolm I. Heywood
    • 1
  1. 1.Faculty of Computer ScienceDalhousie UniversityHalifaxCanada

Personalised recommendations