Using Reinforcement Learning to Conceal Honeypot Functionality

  • Seamus DowlingEmail author
  • Michael Schukat
  • Enda Barrett
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11053)


Automated malware employ honeypot detecting mechanisms within its code. Once honeypot functionality has been exposed, malware such as botnets will cease the attempted compromise. Subsequent malware variants employ similar techniques to evade detection by known honeypots. This reduces the potential size of a captured dataset and subsequent analysis. This paper presents findings on the deployment of a honeypot using reinforcement learning, to conceal functionality. The adaptive honeypot learns the best responses to overcome initial detection attempts by implementing a reward function with the goal of maximising attacker command transitions. The paper demonstrates that the honeypot quickly identifies the best response to overcome initial detection and subsequently increases attack command transitions. It also examines the structure of a captured botnet and charts the learning evolution of the honeypot for repetitive automated malware. Finally it suggests changes to an existing taxonomy governing honeypot development, based on the learning evolution of the adaptive honeypot. Code related to this paper is available at:


Reinforcement learning Honeypot Adaptive 


  1. 1.
    Bellovin, S.M.: Packets found on an internet. ACM SIGCOMM Comput. Commun. Rev. 23(3), 26–31 (1993)CrossRefGoogle Scholar
  2. 2.
    Pa, Y.M.P., Suzuki, S., Yoshioka, K., Matsumoto, T., Kasama, T., Rossow, C.: IoTPOT: analysing the rise of IoT compromises. EMU 9, 1 (2015)Google Scholar
  3. 3.
    Watson, D., Riden, J.: The honeynet project: data collection tools, infrastructure, archives and analysis. In: WOMBAT Workshop on 2008 IEEE Information Security Threats Data Collection and Sharing. WISTDCS 2008, pp. 24–30 (2008)Google Scholar
  4. 4.
    Provos, N., et al.: A virtual honeypot framework. In: USENIX Security Symposium, vol. 173, pp. 1–14 (2004)Google Scholar
  5. 5.
    Krawetz, N.: Anti-honeypot technology. IEEE Secur. Priv. 2(1), 76–79 (2004)CrossRefGoogle Scholar
  6. 6.
    Kolias, C., Kambourakis, G., Stavrou, A., Voas, J.: DDoS in the IoT: Mirai and other botnets. Computer 50(7), 80–84 (2017)CrossRefGoogle Scholar
  7. 7.
    Spitzner, L.: Honeypots: Tracking Hackers, vol. 1. Addison-Wesley, Reading (2003)Google Scholar
  8. 8.
    Zhang, F., et al.: Honeypot: a supplemented active defense system for network security. In: 2003 Proceedings of the Fourth International Conference on Parallel and Distributed Computing, Applications and Technologies. PDCAT-2003, pp. 231–235. IEEE (2003)Google Scholar
  9. 9.
    Spitzner, L.: Honeytokens: the other honeypot (2003). Accessed 17 Feb 2014
  10. 10.
    Seifert, C., Welch, I., Komisarczuk, P.: Taxonomy of honeypots. Technical report CS-TR-06/12, School of Mathematical and Computing Sciences, Victoria University of Wellington, June 2006Google Scholar
  11. 11.
    Kuwatly, I., et al.: A dynamic honeypot design for intrusion detection. In: 2004 Proceedings of The IEEE/ACS International Conference on Pervasive Services. ICPS 2004, pp. 95–104. IEEE (2004)Google Scholar
  12. 12.
    Prasad, R., Abraham, A.: Hybrid framework for behavioral prediction of network attack using honeypot and dynamic rule creation with different context for dynamic blacklisting. In: 2010 Second International Conference on Communication Software and Networks. ICCSN 2010, pp. 471–476. IEEE (2010)Google Scholar
  13. 13.
    Jicha, A., Patton, M., Chen, H.: SCADA honeypots: an indepth analysis of Conpot. In: 2016 IEEE Conference on Intelligence and Security Informatics (ISI) 2016Google Scholar
  14. 14.
    Vormayr, G., Zseby, T., Fabini, J.: Botnet communication patterns. IEEE Commun. Surv. Tutor. 19(4), 2768–2796 (2017)CrossRefGoogle Scholar
  15. 15.
    Wang, P., et al.: Honeypot detection in advanced botnet attacks. Int. J. Inf. Comput. Secur. 4(1), 30–51 (2010)Google Scholar
  16. 16.
    Holz, T., Raynal, F.: Detecting honeypots and other suspicious environments. In: 2005 Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop. IAW 2005, pp. 29–36. IEEE (2005)Google Scholar
  17. 17.
    Antonakakis, M., et al.: Understanding the Mirai botnet. In: USENIX Security Symposium, pp. 1092–1110 (2017)Google Scholar
  18. 18.
    Valli, C., Rabadia, P., Woodward, A.: Patterns and patter-an investigation into SSH activity using kippo honeypots (2013)Google Scholar
  19. 19.
    Not capturing any Mirai samples. Accessed 02 Feb 2018
  20. 20.
    SSH Mirai-like bot. Accessed 28 Nov 2017
  21. 21.
    Khattak, S., et al.: A taxonomy of botnet behavior, detection, and defense. IEEE Commun. Surv. Tutor. 16(2), 898–924 (2014)CrossRefGoogle Scholar
  22. 22.
    Hayatle, O., Otrok, H., Youssef, A.: A Markov decision process model for high interaction honeypots. Inf. Secur. J.: A Global Perspect. 22(4), 159–170 (2013)Google Scholar
  23. 23.
    Ghourabi, A., Abbes, T., Bouhoula, A.: Characterization of attacks collected from the deployment of Web service honeypot. Secur. Commun. Netw. 7(2), 338–351 (2014)CrossRefGoogle Scholar
  24. 24.
    Goseva-Popstojanova, K., Anastasovski, G., Pantev, R.: Using multiclass machine learning methods to classify malicious behaviors aimed at web systems. In: 2012 IEEE 23rd International Symposium on Software Reliability Engineering (ISSRE), pp. 81–90. IEEE (2012)Google Scholar
  25. 25.
    Wagener, G., Dulaunoy, A., Engel, T., et al.: Heliza: talking dirty to the attackers. J. Comput. Virol. 7(3), 221–232 (2011)CrossRefGoogle Scholar
  26. 26.
    Wagener, G., State, R., Dulaunoy, A., Engel, T.: Self adaptive high interaction honeypots driven by game theory. In: Guerraoui, R., Petit, F. (eds.) SSS 2009. LNCS, vol. 5873, pp. 741–755. Springer, Heidelberg (2009). Scholar
  27. 27.
    Pauna, A., Bica, I.: RASSH-reinforced adaptive SSH honeypot. In: 2014 10th International Conference on Communications (COMM), pp. 1–6. IEEE (2014)Google Scholar
  28. 28.
    Schaul, T., et al.: PyBrain. J. Mach. Learn. Res. 11(Feb), 743–746 (2010)Google Scholar
  29. 29.
    Initial analysis of four million login attempts. Accessed 17 Nov 2017
  30. 30.
    Dowling, S., Schukat, M., Melvin, H.: A ZigBee honeypot to assess IoT cyberattack behaviour. In: 2017 28th Irish Signals and Systems Conference (ISSC), pp. 1–6. IEEE (2017)Google Scholar
  31. 31.
    Dowling, S., Schukat, M., Barrett, E.: Improving adaptive honeypot functionality with efficient reinforcement learning parameters for automated malware. J. Cyber Secur. Technol. 1–17 (2018) Scholar
  32. 32.
    An adaptive honeypot using reinforcement learning implementation. Accessed 19 Dec 2017
  33. 33.
    Bringer, M.L., Chelmecki, C.A., Fujinoki, H.: A survey: recent advances and future trends in honeypot research. Int. J. Comput. Netw. Inf. Secur. 4(10), 63 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Galway Mayo Institute of TechnologyCastlebar, MayoIreland
  2. 2.National University of Ireland GalwayGalwayIreland

Personalised recommendations