Art and Automation of Teaching Malware Reverse Engineering

  • Toomas LepikEmail author
  • Kaie Maennel
  • Margus Ernits
  • Olaf Maennel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10925)


The threat environment is rapidly changing and the cyber security skill shortage is a widely acknowledged problem. However, teaching such skills and keeping professionals up-to-date is not trivial. New malware types appear daily, and it requires significant time and effort by a teacher to prepare a unique, current and challenging courses in the malware reverse engineering. Novel teaching methods and tools are required. This paper describes an experience with an automated hands-on learning environment in a malware reverse engineering class taught at Tallinn University of Technology in Estonia. Our hands-on practical lab is using a fully automated Cyber Defense Competition platform Intelligent Training Exercise Environment (i-tee) [1] combined with typical Capture-The-Flag competition structure and open-source tools where possible. We describe the process of generating a unique and comparable reverse-engineering challenge and measuring the students’ progress through the process of analysis, reporting flags and debugging data, recording and taking into account their unique approach to the task. We aim to measure the students’ using the Bloom’s taxonomy, i.e., mastering the art of malware reverse engineering at the higher cognitive levels. The presented teaching and assessment method builds foundation for enhancing the future malware reverse engineering training quality and impact.


Higher education teaching Cyber defence exercises Malware reverse engineering 



This work is partially supported by the European Regional Development Fund (Tallinn University of Technology project VOSA - 2014–2020.4.01.16-0038)


  1. 1.
    Ernits, M., Tammekänd, J., Maennel, O.: i-tee: a fully automated cyber defense competition for students. ACM SIGCOMM Comput. Commun. Rev. 45, 113–114 (2015)CrossRefGoogle Scholar
  2. 2.
    Sisco, Z.D., Dudenhofer, P.P., Bryant, A.R.: Modeling information flow for an autonomous agent to support reverse engineering work. J. Def. Model. Simul. 14(3), 245–256 (2017)CrossRefGoogle Scholar
  3. 3.
    Mahoney, W., Gandhi, R.A.: Reverse engineering: is it art? ACM Inroads 3(1), 56–61 (2012)CrossRefGoogle Scholar
  4. 4.
    Eagle, C.: Computer security competitions: expanding educational outcomes. IEEE Secur. Priv. 11(4), 69–71 (2013)CrossRefGoogle Scholar
  5. 5.
    Hohwy, J.: The predictive mind. Oxford University Press, Oxford (2013)CrossRefGoogle Scholar
  6. 6.
    Ernits, M., Kikkas, K.: A live virtual simulator for teaching cybersecurity to information technology students. In: Zaphiris, P., Ioannou, A. (eds.) LCT 2016. LNCS, vol. 9753, pp. 474–486. Springer, Cham (2016). Scholar
  7. 7.
    CTF time. Accessed: 06 Feb 2018.
  8. 8.
    Burns, T.J., Rios, S.C., Jordan, T.K., Gu, Q., Underwood, T.: Analysis and exercises for engaging beginners in online CTF competitions for security education. In: USENIX Workshop on Advances in Security Education (ASE 17), USENIX Association (2017)Google Scholar
  9. 9.
    Chapman, P., Burket, J., Brumley, D.: PicocTF: A game-based computer security competition for high school students. In: 3GSE (2014)Google Scholar
  10. 10.
    Taylor, C., Colberg, C.: A tool for teaching reverse engineering. In: USENIX Workshop on Advances in Security Education (ASE 16), Austin, TX, USENIX Association (2016)Google Scholar
  11. 11.
    Gorder, P.F.: Multicore processors for science and engineering. Comput. Sci. Eng. 9(2), 3 (2007)CrossRefGoogle Scholar
  12. 12.
    Bisson, D.: The four most common evasive techniques used by malware, April 2015. Accessed 14 Jan 2018.
  13. 13.
    Biggs, J.B., Collis, K.F.: Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). Academic Press, Massachusetts (2014)Google Scholar
  14. 14.
    Buchanan, L., Wolanczyk, F., Zinghini, F.: Blending bloom’s taxonomy and serious game design. In: Proceedings of the 2011 International Conference on Security and Management, July 2011, SAM, vol. 11 (2011)Google Scholar
  15. 15.
    Moses, K.V., Petullo, W.M.: Teaching computer security (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Toomas Lepik
    • 1
    Email author
  • Kaie Maennel
    • 1
  • Margus Ernits
    • 1
  • Olaf Maennel
    • 1
  1. 1.Tallinn University of TechnologyTallinnEstonia

Personalised recommendations