Stenmap: Framework for Evaluating Cybersecurity-Related Skills Based on Computer Simulations

  • Sten MäsesEmail author
  • Liina Randmann
  • Olaf Maennel
  • Birgy Lorenz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10925)


Cybersecurity exercises have become increasingly popular for training and assessing practical skills of information security. Nevertheless, the main focus of those exercises still tends to be on achieving completion or winning condition, not on understanding the individual skill set of each participant. This paper builds upon related work in creating and implementing cybersecurity exercises and proposes a revised process that includes competency mapping. A new conceptual framework called “Stenmap” is introduced that turns the results of cybersecurity simulations into a more meaningful evaluation of each participant’s individual skills. Cybersec-Tech window is introduced as a tool for discussing the high-level classification of cybersecurity-related skills and reaching a common understanding among exercise organisers coming from diverse backgrounds. An Estonian national cybersecurity competition is used as an example for implementing the suggested process and Stenmap framework.


Cybersecurity simulations Cybersecurity exercises Competency mapping Serious games 



This research was supported by the European Regional Development Fund of European Union and Estonian Ministry of Defence.


  1. 1.
    Conti, G., Babbitt, T., Nelson, J.: Hacking competitions and their untapped potential for security education. IEEE Secur. Priv. 9, 56–59 (2011)CrossRefGoogle Scholar
  2. 2.
    Knowles, W., Such, J.M., Gouglidis, A., Misra, G., Rashid, A.: All that glitters is not gold: on the effectiveness of cyber security qualifications. IEEE Computer 50, 60–71 (2017)CrossRefGoogle Scholar
  3. 3.
    Furnell, S., Fischer, P., Finch, A.: Can’t get the staff? The growing need for cyber-security skills. Comput. Fraud Secur. 2017, 5–10 (2017)Google Scholar
  4. 4.
    ENISA: Good Practice Guide on National Exercises. 80 (2009) Google Scholar
  5. 5.
    Bjerke, M.B., Renger, R.: Being smart about writing SMART objectives. Eval. Prog. Plann. 61, 125–127 (2017)CrossRefGoogle Scholar
  6. 6.
    Patriciu, V., Furtuna, A.C.: Guide for designing cyber security exercises 2. The need for a uniform structure. In: Proceedings of the 8th WSEAS International Conference on E-Activities and Information Security and Privacy, pp. 72–177 (2009)Google Scholar
  7. 7.
    Wilhelmson, N., Svensson, T.: Handbook for planning, running and evaluating information technology and cyber security exercises. Elanders Sverige AB, Vällingby (2014).,%20running%20and%20evaluating%20information%20technology%20and%20cyber%20security%20exercises.pdf. ISBN 978-91-86137-36-6
  8. 8.
    Katsantonis, M., Fouliras, P., Mavridis, I.: Conceptual analysis of cyber security education based on live competitions, pp. 771–779 (2017)Google Scholar
  9. 9.
    Bashir, M., Lambert, A., Wee, J.M.C., Guo, B.: An examination of the vocational and psychological characteristics of cybersecurity competition participants. In: USENIX Summit on Gaming, Games, and Gamification in Security Education, pp. 1–8 (2015)Google Scholar
  10. 10.
  11. 11.
    ENISA: European Cyber Security Challenge.
  12. 12.
    Chothia, T., Holdcroft, S., Radu, A., Thomas, R.J.: Jail, hero or drug lord? Turning a cyber security course into an 11 week choose your own adventure story. In: 2017 USENIX Workshop on Advances in Security Education (ASE 2017) (2017)Google Scholar
  13. 13.
    Dickey, M.D.: Game design narrative for learning: appropriating adventure game design narrative devices and techniques for the design of interactive learning environments. Educ. Tech. Res. Dev. 54, 245–263 (2006)CrossRefGoogle Scholar
  14. 14.
    Simpson, J., Coombes, P.: Adult learning as a hero’s journey: researching mythic structure as a model for transformational change. Queensland J. Educ. Res. 17, 164–177 (2001)Google Scholar
  15. 15.
    Chou, Y.: Actionable Gamification - Beyond Points, Badges, and Leaderboards. Octalysis Media, Fremont (2015)Google Scholar
  16. 16.
    Le Deist, F.D., Winterton, J.: What is competence? Hum. Resour. Dev. Int. 8, 27–46 (2005)CrossRefGoogle Scholar
  17. 17.
    Tobey, D.H.: A vignette-based method for improving cybersecurity talent management through cyber defense competition design. In: Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research, SIGMIS-CPR 2015, pp. 31–39. ACM, New York (2015)Google Scholar
  18. 18.
    Fernández-Sanz, L., Gómez-Pérez, J., Castillo-Martínez, A.: e-Skills match: a framework for mapping and integrating the main skills, knowledge and competence standards and models for ICT occupations. Comput. Stan. Interfaces 51, 30–42 (2017). Scholar
  19. 19.
    Newhouse, W., Keith, S., Scribner, B., Witte, G.: National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework. NIST Special Publication 800-181 (2017)Google Scholar
  20. 20.
    Mäses, S., Hallaq, B., Maennel, O.: Obtaining better metrics for complex serious games within virtualised simulation environments. In: 11th European Conference on Game-Based Learning, ECGBL 2017, pp. 428–434 (2017)Google Scholar
  21. 21.
    Umbleja, K.: Competence Based Learning – Framework, Implementation, Analysis and Management of Learning Process. TTÜ Press, TTÜ Kirjastus. (2017). ISBN 9789949830886
  22. 22.
    Starr, C.W., Manaris, B., Stalvey, R.H., Starr, C.W., Manaris, B., Stalvey, R.H.: Bloom’s taxonomy revisited. ACM SIGCSE Bull. 40, 261 (2008)CrossRefGoogle Scholar
  23. 23.
    Simonin, B.L.: N-loop learning: part I – of hedgehog, fox, dodo bird and sphinx. Learn. Organ. 24, 169–179 (2017)CrossRefGoogle Scholar
  24. 24.
    Brabrand, C., Dahl, B.: Using the SOLO taxonomy to analyze competence progression of university science curricula. High. Educ. 58, 531–549 (2009)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Sten Mäses
    • 1
    Email author
  • Liina Randmann
    • 1
  • Olaf Maennel
    • 1
  • Birgy Lorenz
    • 1
  1. 1.Tallinn University of TechnologyTallinnEstonia

Personalised recommendations