Skip to main content

Advertisement

SpringerLink
Log in

Applications of educational data mining and learning analytics on data from cybersecurity training

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Cybersecurity professionals need hands-on training to prepare for managing the current advanced cyber threats. To practice cybersecurity skills, training participants use numerous software tools in computer-supported interactive learning environments to perform offensive or defensive actions. The interaction involves typing commands, communicating over the network, and engaging with the training environment. The training artifacts (data resulting from this interaction) can be highly beneficial in educational research. For example, in cybersecurity education, they provide insights into the trainees’ learning processes and support effective learning interventions. However, this research area is not yet well-understood. Therefore, this paper surveys publications that enhance cybersecurity education by leveraging trainee-generated data from interactive learning environments. We identified and examined 3021 papers, ultimately selecting 35 articles for a detailed review. First, we investigated which data are employed in which areas of cybersecurity training, how, and why. Second, we examined the applications and impact of research in this area, and third, we explored the community of researchers. Our contribution is a systematic literature review of relevant papers and their categorization according to the collected data, analysis methods, and application contexts. These results provide researchers, developers, and educators with an original perspective on this emerging topic. To motivate further research, we identify trends and gaps, propose ideas for future work, and present practical recommendations. Overall, this paper provides in-depth insight into the recently growing research on collecting and analyzing data from hands-on training in security contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

The accompanying data and materials are published in a free, open-source repository on Zenodo: https://doi.org/10.5281/zenodo.6573117.

Notes

  1. For an in-depth overview of learning events and architecture for collecting them, see (Estévez-Ayres et al., 2017).

  2. When computing the median of participants, we performed a small simplification for P7: by assuming two or three people per team, we estimated 60 participants in the 24 teams.

  3. The counts sum to 127 because two authors had two affiliations.

References

  • Abbott, R.G., McClain, J., Anderson, B., Nauer, K., Silva, A., & Forsythe, C. (2015). Log analysis of cyber security training exercises. Procedia Manufacturing, 3, 5088–5094. Retrieved from https://doi.org/10.1016/j.promfg.2015.07.523.

    Article  Google Scholar 

  • Andreolini, M., Colacino, V.G., Colajanni, M., & Marchetti, M. (2019). A framework for the evaluation of trainee performance in cyber range exercises. Mobile Networks and Applications, 25, 236–247. Retrieved from https://doi.org/10.1007/s11036-019-01442-0.

    Article  Google Scholar 

  • Espinha Gasiba, T., Lechner, U., & Pinto-Albuquerque, M. (2020). Cybersecurity challenges in industry: measuring the challenge solve time to inform future challenges. Information, 11(11), 533. Retrieved from https://doi.org/10.3390/info11110533.

    Article  Google Scholar 

  • Estévez-Ayres, I., Arias Fisteus, J., & Delgado-Kloos, C. (2017). Lostrego: A distributed stream-based infrastructure for the real-time gathering and analysis of heterogeneous educational data. Journal of Network and Computer Applications, 100, 56–68. Retrieved from https://doi.org/10.1016/j.jnca.2017.10.014.

    Article  Google Scholar 

  • Graham, K., Anderson, J., Rife, C., Heitmeyer, B., Patel, P.R. , Nykl, S., Lin, A.C., & Merkle, D.L. (2020). Cyberspace odyssey: A competitive team-oriented serious game in computer networking. IEEE Transactions on Learning Technologies, 13(3), 502–515. Retrieved from https://doi.org/10.1109/TLT.2020.3008607.

    Article  Google Scholar 

  • Granåsen, M., & Andersson, D. (2016). Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study. Cognition, Technology & Work, 18(1), 121–143. Retrieved from https://doi.org/10.1007/s10111-015-0350-2.

    Article  Google Scholar 

  • Hundhausen, C., Olivares, D., & Carter, A. (2017). IDE-based learning analytics for computing education: a process model, critical review, and research agenda. ACM Transactions on Computing Education, 17(3), 11:1–11:26. Retrieved from https://doi.org/10.1145/3105759.

    Article  Google Scholar 

  • Imani, M., & Montazer, G.A. (2019). A survey of emotion recognition methods with emphasis on e-learning environments. Journal of Network and Computer Applications, 147, 102423. Retrieved from https://doi.org/10.1016/j.jnca.2019.102423.

    Article  Google Scholar 

  • Kasurinen, J., & Knutas, A. (2018). Publication trends in gamification: A systematic mapping study. Elsevier Computer Science Review, 27, 33–44. Retrieved from https://doi.org/10.1016/j.cosrev.2017.10.003.

    Article  Google Scholar 

  • Khando, K., Gao, S., Islam, S.M., & Salman, A. (2021). Enhancing employees information security awareness in private and public organisations: A systematic literature review. Computers & Security, 106, 102267. Retrieved from https://doi.org/10.1016/j.cose.2021.102267.

    Article  Google Scholar 

  • Knobbout, J., & Van Der Stappen, E. (2020). Where is the learning in learning analytics? a systematic literature review on the operationalization of learning-related constructs in the evaluation of learning analytics interventions. IEEE Transactions on Learning Technologies, 13(3), 631–645. Retrieved from https://doi.org/10.1109/TLT.2020.2999970.

    Article  Google Scholar 

  • Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. Retrieved from https://doi.org/10.1111/j.1468-2958.2004.tb00738.x.

    Google Scholar 

  • Liñán, L.C., & Pérez, Á.A.J. (2015). Educational data mining and learning analytics: differences, similarities, and time evolution. International Journal of Educational Technology in Higher Education, 12(3), 98–112. Retrieved from https://doi.org/10.7238/rusc.v12i3.2515.

    Google Scholar 

  • Maennel, K., Mäses, S., Sütterlin, S., Ernits, M., & Maennel, O. (2019). Using technical cybersecurity exercises in university admissions and skill evaluation. IFAC-PapersOnLine, 52(19), 169–174. Retrieved from https://doi.org/10.1016/j.ifacol.2019.12.169 (14th IFAC Symposium on Analysis, Design, and Evaluation of Human Machine Systems (HMS 2019)).

    Article  Google Scholar 

  • Mangaroska, K., & Giannakos, M. (2019). Learning analytics for learning design: a systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. Retrieved from https://doi.org/10.1109/TLT.2018.2868673.

    Article  Google Scholar 

  • Margulieux, L., Ketenci, T.A., & Decker, A. (2019). Review of measurements used in computing education research and suggestions for increasing standardization. Computer Science Education, 29(1), 49–78. Retrieved from https://doi.org/10.1080/08993408.2018.1562145.

    Article  Google Scholar 

  • Matcha, W., Uzir, N.A., Gašević, D., & Pardo, A. (2020). A systematic review of empirical studies on learning analytics dashboards: a self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13 (2), 226–245. Retrieved from https://doi.org/10.1109/TLT.2019.2916802.

    Article  Google Scholar 

  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & Group, T.P. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Medicine, 6(7), 1–6. Retrieved from https://doi.org/10.1371/journal.pmed.1000097.

    Article  Google Scholar 

  • Nunn, S.G., Avella, J.T., Kanai, T., & Kebritchi, M. (2016). Learning analytics methods, benefits, and challenges in higher education: a systematic literature review. Online Learning, 20(2), 13–29. Retrieved from https://doi.org/10.24059/olj.v20i2.790.

    Article  Google Scholar 

  • Peña-Ayala, A. (2014). Educational data mining: A survey and a data mining-based analysis of recent works. Expert Systems with Applications, 41(4, Part 1), 1432–1462. Retrieved from https://doi.org/10.1016/j.eswa.2013.08.042.

    Article  Google Scholar 

  • Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64, 1–18. Retrieved from https://doi.org/10.1016/j.infsof.2015.03.007.

    Article  Google Scholar 

  • Randolph, J. (2009). A guide to writing the dissertation literature review. Practical Assessment, Research, and Evaluation, 14(1), 13. Retrieved from https://doi.org/10.7275/b0az-8t74.

    Google Scholar 

  • Rupp, A.A., Levy, R., Dicerbo, K.E., Sweet, S.J., Crawford, A.V., Calico, T., Benson, M., Fay, D., Kunze, K.L., Mislevy, R.J., & Behrens, J.T. (2012). Putting ECD into practice: The interplay of theory and data in evidence models within a digital learning environment. Journal of Educational Data Mining (JEDM), 4(1), 49–110. Retrieved from https://doi.org/10.5281/zenodo.3554643.

    Google Scholar 

  • Tian, Z., Cui, Y., An, L., Su, S., Yin, X., Yin, L., & Cui, X. (2018). A real-time correlation of host-level events in cyber range service for smart campus. IEEE Access, 6, 35355–35364. Retrieved from https://doi.org/10.1109/ACCESS.2018.2846590.

    Article  Google Scholar 

  • Weiss, R., Turbak, F., Mache, J., & Locasto, M.E. (2017). Cybersecurity education and assessment in EDURange. IEEE Security & Privacy, 15 (3), 90–95. Retrieved from https://doi.org/10.1109/MSP.2017.54.

    Article  Google Scholar 

  • Yamin, M.M., Katt, B., & Gkioulos, V. (2020). Cyber ranges and security testbeds: Scenarios, functions, tools and architecture. Computers & Security, 88, 101636. Retrieved from https://doi.org/10.1016/j.cose.2019.101636.

    Article  Google Scholar 

  • Zurita, G., Shukla, A.K., Pino, J.A., Merigó, J.M., Lobos-Ossandón, V., & Muhuri, P.K. (2020). A bibliometric overview of the journal of network and computer applications between 1997 and 2019. Journal of Network and Computer Applications, 165, 102695. Retrieved from https://doi.org/10.1016/j.jnca.2020.102695.

    Article  Google Scholar 

  • (ISC)2 (2021) Cybersecurity Workforce Study (Tech. Rep.). Retrieved from https://www.isc2.org/Research/Workforce-Study.

  • Almansoori, M., Lam, J., Fang, E., Mulligan, K., Soosai Raj, A. G., & Chatterjee, R. (2020). How secure are our computer systems courses?. In Proceedings of the 2020 ACM conference on international computing education research (pp. 271–281). New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3372782.3406266.

  • Andreatos, A. S. (2017). Designing educational scenarios to teach network security. In 8Th IEEE global engineering education conference, EDUCON, 2017, (pp. 1606–1610). Washington, D.C., USA: IEEE Computer Society. Retrieved from https://doi.org/10.1109/EDUCON.2017.7943063.

  • Burket, J., Chapman, P., Becker, T., Ganas, C., & Brumley, D. (2015). Automatic problem generation for capture-the-flag competitions. In 2015 USENIX summit on gaming, games, and gamification in security education (3GSE 15) (pp. 1–8). Berkeley, CA, USA: USENIX Association. Retrieved from https://www.usenix.org/conference/3gse15/summit-program/presentation/burket.

  • CC2020 Task Force. (2020). Computing curricula 2020: Paradigms for global computing education. New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3467967.

  • CERN Data Centre & Invenio. (2022). Zenodo – research. Shared. Retrieved April 6, 2022 from https://zenodo.org.

  • Caliskan, E., Tatar, U., Bahsi, H., Ottis, R., & Vaarandi, R (2017). Capability detection and evaluation metrics for cyber security lab exercises. In Proceedings of the international conference on cyber warfare and security (pp. 407–414). Sonning Common Reading, UK, Academic Conferences and Publishing International.

  • Chapman, P., Burket, J., & Brumley, D. (2014). PicoCTF: A game-based computer security competition for high school students. In 2014 USENIX summit on gaming, games, and gamification in security education (3GSE 14) (pp. 1–10). Berkeley, CA, USA: USENIX Association. Retrieved from https://www.usenix.org/conference/3gse14/summit-program/presentation/chapman.

  • Chothia, T., Holdcroft, S., Radu, A.-I., & Thomas, R.J. (2017). Jail, hero or drug lord? turning a cyber security course into an 11 week choose your own adventure story. In 2017 USENIX workshop on advances in security education (ASE 17) (pp. 1–11). Berkeley, CA, USA: USENIX Association. Retrieved from https://www.usenix.org/conference/ase17/workshop-program/presentation/chothia.

  • Chothia, T., & Novakovic, C. (2015). An offline capture the flag-style virtual machine and an assessment of its value for cybersecurity education. In 2015 USENIX summit on gaming, games, and gamification in security education (3GSE 15) (pp. 1–8). Berkeley, CA, USA: USENIX Association. Retrieved from https://www.usenix.org/conference/3gse15/summit-program/presentation/chothia.

  • Clarivate. (2022). InCites journal citation reports. Retrieved April 6, 2022 from https://jcr.clarivate.com/jcr/home.

  • Computing Research and Education Association of Australasia. (2021). CORE. Retrieved April 6, 2022 from http://portal.core.edu.au/conf-ranks/.

  • Cybersecurity & Infrastructure Security Agency. (2018). Cyber Storm: Securing Cyber Space. Retrieved April 6, 2022 from https://www.cisa.gov/cyber-storm-securing-cyber-space.

  • DEF CON. (2021). CTF Archive. Retrieved April 6 2022 From https://www.defcon.org/html/links/dc-ctf.html.

  • Deng, Y., Lu, D., Chung, C. -J., Huang, D., & Zeng, Z. (2018). Personalized learning in a virtual hands-on lab platform for computer science education. In 2018 IEEE Frontiers in education conference (FIE) (pp. 1–8). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/FIE.2018.8659291.

  • Denny, P., Becker, B. A., Craig, M., Wilson, G., & Banaszkiewicz, P. (2019). Research this! questions that computing educators most want computing education researchers to answer. In Proceedings of the 2019 ACM conference on international computing education research (pp. 259–267). New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3291279.3339402.

  • Elsevier. (2021). Scopus. Retrieved April 6, 2022 from https://www.scopus.com.

  • Falah, A., Pan, L., & Chen, F. (2019). A Quantitative Approach to Design Special Purpose Systems to Measure Hacking Skills. In 2018 IEEE International conference on teaching, assessment, and learning for engineering (TALE) (pp. 54–61). New York NY, USA, IEEE. Retrieved from https://doi.org/10.1109/TALE.2018.8615431.

  • Fincher, S. A., & Robins, A. V. (2019). The cambridge handbook of computing education research. Cambridge, United Kingdom: Cambridge University Press. Retrieved from https://doi.org/10.1017/9781108654555.

  • Google. (2021). Capture the Flag. Retrieved April 6 2022 from https://capturetheflag.withgoogle.com.

  • Henshel, D.S., Deckard, G.M., Lufkin, B., Buchler, N., Hoffman, B., Rajivan, P., & Collman, S. (2016). Predicting proficiency in cyber defense team exercises. In MILCOM 2016 – IEEE military communications conference (pp. 776–781). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/MILCOM.2016.7795423.

  • Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M. A., Sheard, J., Skupas, B., Spacco, J., Szabo, C., & Toll, D. (2015). Educational data mining and learning analytics in programming: literature review and case studies. In Proceedings of the 2015 ITiCSE on working group reports (pp. 41–63). New York NY, USA, ACM. Retrieved from https://doi.org/10.1145/2858796.2858798.

  • Joint Task Force on Computing Curricula Association for Computing Machinery (ACM) and IEEE Computer Society. (2013). Computer science curricula 2013: curriculum guidelines for undergraduate degree programs in computer science. New York, NY, USA: ACM. Retrieved from https://doi.org/10.1145/2534860.

  • Joint Task Force on Cybersecurity Education. (2017). Cybersecurity curricular guideline. Retrieved April 6, 2022 from https://cybered.acm.org.

  • Kaneko, K., Igarashi, T., Kayama, K., Takeuchi, T., Suzuki, T., Kawase, A., Sunaga, T., Okuhara, M., & Okamura, K. (2020). Learning analytics with multi-faced data for cybersecurity education. In 9th International congress on advanced applied informatics (IIAI-AAI) (pp. 244–249). Retrieved from https://doi.org/10.1109/IIAI-AAI50415.2020.00055.

  • Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering (Tech. Rep.). EBSE.

  • Kokkonen, T., & Puuska, S. (2018). Blue team communication and reporting for enhancing situational awareness from white team perspective in cyber security exercises. In 18Th International conference on next generation teletraffic and wired/wireless advanced networks and systems, NEW2AN 2018 and 11th conference on internet of things and smart spaces, ruSMART, 2018 (Vol. 11118 LNCS, pp. 277–288). Vienna, Austria: Springer. Retrieved from https://doi.org/10.1007/978-3-030-01168-0_26.

  • Kont, M., Pihelgas, M., Maennel, K., Blumbergs, B., & Lepik, T. (2017). Frankenstack: Toward real-time red team feedback. In 2017 IEEE Military communications conference, MILCOM, 2017, (pp. 400–405), New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/MILCOM.2017.8170852.

  • Kucek, S., & Leitner, M. (2020). An empirical survey of functions and configurations of open-source capture the flag (CTF) environments. Journal of Network and Computer Applications 151. Retrieved from https://doi.org/10.1016/j.jnca.2019.102470.

  • Labuschagne, W. A., & Grobler, M. (2017). Developing a capability to classify technical skill levels within a cyber range. In 16Th European conference on cyber warfare and security, ECCWS 2017 (pp. 224–234). Red Hook, NY, USA, Curran Associates Inc. Retrieved from https://www.proquest.com/docview/1966803837.

  • Lang, C., Siemens, G., Wise, A., & Gašević, D. (2017). Handbook of Learning Analytics (1st edn). Society for Learning Analytics Research (SoLAR). Retrieved from https://doi.org/10.18608/hla17.

  • Lishinski, A., Good, J., Sands, P., & Yadav, A. (2016). Methodological rigor and theoretical foundations of CS education research. In Proceedings of the 2016 ACM conference on international computing education research, (pp. 161–169) New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/2960310.2960328.

  • Luxton-Reilly, A., Albluwi, I.S., Becker, B.A., Giannakos, M., Kumar, A.N., Ott, L., Paterson, J., Scott, M.J., Sheard, J., & Szabo, C. (2018). Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd annual ACM conference on innovation and technology in computer science education (pp. 55–106). New York NY, USA, ACM. Retrieved from https://doi.org/10.1145/3293881.3295779.

  • Maennel, K. (2020). Learning analytics perspective: evidencing learning from digital datasets in cybersecurity exercises. In 2020 IEEE European symposium on security and privacy workshops (EuroSPW), (pp. 27–36). Retrieved from https://doi.org/10.1109/EuroSPW51379.2020.00013.

  • Maennel, K., Ottis, R., & Maennel, O. (2017). Improving and measuring learning effectiveness at cyber defense exercises. In 22Nd nordic conference on secure IT systems, nordsec 2017, (pp. 123–138), Vienna, Austria, Springer. Retrieved from https://doi.org/10.1007/978-3-319-70290-2_8.

  • Malmi, L., Sheard, J., Bednarik, R.S., Helminen, J., Korhonen, A., Myller, N., Sorva, J., & Taherkhani, A (2010). Characterizing research in computing education: a preliminary analysis of the literature. In Proceedings of the sixth international workshop on computing education research (pp. 3–12) New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/1839594.1839597.

  • Mendeley. (2021). Reference Manager. Retrieved April 6, 2022 from https://www.mendeley.com/reference-management/reference-manager.

  • Nadeem, M., Allen, E. B., & Williams, B. J. (2015). A Method for Recommending Computer-Security Training for Software Developers: Leveraging the Power of Static Analysis Techniques and Vulnerability Repositories. In 12th International conference on information technology – new generations, (pp. 534–539). New York NY, USA, IEEE. Retrieved from https://doi.org/10.1109/ITNG.2015.90.

  • Natural Language Toolkit (NLTK) Project. (2022). Source code for nltk.metrics.agreement. Retrieved April 6, 2022 from http://www.nltk.org/_modules/nltk/metrics/agreement.html.

  • Palmer, N. (2019). Automating the Assessment of Network Security in Higher Education. In 2019 International conference on computing, electronics communications engineering (iCCECE) (pp. 141–146). Retrieved from https://doi.org/10.1109/iCCECE46942.2019.8941804.

  • Papamitsiou, Z., Giannakos, M., & Luxton-Reilly, SA (2020). Computing education research landscape through an analysis of keywords. In Proceedings of the 2020 ACM conference on international computing education research (pp. 102–112). New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3372782.3406276.

  • Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008). Systematic mapping studies in software engineering. In Proceedings of the 12th international conference on evaluation and assessment in software engineering (pp. 68–77). Swindon, UK: BCS Learning & Development Ltd. Retrieved https://dl.acm.org/doi/10.5555/2227115.2227123.

  • Reed, T., Nauer, K., & Silva, A. (2013). Instrumenting competition-based exercises to evaluate cyber defender situation awareness. In International conference on augmented cognition (pp. 80–89) Vienna, Austria, Springer. Retrieved from https://doi.org/10.1007/978-3-642-39454-6_9.

  • Rege, A., Obradovic, Z., Asadi, N., Parker, E., Masceri, N., Singer, B., & Pandit, R. (2017). Using a real-time cybersecurity exercise case study to understand temporal characteristics of cyberattacks. In Social, cultural, and behavioral modeling (pp. 127–132) Cham, Switzerland, Springer International Publishing. Retrieved from https://doi.org/10.1007/978-3-319-60240-0_16.

  • Romero, C., Ventura, S., Pechenizkiy, M., & Baker, R. S. (2010). Handbook of educational data mining. Boca Raton, FL, USA: CRC Press. Retrieved from: https://doi.org/10.1201/b10274.

  • Sanders, K., Sheard, J., Becker, B. A., Eckerdal, A., & Hamouda, S. ((2019). Inferential statistics in computing education research: a methodological review. In Proceedings of the 2019 ACM conference on international computing education research, (p. 177–185). New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3291279.3339408.

  • Sheng, Q. -W. (2020). Effectiveness evaluation of network security knowledge training based on machine learning. In S. Liu, G. Sun, & W Fu (Eds.) E-learning, e-education, and online training (pp. 25–37), Cham, Springer International Publishing. Retrieved from https://doi.org/10.1007/978-3-030-63955-6_3.

  • Simon, S., Becker, B. A., Hamouda, S., McCartney, R., Sanders, K., & Sheard, J. (2019). Visual portrayals of data and results at ITiCSE. In Proceedings of the 2019 ACM conference on innovation and technology in computer science education, (p. 51–57). New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3304221.3319742.

  • Švábenský, V., Čeleda, P., Vykopal, J., & Brišáková, S. (2020). Cybersecurity knowledge and skills taught in capture the flag challenges. Elsevier Computers & Security 102(102154). Retrieved from https://doi.org/10.1016/j.cose.2020.1021548.

  • Švábenský, V., & Vykopal, J. (2018a). Challenges arising from prerequisite testing in cybersecurity games. In Proceedings of the 49th ACM technical symposium on computer science education (pp. 56–61). New York NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3159450.3159454.

  • Švábenský, V., & Vykopal, J. (2018b). Gathering insights from teenagers’ hacking experience with authentic cybersecurity tools. In Proceedings of the 48th IEEE frontiers in education conference (pp. 1–4). New York NY, USA, IEEE. Retrieved from https://doi.org/10.1109/FIE.2018.8658840.

  • Švábenský, V., Vykopal, J., & Čeleda, P. (2020). What are cybersecurity education papers about? a systematic literature review of SIGCSE and ITiCSE conferences. In Proceedings of the 51st ACM technical symposium on computer science education (pp. 2–8). New York, NY, USA, Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3328778.3366816.

  • Taylor, C., Arias, P., Klopchic, J., Matarazzo, C., & Dube, E. (2017). CTF: State-of-the-art And building the next generation. In 2017 USENIX workshop on advances in security education (ASE 17). USENIX Association. Retrieved from https://www.usenix.org/conference/ase17/workshop-program/presentation/taylor.

  • The Graphics Replicability Stamp Initiative. (2017). GRSI. Retrieved April 6, 2022 from http://www.replicabilitystamp.org.

  • The NATO Cooperative Cyber Defence Centre of Excellence. (2021a). Crossed Swords. Retrieved April 6, 2022 from https://ccdcoe.org/exercises/crossed-swords.

  • The NATO Cooperative Cyber Defence Centre of Excellence. (2021b). Locked Shields. Retrieved April 6, 2022 from https://ccdcoe.org/exercises/locked-shields.

  • Tobarra, L., Robles-Gómez, A., Pastor, R., Hernández, R., Duque, A., & Cano, J. (2020). Students acceptance and tracking of a new container-based virtual laboratory. Applied Sciences 10(3) Retrieved from https://doi.org/10.3390/app10031091.

  • Tobarra, L., Trapero, A.P., Pastor, R., Robles-Gómez, A., Hernández, R., Duque, A., & Cano, J. (2020). Game-based learning approach to cybersecurity. In 2020 IEEE global engineering education conference (EDUCON) (EDUCON) (pp. 1125-1132). Retrieved from https://doi.org/10.1109/EDUCON45650.2020.9125202.

  • Tseng, S. -S., Lin, S. -C., Mao, C. -H., Lee, T. -J., Qiu, G. -W., & Lin, M. -H. (2017). An ontology guiding assessment framework for hacking competition. In 2017 10th International conference on ubi-media computing and workshops (Ubi-Media) (pp. 1–4). New York NY, USA, IEEE. Retrieved from https://doi.org/10.1109/UMEDIA.2017.8074131.

  • Vigna, G., Borgolte, K., Corbetta, J., Doupé, A., Fratantonio, Y., Invernizzi, L., Kirat, D., & Shoshitaishvili, Y. (2014). Ten Years of iCTF: The good, the bad, and the ugly. In 2014 USENIX summit on gaming, games, and gamification in security education (3GSE 14), (pp. 1–7). San Diego, CA: USENIX Association. Retrieved from https://www.usenix.org/conference/3gse14/summit-program/presentation/vigna.

  • Vykopal, J., & Barták, M. (2016). On the design of security games: from frustrating to engaging learning. In 2016 USENIX workshop on advances in security education (ASE 16) (pp. 1–8). Berkeley, CA, USA: USENIX Association. Retrieved from https://www.usenix.org/conference/ase16/workshop-program/presentation/vykopal.

  • Vykopal, J., Vizvary, M., Oslejsek, R., Celeda, P., & Tovarnak, D. (2017). Lessons learned from complex hands-on defence exercises in a cyber range. In 2017 IEEE Frontiers in education conference (FIE) (pp. 1–8), New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/FIE.2017.8190713.

  • Weiss, R., Locasto, M. E., & Mache, J. (2016). A reflective approach to assessing student performance in cybersecurity exercises. In Proceedings of the 47th ACM technical symposium on computing science education (pp. 597–602). New York NY, USA, ACM. Retrieved from https://doi.org/10.1145/2839509.2844646.

  • Yett, B., Snyder, C., Zhang, N., Hutchins, N., Mishra, S., & Biswas, G. (2020). Using log and discourse analysis to improve understanding of collaborative programming. In Proceedings of the 28th International Conference on Computers in Education (pp. 1–10). Retrieved from https://apsce.net/icce/icce2020/proceedings/paper_158.pdf.

  • Zeng, Z., Deng, Y., Hsiao, I., Huang, D., & Chung, C.-J. (2018). Improving student learning performance in a virtual hands-on lab system in cybersecurity education. In 2018 IEEE Frontiers In Education Conference (FIE) (pp. 1–5). New York, NY, USA, IEEE. Retrieved from https://doi.org/10.1109/FIE.2018.8658855.

Download references

Acknowledgements

This research was supported by the ERDF project CyberSecurity, CyberCrime and Critical Information Infrastructures Center of Excellence (No. CZ.02.1.01/0.0/0.0/16_019/0000822). The authors thank cybersecurity experts from the CSIRT-MU team who helped select the keywords for the automated search. We also thank Radek Pelánek for his comments on the early stages of this article.

Author information

Authors and Affiliations

  1. Institute of Computer Science, Masaryk University, Šumavská 15, 60200, Brno, Czech Republic

    Valdemar Švábenský, Jan Vykopal, Pavel Čeleda & Lydia Kraus

  2. Faculty of Informatics, Masaryk University, Botanická 68a, 60200, Brno, Czech Republic

    Valdemar Švábenský

Authors
  1. Valdemar Švábenský
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Vykopal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pavel Čeleda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lydia Kraus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valdemar Švábenský.

Ethics declarations

Conflict of interests

The authors have no competing interests to declare.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Švábenský, V., Vykopal, J., Čeleda, P. et al. Applications of educational data mining and learning analytics on data from cybersecurity training. Educ Inf Technol 27, 12179–12212 (2022). https://doi.org/10.1007/s10639-022-11093-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-022-11093-6

Keywords

Search

Navigation