Abstract
The human aspect of cybersecurity continues to present challenges to researchers and practitioners worldwide. While measures are being taken to improve the situation, a vast majority of security incidents can be attributed to user behavior. Security and Awareness Training (SAT) has been available for several decades and is commonly given as a suggestion for improving the cybersecurity behavior of end-users. However, attackers continue to exploit the human factor suggesting that current SAT methods are not enough. Researchers argue that providing knowledge alone is not enough, and some researchers suggest that many currently used SAT methods are, in fact, not empirically evaluated. This paper aims to examine how SAT has been evaluated in recent research using a structured literature review. The result is an overview of evaluation methods which describes what results that can be obtained using them. The study further suggests that SAT methods should be evaluated using a variety of methods since different methods will inevitably provide different results. The presented results can be used as a guide for future research projects seeking to develop or evaluate methods for SAT.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Al-Daeef, M.M., Basir, N., Saudi, M.M.: Security awareness training: a review. Proc. World Congress Eng. 1, 5–7 (2017)
Alshaikh, M., Maynard, S.B., Ahmad, A., Chang, S.: An exploratory study of current information security training and awareness practices in organizations. In: Proceedings of the 51st Hawaii International Conference on System Sciences (2018)
Ayyagari, R., Figueroa, N.: Is seeing believing? training users on information security: evidence from java applets. J. Inf. Syst. Educ. 28(2), 115–120 (2017)
Boss, S., Galletta, D., Lowry, P.B., Moody, G.D., Polak, P.: What do systems users have to fear? using fear appeals to engender threats and fear that motivate protective security behaviors. MIS Q. (MISQ) 39(4), 837–864 (2015)
Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Res. Psychol. 3(2), 77–101 (2006)
Bulgurcu, B., Cavusoglu, H., Benbasat, I.: Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness. MIS Quarterly 34(3), 523–548 (2010)
Burris, J., Deneke, W., Maulding, B.: Activity simulation for experiential learning in cybersecurity workforce development. In: Nah, F.F.-H., Xiao, B.S. (eds.) HCIBGO 2018. LNCS, vol. 10923, pp. 17–25. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91716-0_2
Campbell, D.T.: Factors relevant to the validity of experiments in social settings. Psychol. Bull. 54(4), 297 (1957)
Choi, K.H., Lee, D.H.: A study on strengthening security awareness programs based on an rfid access control system for inside information leakage prevention. Multimed. Tools Appl. 74(20), 8927–8937
Cole, J.R., Pence, T., Cummings, J., Baker, E.: Gamifying security awareness: a new prototype. In: Moallem, A. (ed.) HCII 2019. LNCS, vol. 11594, pp. 115–133. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22351-9_8
Cuchta, T., et al.: Human risk factors in cybersecurity, pp. 87–92
Cybint: (2020) https://www.cybintsolutions.com/cyber-security-facts-stats/
Desman, M.B.: The ten commandments of information security awareness training. Inf. Secur. J. A Glob. Perspect. 11(6), 39–44 (2003)
Dincelli, E., Chengalur-Smith, I.: Choose your own training adventure: designing a gamified seta artefact for improving information security and privacy through interactive storytelling. European Journal of Information Systems
EC-Council: (2019). https://blog.eccouncil.org/the-top-types-of-cybersecurity-attacks-of-2019-till-date/
Eck, J.E., Liu, L.: Contrasting simulated and empirical experiments in crime prevention. J. Exp. Criminol. 4(3), 195–213 (2008)
Gjertsen, E.G.B., Gjaere, E.A., Bartnes, M., Flores, W.R.: Gamification of Information Security Awareness and Training. Icissp (2017)
Gokul, C.J., Pandit, S., Vaddepalli, S., Tupsamudre, H., Banahatti, V., Lodha, S., Acm: PHISHY - a serious game to train enterprise users on phishing awareness. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (2018)
Gundu, T.: Acknowledging and Reducing the Knowing and Doing gap in Employee Cybersecurity Compliance, pp. 94–102. International Conference on Cyber Warfare and Security (2019)
Huynh, D., Luong, P., Iida, H., Beuran, R.: Design and evaluation of a cybersecurity awareness training game. In: Munekata, N., Kunita, I., Hoshino, J. (eds.) ICEC 2017. LNCS, vol. 10507, pp. 183–188. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66715-7_19
Jayakrishnan, G.C., Sirigireddy, G.R., Vaddepalli, S., Banahatti, V., Lodha, S.P., Pandit, S.S.: Passworld: a serious game to promote password awareness and diversity in an enterprise. In: (SOUPS 2020), pp. 1–18 (2020)
Jesson, J., Matheson, L., Lacey, F.M.: Doing your literature review: Traditional and systematic techniques. Sage (2011)
Joinson, A., van Steen, T.: Human aspects of cyber security: behaviour or culture change? Cyber Secur. Peer-Reviewed J. 1(4), 351–360 (2018)
Kunz, A., Volkamer, M., Stockhardt, S., Palberg, S., Lottermann, T., Piegert, E.: Nophish: evaluation of a web application that teaches people being aware of phishing attacks, vol. P-259, pp. 509–518 (2016)
Lastdrager, E., Gallardo, I.C., Hartel, P., Junger, M.: How effective is anti-phishing training for children? pp. 229–239 (2017)
Lim, I.K., Park, Y.G., Lee, J.K.: Design of security training system for individual users. Wirel. Personal Commun. 90(3), 1105–1120 (2016)
Meline, T.: Selecting studies for systematic review: inclusion and exclusion criteria. Contemporary Issues in Communication Science and Disorders 33(21–27) (2006)
Micallef, N., Arachchilage, N.A.G.: Involving users in the design of a serious game for security questions education. arXiv preprint arXiv:1710.03888 (2017)
Moreno-Fernández, M.M., Blanco, F., Garaizar, P., Matute, H.: Fishing for phishers. improving internet users’ sensitivity to visual deception cues to prevent electronic fraud. Comput. Hum. Behav. 69, 421–436 (2017)
Paré, G., Kitsiou, S.: Methods for literature reviews. In: Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. University of Victoria (2017)
Parsons, K., Butavicius, M.A., Lillie, M., Calic, D., McCormac, A., Pattinson, M.R.: Which individual, cultural, organisational and interventional factors explain phishing resilience? In: HAISA, pp. 1–11 (2018)
Puhakainen, P., Siponen, M.: Improving employees’ compliance through information systems security training: an action research study. MIS quarterly, pp. 757–778 (2010)
Rastenis, J., Ramanauskaitė, S., Janulevičius, J., Čenys, A.: Impact of information security training on recognition of phishing attacks: A case study of vilnius gediminas technical university. vol. 1243. CCIS, pp. 311–324
Reinheimer, B., et al.: An investigation of phishing awareness and education over time: when and how to best remind users. In: (SOUPS 2020), pp. 259–284 (2020)
Renaud, K., Zimmermann, V.: Ethical guidelines for nudging in information security & privacy. Int. J. Hum. Comput. Stud. 120, 22–35 (2018)
Safa, N.S., Von Solms, R.: An information security knowledge sharing model in organizations. Comput. Hum. Behav. 57, 442–451 (2016)
Silic, M., Lowry, P.B.: Using design-science based gamification to improve organizational security training and compliance. J. Manage. Inf. Syst. 37(1), 129–161 (2020)
Siponen, M.T.: A conceptual foundation for organizational information security awareness. Information Management & Computer Security (2000)
Soare, B.: (2020). https://heimdalsecurity.com/blog/vectors-of-attack/
Stockhardt, Simon, et al.: Teaching phishing-security: which way is best? In: Hoepman, Jaap-Henk., Katzenbeisser, Stefan (eds.) SEC 2016. IAICT, vol. 471, pp. 135–149. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33630-5_10
Takata, T., Ogura, K., IEEE: Confront Phishing Attacks - from a Perspective of Security Education, pp. 10–13. International Conference on Awareness Science and Technology (2019)
Taneski, V., Heričko, M., Brumen, B.: Impact of security education on password change, pp. 1350–1355 (2015)
Tichy, W.F.: Should computer scientists experiment more? Computer 31(5), 32–40 (1998)
Tschakert, K.F., Ngamsuriyaroj, S.: Effectiveness of and user preferences for security awareness training methodologies. Heliyon 5(6), e02010 (2019)
Van Rensburg, W.J., Thomson, K.L., Futcher, L.: An educational intervention towards safe smartphone usage. In: HAISA 2018 (2018)
Vroom, C., Von Solms, R.: Towards information security behavioural compliance. Comput. Secur. 23(3), 191–198 (2004)
Wen, Z.A., Lin, Z.Q., Chen, R., Andersen, E.: What. Hack: engaging anti-phishing training through a role-playing phishing simulation game. In: Chi 2019 (2019)
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer Science & Business Media (2012)
Xiong, A.P., Proctor, R.W., Yang, W.N., Li, N.H.: Embedding training within warnings improves skills of identifying phishing webpages. Human Factors 61(4), 577–595 (2019)
Yang, W., Xiong, A., Chen, J., Proctor, R.W., Li, N.: Use of phishing training to improve security warning compliance: Evidence from a field experiment. vol. Part F127186, pp. 52–61 (2017)
Zhou, L.M., Parmanto, B., Alfikri, Z., Bao, J.: A mobile app for assisting users to make informed selections in security settings for protecting personal health data: Development and feasibility study. Jmir Mhealth and Uhealth 6(12), e11210 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 IFIP International Federation for Information Processing
About this paper
Cite this paper
Kävrestad, J., Nohlberg, M. (2021). Evaluation Strategies for Cybersecurity Training Methods: A Literature Review. In: Furnell, S., Clarke, N. (eds) Human Aspects of Information Security and Assurance. HAISA 2021. IFIP Advances in Information and Communication Technology, vol 613. Springer, Cham. https://doi.org/10.1007/978-3-030-81111-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-81111-2_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-81110-5
Online ISBN: 978-3-030-81111-2
eBook Packages: Computer ScienceComputer Science (R0)