A Review of Methods for Evaluating Security Awareness Initiatives

Abstract

The ‘human factor’ is commonly considered to be the weakest link in an organization’s security chain, and a significant percentage of companies have implemented security awareness (SA) programs to address this vulnerability. However, an element whose usefulness is still underestimated is the importance to perform measurements of the different SA programs’ effectiveness in order to assess their adequateness for achieving the intended goals. This gap has serious consequences as most of the security awareness campaigns have resulted to be largely unsuccessful. Awareness measurement tools might be determinant in providing feedback on the outcome of a program as well as in helping with the strategic planning for endorsing security. This article will introduce and critically compare a set of measurement methods. It will then discuss their attributes and suggested applications.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

References

  1. Abawajy J, Thatcher K, Kim TH (2008) Investigation of stakeholders commitment to information security awareness programs. In: 2008 international conference on information security and assurance (ISA 2008) IEEE, pp 472–476

  2. Academic Frontier Project for Private Universities (2009) Survey on the internet security awareness. March. https://www.kansai-u.ac.jp/riss/en/shareduse/data/17_E_questionnaire.pdf. Accessed 4 Sept 2019

  3. Agresti A (2018) An introduction to categorical data analysis. Wiley, New York

    Google Scholar 

  4. Al-Awadi M (2009) A study of employees’ attitudes towards organisational information security policies in the UK and Oman (Doctoral dissertation, University of Glasgow)

  5. BBC News (2004) The Chernobyl disaster. BBC special reports. http://news.bbc.co.uk/1/shared/spl/hi/guides/456900/456957/html/nn1page1.stm. Accessed 4 Sept 2019

  6. Bradburn NM, Sudman S, Wansink B (2004) Asking questions: the definitive guide to questionnaire design–for market research, political polls, and social and health questionnaires. Wiley

  7. Bresz FP (2004) People—often the weakest link in security, but one of the best places to start. J Health Care Compliance 6(4):57–60

    Google Scholar 

  8. Brink DE (2017) Security awareness training: small investment, large reduction risk. Aberdeen Group. https://www.proofpoint.com/us/resources/analyst-reports/aberdeen-security-awareness-training. Accessed 4 Sept 2019

  9. Brunner EM, Suter M (2008) International CIIP handbook 2008/2009. Center for Security Studies, ETH Zurich, Zurich

    Google Scholar 

  10. Bulgurcu B, Cavusoglu H, Benbasat I (2010) Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness. MIS Q 34(3):523–548

    Google Scholar 

  11. Byres E, Lowe J (2004) The myths and facts behind cyber security risks for industrial control systems. In: Proceedings of the VDE kongress, vol 116, pp 213–218

  12. Caputo DD, Pfleeger SL, Freeman JD, Johnson ME (2014) Going spear phishing: exploring embedded training and awareness. IEEE Secur Priv 12(1):28–38

    Google Scholar 

  13. Choo KKR (2011) The cyber threat landscape: challenges and future research directions. Comput Secur 30(8):719–731

    Google Scholar 

  14. Computer Fraud & Security (2016) News—employees prone to phishing. Comput Fraud Secur 2016(1):3. https://doi.org/10.1016/S1361-3723(16)30004-5

    Article  Google Scholar 

  15. Cone BD, Irvine CE, Thompson MF, Nguyen TD (2007) A video game for cyber security training and awareness. Comput Secur 26(1):63–72

    Google Scholar 

  16. Corbetta P (1999) Metodologia e tecniche della ricerca sociale. http://www.uniroma2.it/didattica/statistica_sociale_B/deposito/corbettametodologia_e_tecniche_della_ricerca_socialeriassunto.pdf. Accessed 4 Sept 2019

  17. Crossler RE, Johnston AC, Lowry PB, Hu Q, Warkentin M, Baskerville R (2013) Future directions for behavioral information security research. Comput Secur 32:90–101

    Google Scholar 

  18. Das SK, Kant K, Zhang N (2012) Handbook on securing cyber-physical critical infrastructure. Elsevier, Amsterdam

    Google Scholar 

  19. Davie M (1986) The Titanic: the full story of a tragedy. Random House, London

    Google Scholar 

  20. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 13:319–340

    Google Scholar 

  21. De Maggio MC, Mastrapasqua M, Tesei M, Chittaro A, Setola R (2017) How to improve the security awareness in complex organizations. Eur J Secur Res 4:1–17

    Google Scholar 

  22. Durkheim É (1897) Le suicide: étude de sociologie. Alcan, Paris

    Google Scholar 

  23. ENISA (2012) Introduction to return on security investment. https://www.enisa.europa.eu/publications/introduction-to-return-on-security-investment. Accessed 4 Sept 2019

  24. Furnell SM, Jusoh A, Katsabas D (2006) The challenges of understanding and using security: a survey of end-users. Comput Secur 25(1):27–35

    Google Scholar 

  25. Furnell SM, Bryant P, Phippen AD (2007) Assessing the security perceptions of personal Internet users. Comput Secur 26(5):410–417

    Google Scholar 

  26. Granger S (2001) Social engineering fundamentals, part I: hacker tactics. Secur Focus

  27. Groves RM, Fowler FJ Jr, Couper MP, Lepkowski JM, Singer E, Tourangeau R (2011) Survey methodology, vol 561. Wiley, New York

    Google Scholar 

  28. Gubrium JF, Holstein JA (2001) Handbook of interview research: context and method. Sage Publications, Thousand Oaks

    Google Scholar 

  29. Hansche S (2001) Designing a security awareness program: part I. Inf Syst Secur 9(6):14–23

    Google Scholar 

  30. Herath T, Rao HR (2009) Encouraging information security behaviors in organizations: role of penalties, pressures and perceived effectiveness. Decis Support Syst 47(2):154–165

    Google Scholar 

  31. Hills M, Anjali A (2017) A human factors contribution to countering insider threats: practical prospects from a novel approach to warning and avoiding. Secur J 30(1):142–152

    Google Scholar 

  32. Information Security Forum (2014) From promoting awareness to embedding behaviours. https://www.securityforum.org/uploads/2015/03/From-Promoting-Awareness-ES-2014_Marketing.pdf. Accessed 4 Sept 2019

  33. Karjalainen M, Siponen M (2011) Toward a new meta-theory for designing information systems (IS) security training approaches. Jo Assoc Inf Syst 12(8):518–555

    Google Scholar 

  34. Kruger HA, Kearney WD (2006) A prototype for assessing information security awareness. Comput Secur 25(4):289–296

    Google Scholar 

  35. Larsson R (1993) Case survey methodology: quantitative analysis of patterns across case studies. Acad Manag J 36(6):1515–1546

    Google Scholar 

  36. Lee R (2017) Crashoverride: analysis of the threat to electric grid operations. Dragos Inc., Rome

    Google Scholar 

  37. Mani D, Raymond Choo KK, Mubarak S (2014) Information security in the South Australian real estate industry: a study of 40 real estate organisations. Inf Manag Comput Secur 22(1):24–41

    Google Scholar 

  38. Manke S, Winkler I (2012) The habits of highly successful security awareness programs: a cross-company comparison. Technical report, secure mentem, 2012. http://www.securementem.com/wp-content/uploads/2013/07/Habits_white_paper.pdf. Accessed 4 Sept 2019

  39. Mishra S, Dhillon G (2006) Information systems security governance research: a behavioral perspective. In: 1st annual symposium on information assurance, academic track of 9th annual NYS cyber security conference, pp 27–35

  40. Mitnick KD, Simon WL (2011) The art of deception: controlling the human element of security. Wiley, New York

    Google Scholar 

  41. Moore T, Dynes S, Chang FR (2015) Identifying how firms manage cybersecurity investment. Southern Methodist University. http://blog.smu.edu/research/files/2015/10/SMU-IBM.pdf. Accessed 4 Sept 2019

  42. Moteff J, Parfomak P (2004) Critical infrastructure and key assets: definition and identification. Library of Congress Washington DC Congressional Research Service

  43. Muir A, Lopatto J (2004) Final report on the August 14, 2003 blackout in the United States and Canada: causes and recommendations

  44. Murchison C (1935) A handbook of social psychology. Clark University Press, Worcester, pp 789–844

    Google Scholar 

  45. NIST (1998) Information technology security training requirements: a role-and performance-based model (supersedes NIST Spec. Pub.500-172), SP 800-16, March

  46. Parsons K, McCormac A, Pattinson M, Butavicius M, Jerram C (2014) A study of information security awareness in Australian government organisations. Inf Manag Comput Secur 22(4):334–345

    Google Scholar 

  47. Pastor V, Díaz G, Castro M (2010). State-of-the-art simulation systems for information security education, training and awareness. In: 2010 IEEE education engineering (EDUCON). IEEE, pp 1907–1916

  48. Patrick AS, Long AC, Flinn S (2003). HCI and security systems. In: CHI’03 extended abstracts on human factors in computing systems. ACM, pp 1056–1057

  49. Peltier TR (2005) Implementing an information security awareness program. Inf Syst Secur 14(2):37–49

    Google Scholar 

  50. Perrow C (2011) Normal accidents: living with high risk technologies-updated edition. Princeton University Press, Princeton

    Google Scholar 

  51. Pescaroli G, Alexander D (2016) Critical infrastructure, panarchies and the vulnerability paths of cascading disasters. Nat Hazards 82(1):175–192

    Google Scholar 

  52. Pfleeger SL, Sasse MA, Furnham A (2014) From weakest link to security hero: transforming staff security behavior. J Homel Secur Emerg Manag 11(4):489–510

    Google Scholar 

  53. Ponemon Institute LLC (2012) The human factor in data protection. https://www.ponemon.org/blog/the-human-factor-in-data-protection. Accessed 4 Sept 2019

  54. Rahim NHA, Hamid S, Mat Kiah ML, Shamshirband S, Furnell S (2015) A systematic review of approaches to assessing cybersecurity awareness. Kybernetes 44(4):606–622

    Google Scholar 

  55. Reason J (2000) Human error: models and management. BMJ 320(7237):768–770

    Google Scholar 

  56. Ruan K (2017) Introducing cybernomics: a unifying economic framework for measuring cyber risk. Comput Secur 65:77–89

    Google Scholar 

  57. Ryan RM, Deci EL (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 55(1):68

    Google Scholar 

  58. Saaty TL (1988) What is the analytic hierarchy process? In: Mitra G (ed) Mathematical models for decision support. Springer, Berlin, pp 109–121

    Google Scholar 

  59. SANS (2012) Security awareness survey. Sans Institute, April. https://www.sans.org/sites/default/files/2018-01/security-awareness-survey.pdf. Accessed 4 Sept 2019

  60. Schein EH (2009) The corporate culture survival guide, vol 158. Wiley, New York

    Google Scholar 

  61. Schultz E (2005) From the Editor-in-Chief: the human factor in security. Comput Secur 24(6):425–426

    Google Scholar 

  62. Setola R, Mastrapasqua M, Tesei M, De Maggio MC, Corradini I, Pantaleo C, Capitello ME, De Simio F (2015) Study on security awareness in gas infrastructure. NITEL, March

  63. Setola R, Rosato V, Kyriakides E, Rome E (2016) Managing the complexity of critical infrastructures. In: Janusz K (ed) Studies in systems, decision and control book series, vol 90. Springer, Berlin

    Google Scholar 

  64. Siponen MT (2000) A conceptual foundation for organizational information security awareness. Inf Manag Comput Secur 8(1):31–41

    Google Scholar 

  65. Siponen MT (2001) Five dimensions of information security awareness. SIGCAS Comput Soc 31(2):24–29

    Google Scholar 

  66. Siponen M, Pahnila S, Mahmood MA (2010) Compliance with information security policies: an empirical investigation. Computer 43(2):64–71

    Google Scholar 

  67. Solms BV (2000) Information security—the third wave? Comput Secur 19(7):615–615

    Google Scholar 

  68. Soomro ZA, Shah MH, Ahmed J (2016) Information security management needs more holistic approach: a literature review. Int J Inf Manag 36(2):215–225

    Google Scholar 

  69. Stone DH (1993) Design a questionnaire. BMJ 307(6914):1264–1266

    Google Scholar 

  70. Szilagyi AD, Wallace MJ (1983) Organizational behavior and performance. Good Year Books, Culver

    Google Scholar 

  71. Talib S, Clarke NL, Furnell SM (2010) An analysis of information security awareness within home and work environments. In: 2010 international conference on availability, reliability and security. IEEE, pp 196–203

  72. Tsohou A, Kokolakis S, Karyda M, Kiountouzis E (2008) Investigating information security awareness: research and practice gaps. Inf Secur J Glob Perspect 17(5–6):207–227

    Google Scholar 

  73. US-Canada Power System Outage Task Force (2004) Final report on the August 14, 2003 blackout in the United States and Canada: causes and recommendations. US-Canada Power System Outage Task Force

  74. Velki T, Solic K, Ocevcic H (2014) Development of users’ information security awareness questionnaire (UISAQ)—ongoing work. In: 2014 37th international convention on information and communication technology, electronics and microelectronics (MIPRO). IEEE, pp 1417–1421

  75. Verizon (2016) 2016 data breach investigations report. http://www.verizonenterprise.com/verizon-insights-lab/dbir/. Accessed 4 Sept 2019

  76. Williams EJ, Hinds J, Joinson AN (2018) Exploring susceptibility to phishing in the workplace. Int J Hum Comput Stud 120:1–13

    Google Scholar 

  77. Wilson M, Hash J (2003) Building an information technology security awareness and training program. NIST Spec Publ 800(50):1–39

    Google Scholar 

  78. Workman M, Bommer WH, Straub D (2008) Security lapses and the omission of information security measures: a threat control model and empirical test. Comput Hum Behav 24(6):2799–2816

    Google Scholar 

  79. Zimmerman R (2004). Decision-making and the vulnerability of interdependent critical infrastructure. In: 2004 IEEE international conference on systems, man and cybernetics, vol 5. IEEE, pp 4059–4063

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Roberto Setola.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Assenza, G., Chittaro, A., De Maggio, M.C. et al. A Review of Methods for Evaluating Security Awareness Initiatives. Eur J Secur Res 5, 259–287 (2020). https://doi.org/10.1007/s41125-019-00052-x

Download citation

Keywords

  • Security awareness
  • Security awareness measurement
  • Security management
  • Critical infrastructure security