Cognition, Technology & Work

, Volume 18, Issue 1, pp 121–143 | Cite as

Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study

  • Magdalena Granåsen
  • Dennis AnderssonEmail author
Original Article


In 2010, IT-security experts from northern European governments and organizations gathered to conduct the first of a series of NATO-led cyber-defense exercises in a pilot attempt of training cyber defense. To gain knowledge on how to assess team effectiveness in cyber-defense exercises, this case study investigates the role of behavioral assessment techniques as a complement to task-based performance measurement. The collected data resulted in a massive data set including system logs, observer reports, and surveys. Six different methods were compared for feasibility in assessing the teams’ performance, including automated availability check, exploratory sequential data analysis, and network intrusion detection system attack analysis. In addition, observer reports and surveys were used to collect aspects relating to team structures and processes, aiming to discover whether these aspects can explain differences in effectiveness. The cross-disciplinary approach and multiple metrics create possibilities to study not only the performance-related outcome of the exercise, but also why this result is obtained. The main conclusions found are (1) a combination of technical performance measurements and behavioral assessment techniques are needed to assess team effectiveness, and (2) cyber situation awareness is required not only for the defending teams, but also for the observers and the game control.


Cyber-defense exercise Cyber SA Performance assessment Team cognition Team effectiveness 


  1. Andersson D (2009) F-REX: event driven synchronized multimedia model visualization. In: Proceedings of the 15th international conference on distributed multimedia systems. Knowledge Systems Institute, Redwood City, pp 140–145Google Scholar
  2. Andersson D (2011) Privacy and distributed tactical operations evaluation. In: Proceedings of the 4th international conference on advances in human-oriented and personalized mechanisms, technologies, and services. BarcelonaGoogle Scholar
  3. Andersson D (2013) A knowledge base for capturing comprehensive mission experience. P Ann HICCS 46. IEEE, Wailea. doi: 10.1109/HICSS.2013.40
  4. Andersson D (2014) An externalizable model of tactical mission control for knowledge transfer. Int J Inf Syst Crisis Response Manag 6(3):16–37. doi: 10.4018/IJISCRAM.2014070102 CrossRefGoogle Scholar
  5. Andersson D, Granåsen M, Sundmark T, Holm H, Hallberg J (2011) Analysis of a cyber defense exercise using exploratory sequential data analysis. In: Proceedings of the 16th international command and control research and technology symposium. DoD CCRP, Québec CityGoogle Scholar
  6. Barford P, Dacier M, Dietterich TG et al (2010) Cyber SA: situational awareness for cyber defense. In: Jajodia S, Liu P, Swarup V, Wang C (eds) Cyber situational awareness: advances in information security 46. Springer, Berlin, pp 3–13. doi: 10.1007/978-1-4419-0140-8_1 CrossRefGoogle Scholar
  7. Branlat M (2011) Challenges to adversarial interplay under high uncertainty: staged-world study of a cyber security event. Dissertation, Ohio State UniversityGoogle Scholar
  8. Champion MA, Rajivan P, Cooke NJ, Jariwala S (2012) Team-based cyber defense analysis. In: P CogSIMA 2. IEEE, New Orleans, pp 218–221. doi: 10.1109/CogSIMA.2012.6188386 Google Scholar
  9. Conklin A (2006) Cyber defense competitions and information security education: an active learning solution for a capstone course. P Ann HICCS 39, Kauai. doi: 10.1109/HICSS.2006.110
  10. Cooke NJ, Salas E, Kiekel PA, Bell B (2004) Advances in measuring team cognition. In: Salas E, Fiore SM (eds) Team cognition: understanding the factors that drive process and performance. American Psychological Association, Washington, pp 83–106CrossRefGoogle Scholar
  11. Cowger CD (1984) Statistical significance tests: scientific ritualism or scientific method? Soc Serv Rev 58:358–372CrossRefGoogle Scholar
  12. Cowger CD (1985) Author’s reply. Soc Serv Rev 59:520–522CrossRefGoogle Scholar
  13. Doupé A, Egele M, Caillat B et al (2011) Hit’em where it hurts: a live security exercise on cyber situational awareness. In: P ACSAC 27: 51–61. ACM, OrlandoGoogle Scholar
  14. Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. Human Factors 37:32–64. doi: 10.1518/001872095779049543 CrossRefGoogle Scholar
  15. Endsley MR (2000) Direct measurement of situation awareness: validity and use of SAGAT. In: Endsley MR, Garland DJ (eds) Situation awareness analysis and measurement. Lawrence Erlbaum, MahwahGoogle Scholar
  16. Flyvbjerg B (2011) Case study. In: Denzin NK, Lincoln YS (eds) The Sage handbook of qualitative research, 4th edn. Sage, Thousand Oaks, pp 301–316Google Scholar
  17. Franke U, Brynielsson J (2014) Cyber situational awareness—a systematic review of the literature. Comput Secur 46:18–31. doi: 10.1016/j.cose.2014.06.008 CrossRefGoogle Scholar
  18. Geers K (2010) Live fire exercise: preparing for cyber war. J Homel Secur Emerg 7. doi: 10.2202/1547-7355.1780 Google Scholar
  19. Greenemeier L (2007) China’s cyber attacks signal new battlefield is online. Scientific American, New YorkGoogle Scholar
  20. Hammervik M, Andersson D, Hallberg J (2010) Capturing a cyber defence exercise. In: Proceedings of the first national symposium on technology and methodology for security and crisis management, Linköping, SwedenGoogle Scholar
  21. Hoffman LJ, Rosenberg T, Dodge R, Ragsdale D (2005) Exploring a national cybersecurity exercise for universities. IEEE Secur Priv 3:27–33. doi: 10.1109/MSP.2005.120 CrossRefGoogle Scholar
  22. Holm H, Ekstedt M, Andersson D (2012) Empirical analysis of system-level vulnerability metrics through actual attacks. IEEE Trans Dependable Secur 9:825–837. doi: 10.1109/TDSC.2012.66 CrossRefGoogle Scholar
  23. Igure VM, Laughter SA, Williams RD (2006) Security issues in SCADA networks. Comput Secur 25:498–506. doi: 10.1016/j.cose.2006.03.001 CrossRefGoogle Scholar
  24. Lim KH, Benbasat I (2000) The effect of multimedia on perceived equivocality and perceived usefulness of information systems. MIS Q 24:449–471. doi: 10.2307/3250969 CrossRefGoogle Scholar
  25. Malek J (2005) Informed consent. In: Mitcham C (ed) Encyclopedia of science, technology and ethics, vol 2. Macmillan, Detroit, pp 1016–1019Google Scholar
  26. NATO (2010) Cyber defence exercise baltic cyber shield 2010: after action report. CCDCoE, TallinnGoogle Scholar
  27. NATO (2012) Cyber defence exercise locked shields 2012: after action report. CCDCoE, TallinnGoogle Scholar
  28. NATO (2013) Cyber defence exercise locked shields 2013: after action report. CCDCoE, TallinnGoogle Scholar
  29. Otondo RF, van Scotter JR, Allen DG, Palvia P (2008) The complexity of richness: media, message, and communication outcomes. Inf Manag 40:21–30. doi: 10.1016/ CrossRefGoogle Scholar
  30. Pfleeger SL, Caputo DD (2012) Leveraging behavioral science to mitigate cyber security risk. Comput Secur 31:597–611. doi: 10.1016/j.cose.2011.12.010 CrossRefGoogle Scholar
  31. Pilemalm S, Andersson D, Hallberg N (2008) Reconstruction and exploration of large-scale distributed operations: multimedia tools for evaluation of emergency management response. J Emerg Manag 6:31–47Google Scholar
  32. Riegelsberger J, Sasse MA, McCarthy J (2003) The researcher’s dilemma: evaluating trust in computer-mediated communication. Int J Human Comput Stud 58:759–781. doi: 10.1016/S1071-5819(03)00042-9 CrossRefGoogle Scholar
  33. Rubin A (1985) Significance testing with population data. Soc Serv Rev 59:518–520CrossRefGoogle Scholar
  34. Salas E, Sims DE, Burke CS (2005) Is there a “Big Five” in teamwork? Small Group Res 36:555–599. doi: 10.1177/1046496405277134 CrossRefGoogle Scholar
  35. Sanderson PM, Fisher C (1994) Exploratory sequential data analysis: foundations. Human Comput Interact 9:251–317. doi: 10.1207/s15327051hci0903&4_2 CrossRefGoogle Scholar
  36. Sommestad T, Hallberg J (2012) Cyber security exercises and competitions as a platform for cyber security experiments. In: Jøsang A, Carlsson B (eds) Proceedings of the 17th Nordic conference on secure IT systems. Springer, Berlin, pp 47–60. doi: 10.1007/978-3-642-34210-3_4 CrossRefGoogle Scholar
  37. Stake RE (1995) The art of case study research. Sage, Thousand OaksGoogle Scholar
  38. Thorstensson M (2012) Supporting observers in the field to perform model based data collection. In: Rothkrantz L, Ristvej J, Franco Z (eds) P ISCRAM 9. Simon Fraser University, Vancouver, CanadaGoogle Scholar
  39. Tyworth M, Giacobe NA, Mancuso V, Dancy C (2012) The distributed nature of cyber situation awareness. In: P CogSIMA 2. IEEE, New Orleans, pp 174–178. doi: 10.1109/CogSIMA.2012.6188375 Google Scholar
  40. Wildman JL, Salas E, Scott CPR (2013) Measuring cognition in teams: a cross-domain review. Human Factors 56:911–941. doi: 10.1177/0018720813515907 CrossRefGoogle Scholar
  41. Yin RK (2009) Case study research: design and methods, 4th edn. Sage, Thousand OaksGoogle Scholar

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  1. 1.Division for Information- and Aeronautical SystemsSwedish Defense Research AgencyLinköpingSweden

Personalised recommendations