Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study

Abstract

In 2010, IT-security experts from northern European governments and organizations gathered to conduct the first of a series of NATO-led cyber-defense exercises in a pilot attempt of training cyber defense. To gain knowledge on how to assess team effectiveness in cyber-defense exercises, this case study investigates the role of behavioral assessment techniques as a complement to task-based performance measurement. The collected data resulted in a massive data set including system logs, observer reports, and surveys. Six different methods were compared for feasibility in assessing the teams’ performance, including automated availability check, exploratory sequential data analysis, and network intrusion detection system attack analysis. In addition, observer reports and surveys were used to collect aspects relating to team structures and processes, aiming to discover whether these aspects can explain differences in effectiveness. The cross-disciplinary approach and multiple metrics create possibilities to study not only the performance-related outcome of the exercise, but also why this result is obtained. The main conclusions found are (1) a combination of technical performance measurements and behavioral assessment techniques are needed to assess team effectiveness, and (2) cyber situation awareness is required not only for the defending teams, but also for the observers and the game control.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    IBM SPSS, Commercial statistical analysis software, http://www.ibm.com/software/analytics/spss.

  2. 2.

    F-REX, Tools for Reconstruction and Exploration of heterogeneous datasets (Andersson 2009).

  3. 3.

    Snort, Open source network intrusion detection software, http://www.snort.org.

References

  1. Andersson D (2009) F-REX: event driven synchronized multimedia model visualization. In: Proceedings of the 15th international conference on distributed multimedia systems. Knowledge Systems Institute, Redwood City, pp 140–145

  2. Andersson D (2011) Privacy and distributed tactical operations evaluation. In: Proceedings of the 4th international conference on advances in human-oriented and personalized mechanisms, technologies, and services. Barcelona

  3. Andersson D (2013) A knowledge base for capturing comprehensive mission experience. P Ann HICCS 46. IEEE, Wailea. doi:10.1109/HICSS.2013.40

  4. Andersson D (2014) An externalizable model of tactical mission control for knowledge transfer. Int J Inf Syst Crisis Response Manag 6(3):16–37. doi:10.4018/IJISCRAM.2014070102

    Article  Google Scholar 

  5. Andersson D, Granåsen M, Sundmark T, Holm H, Hallberg J (2011) Analysis of a cyber defense exercise using exploratory sequential data analysis. In: Proceedings of the 16th international command and control research and technology symposium. DoD CCRP, Québec City

  6. Barford P, Dacier M, Dietterich TG et al (2010) Cyber SA: situational awareness for cyber defense. In: Jajodia S, Liu P, Swarup V, Wang C (eds) Cyber situational awareness: advances in information security 46. Springer, Berlin, pp 3–13. doi:10.1007/978-1-4419-0140-8_1

    Google Scholar 

  7. Branlat M (2011) Challenges to adversarial interplay under high uncertainty: staged-world study of a cyber security event. Dissertation, Ohio State University

  8. Champion MA, Rajivan P, Cooke NJ, Jariwala S (2012) Team-based cyber defense analysis. In: P CogSIMA 2. IEEE, New Orleans, pp 218–221. doi:10.1109/CogSIMA.2012.6188386

    Google Scholar 

  9. Conklin A (2006) Cyber defense competitions and information security education: an active learning solution for a capstone course. P Ann HICCS 39, Kauai. doi:10.1109/HICSS.2006.110

  10. Cooke NJ, Salas E, Kiekel PA, Bell B (2004) Advances in measuring team cognition. In: Salas E, Fiore SM (eds) Team cognition: understanding the factors that drive process and performance. American Psychological Association, Washington, pp 83–106

    Google Scholar 

  11. Cowger CD (1984) Statistical significance tests: scientific ritualism or scientific method? Soc Serv Rev 58:358–372

    Article  Google Scholar 

  12. Cowger CD (1985) Author’s reply. Soc Serv Rev 59:520–522

    Article  Google Scholar 

  13. Doupé A, Egele M, Caillat B et al (2011) Hit’em where it hurts: a live security exercise on cyber situational awareness. In: P ACSAC 27: 51–61. ACM, Orlando

  14. Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. Human Factors 37:32–64. doi:10.1518/001872095779049543

    Article  Google Scholar 

  15. Endsley MR (2000) Direct measurement of situation awareness: validity and use of SAGAT. In: Endsley MR, Garland DJ (eds) Situation awareness analysis and measurement. Lawrence Erlbaum, Mahwah

    Google Scholar 

  16. Flyvbjerg B (2011) Case study. In: Denzin NK, Lincoln YS (eds) The Sage handbook of qualitative research, 4th edn. Sage, Thousand Oaks, pp 301–316

    Google Scholar 

  17. Franke U, Brynielsson J (2014) Cyber situational awareness—a systematic review of the literature. Comput Secur 46:18–31. doi:10.1016/j.cose.2014.06.008

    Article  Google Scholar 

  18. Geers K (2010) Live fire exercise: preparing for cyber war. J Homel Secur Emerg 7. doi:10.2202/1547-7355.1780

    Google Scholar 

  19. Greenemeier L (2007) China’s cyber attacks signal new battlefield is online. Scientific American, New York

    Google Scholar 

  20. Hammervik M, Andersson D, Hallberg J (2010) Capturing a cyber defence exercise. In: Proceedings of the first national symposium on technology and methodology for security and crisis management, Linköping, Sweden

  21. Hoffman LJ, Rosenberg T, Dodge R, Ragsdale D (2005) Exploring a national cybersecurity exercise for universities. IEEE Secur Priv 3:27–33. doi:10.1109/MSP.2005.120

    Article  Google Scholar 

  22. Holm H, Ekstedt M, Andersson D (2012) Empirical analysis of system-level vulnerability metrics through actual attacks. IEEE Trans Dependable Secur 9:825–837. doi:10.1109/TDSC.2012.66

    Article  Google Scholar 

  23. Igure VM, Laughter SA, Williams RD (2006) Security issues in SCADA networks. Comput Secur 25:498–506. doi:10.1016/j.cose.2006.03.001

    Article  Google Scholar 

  24. Lim KH, Benbasat I (2000) The effect of multimedia on perceived equivocality and perceived usefulness of information systems. MIS Q 24:449–471. doi:10.2307/3250969

    Article  Google Scholar 

  25. Malek J (2005) Informed consent. In: Mitcham C (ed) Encyclopedia of science, technology and ethics, vol 2. Macmillan, Detroit, pp 1016–1019

    Google Scholar 

  26. NATO (2010) Cyber defence exercise baltic cyber shield 2010: after action report. CCDCoE, Tallinn

    Google Scholar 

  27. NATO (2012) Cyber defence exercise locked shields 2012: after action report. CCDCoE, Tallinn

    Google Scholar 

  28. NATO (2013) Cyber defence exercise locked shields 2013: after action report. CCDCoE, Tallinn

    Google Scholar 

  29. Otondo RF, van Scotter JR, Allen DG, Palvia P (2008) The complexity of richness: media, message, and communication outcomes. Inf Manag 40:21–30. doi:10.1016/j.im.2007.09.003

    Article  Google Scholar 

  30. Pfleeger SL, Caputo DD (2012) Leveraging behavioral science to mitigate cyber security risk. Comput Secur 31:597–611. doi:10.1016/j.cose.2011.12.010

    Article  Google Scholar 

  31. Pilemalm S, Andersson D, Hallberg N (2008) Reconstruction and exploration of large-scale distributed operations: multimedia tools for evaluation of emergency management response. J Emerg Manag 6:31–47

    Google Scholar 

  32. Riegelsberger J, Sasse MA, McCarthy J (2003) The researcher’s dilemma: evaluating trust in computer-mediated communication. Int J Human Comput Stud 58:759–781. doi:10.1016/S1071-5819(03)00042-9

    Article  Google Scholar 

  33. Rubin A (1985) Significance testing with population data. Soc Serv Rev 59:518–520

    Article  Google Scholar 

  34. Salas E, Sims DE, Burke CS (2005) Is there a “Big Five” in teamwork? Small Group Res 36:555–599. doi:10.1177/1046496405277134

    Article  Google Scholar 

  35. Sanderson PM, Fisher C (1994) Exploratory sequential data analysis: foundations. Human Comput Interact 9:251–317. doi:10.1207/s15327051hci0903&4_2

    Article  Google Scholar 

  36. Sommestad T, Hallberg J (2012) Cyber security exercises and competitions as a platform for cyber security experiments. In: Jøsang A, Carlsson B (eds) Proceedings of the 17th Nordic conference on secure IT systems. Springer, Berlin, pp 47–60. doi:10.1007/978-3-642-34210-3_4

    Google Scholar 

  37. Stake RE (1995) The art of case study research. Sage, Thousand Oaks

    Google Scholar 

  38. Thorstensson M (2012) Supporting observers in the field to perform model based data collection. In: Rothkrantz L, Ristvej J, Franco Z (eds) P ISCRAM 9. Simon Fraser University, Vancouver, Canada

    Google Scholar 

  39. Tyworth M, Giacobe NA, Mancuso V, Dancy C (2012) The distributed nature of cyber situation awareness. In: P CogSIMA 2. IEEE, New Orleans, pp 174–178. doi:10.1109/CogSIMA.2012.6188375

    Google Scholar 

  40. Wildman JL, Salas E, Scott CPR (2013) Measuring cognition in teams: a cross-domain review. Human Factors 56:911–941. doi:10.1177/0018720813515907

    Article  Google Scholar 

  41. Yin RK (2009) Case study research: design and methods, 4th edn. Sage, Thousand Oaks

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Dennis Andersson.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Granåsen, M., Andersson, D. Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study. Cogn Tech Work 18, 121–143 (2016). https://doi.org/10.1007/s10111-015-0350-2

Download citation

Keywords

  • Cyber-defense exercise
  • Cyber SA
  • Performance assessment
  • Team cognition
  • Team effectiveness