Skip to main content

Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study

Cognition, Technology & Work volume 18, pages 121–143 (2016)Cite this article

Abstract

In 2010, IT-security experts from northern European governments and organizations gathered to conduct the first of a series of NATO-led cyber-defense exercises in a pilot attempt of training cyber defense. To gain knowledge on how to assess team effectiveness in cyber-defense exercises, this case study investigates the role of behavioral assessment techniques as a complement to task-based performance measurement. The collected data resulted in a massive data set including system logs, observer reports, and surveys. Six different methods were compared for feasibility in assessing the teams’ performance, including automated availability check, exploratory sequential data analysis, and network intrusion detection system attack analysis. In addition, observer reports and surveys were used to collect aspects relating to team structures and processes, aiming to discover whether these aspects can explain differences in effectiveness. The cross-disciplinary approach and multiple metrics create possibilities to study not only the performance-related outcome of the exercise, but also why this result is obtained. The main conclusions found are (1) a combination of technical performance measurements and behavioral assessment techniques are needed to assess team effectiveness, and (2) cyber situation awareness is required not only for the defending teams, but also for the observers and the game control.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. IBM SPSS, Commercial statistical analysis software, http://www.ibm.com/software/analytics/spss.

  2. F-REX, Tools for Reconstruction and Exploration of heterogeneous datasets (Andersson 2009).

  3. Snort, Open source network intrusion detection software, http://www.snort.org.

References

  • Andersson D (2009) F-REX: event driven synchronized multimedia model visualization. In: Proceedings of the 15th international conference on distributed multimedia systems. Knowledge Systems Institute, Redwood City, pp 140–145

  • Andersson D (2011) Privacy and distributed tactical operations evaluation. In: Proceedings of the 4th international conference on advances in human-oriented and personalized mechanisms, technologies, and services. Barcelona

  • Andersson D (2013) A knowledge base for capturing comprehensive mission experience. P Ann HICCS 46. IEEE, Wailea. doi:10.1109/HICSS.2013.40

  • Andersson D (2014) An externalizable model of tactical mission control for knowledge transfer. Int J Inf Syst Crisis Response Manag 6(3):16–37. doi:10.4018/IJISCRAM.2014070102

    Article  Google Scholar 

  • Andersson D, Granåsen M, Sundmark T, Holm H, Hallberg J (2011) Analysis of a cyber defense exercise using exploratory sequential data analysis. In: Proceedings of the 16th international command and control research and technology symposium. DoD CCRP, Québec City

  • Barford P, Dacier M, Dietterich TG et al (2010) Cyber SA: situational awareness for cyber defense. In: Jajodia S, Liu P, Swarup V, Wang C (eds) Cyber situational awareness: advances in information security 46. Springer, Berlin, pp 3–13. doi:10.1007/978-1-4419-0140-8_1

    Chapter  Google Scholar 

  • Branlat M (2011) Challenges to adversarial interplay under high uncertainty: staged-world study of a cyber security event. Dissertation, Ohio State University

  • Champion MA, Rajivan P, Cooke NJ, Jariwala S (2012) Team-based cyber defense analysis. In: P CogSIMA 2. IEEE, New Orleans, pp 218–221. doi:10.1109/CogSIMA.2012.6188386

    Google Scholar 

  • Conklin A (2006) Cyber defense competitions and information security education: an active learning solution for a capstone course. P Ann HICCS 39, Kauai. doi:10.1109/HICSS.2006.110

  • Cooke NJ, Salas E, Kiekel PA, Bell B (2004) Advances in measuring team cognition. In: Salas E, Fiore SM (eds) Team cognition: understanding the factors that drive process and performance. American Psychological Association, Washington, pp 83–106

    Chapter  Google Scholar 

  • Cowger CD (1984) Statistical significance tests: scientific ritualism or scientific method? Soc Serv Rev 58:358–372

    Article  Google Scholar 

  • Cowger CD (1985) Author’s reply. Soc Serv Rev 59:520–522

    Article  Google Scholar 

  • Doupé A, Egele M, Caillat B et al (2011) Hit’em where it hurts: a live security exercise on cyber situational awareness. In: P ACSAC 27: 51–61. ACM, Orlando

  • Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. Human Factors 37:32–64. doi:10.1518/001872095779049543

    Article  Google Scholar 

  • Endsley MR (2000) Direct measurement of situation awareness: validity and use of SAGAT. In: Endsley MR, Garland DJ (eds) Situation awareness analysis and measurement. Lawrence Erlbaum, Mahwah

    Google Scholar 

  • Flyvbjerg B (2011) Case study. In: Denzin NK, Lincoln YS (eds) The Sage handbook of qualitative research, 4th edn. Sage, Thousand Oaks, pp 301–316

    Google Scholar 

  • Franke U, Brynielsson J (2014) Cyber situational awareness—a systematic review of the literature. Comput Secur 46:18–31. doi:10.1016/j.cose.2014.06.008

    Article  Google Scholar 

  • Geers K (2010) Live fire exercise: preparing for cyber war. J Homel Secur Emerg 7. doi:10.2202/1547-7355.1780

    Google Scholar 

  • Greenemeier L (2007) China’s cyber attacks signal new battlefield is online. Scientific American, New York

    Google Scholar 

  • Hammervik M, Andersson D, Hallberg J (2010) Capturing a cyber defence exercise. In: Proceedings of the first national symposium on technology and methodology for security and crisis management, Linköping, Sweden

  • Hoffman LJ, Rosenberg T, Dodge R, Ragsdale D (2005) Exploring a national cybersecurity exercise for universities. IEEE Secur Priv 3:27–33. doi:10.1109/MSP.2005.120

    Article  Google Scholar 

  • Holm H, Ekstedt M, Andersson D (2012) Empirical analysis of system-level vulnerability metrics through actual attacks. IEEE Trans Dependable Secur 9:825–837. doi:10.1109/TDSC.2012.66

    Article  Google Scholar 

  • Igure VM, Laughter SA, Williams RD (2006) Security issues in SCADA networks. Comput Secur 25:498–506. doi:10.1016/j.cose.2006.03.001

    Article  Google Scholar 

  • Lim KH, Benbasat I (2000) The effect of multimedia on perceived equivocality and perceived usefulness of information systems. MIS Q 24:449–471. doi:10.2307/3250969

    Article  Google Scholar 

  • Malek J (2005) Informed consent. In: Mitcham C (ed) Encyclopedia of science, technology and ethics, vol 2. Macmillan, Detroit, pp 1016–1019

    Google Scholar 

  • NATO (2010) Cyber defence exercise baltic cyber shield 2010: after action report. CCDCoE, Tallinn

    Google Scholar 

  • NATO (2012) Cyber defence exercise locked shields 2012: after action report. CCDCoE, Tallinn

    Google Scholar 

  • NATO (2013) Cyber defence exercise locked shields 2013: after action report. CCDCoE, Tallinn

    Google Scholar 

  • Otondo RF, van Scotter JR, Allen DG, Palvia P (2008) The complexity of richness: media, message, and communication outcomes. Inf Manag 40:21–30. doi:10.1016/j.im.2007.09.003

    Article  Google Scholar 

  • Pfleeger SL, Caputo DD (2012) Leveraging behavioral science to mitigate cyber security risk. Comput Secur 31:597–611. doi:10.1016/j.cose.2011.12.010

    Article  Google Scholar 

  • Pilemalm S, Andersson D, Hallberg N (2008) Reconstruction and exploration of large-scale distributed operations: multimedia tools for evaluation of emergency management response. J Emerg Manag 6:31–47

    Google Scholar 

  • Riegelsberger J, Sasse MA, McCarthy J (2003) The researcher’s dilemma: evaluating trust in computer-mediated communication. Int J Human Comput Stud 58:759–781. doi:10.1016/S1071-5819(03)00042-9

    Article  Google Scholar 

  • Rubin A (1985) Significance testing with population data. Soc Serv Rev 59:518–520

    Article  Google Scholar 

  • Salas E, Sims DE, Burke CS (2005) Is there a “Big Five” in teamwork? Small Group Res 36:555–599. doi:10.1177/1046496405277134

    Article  Google Scholar 

  • Sanderson PM, Fisher C (1994) Exploratory sequential data analysis: foundations. Human Comput Interact 9:251–317. doi:10.1207/s15327051hci0903&4_2

    Article  Google Scholar 

  • Sommestad T, Hallberg J (2012) Cyber security exercises and competitions as a platform for cyber security experiments. In: Jøsang A, Carlsson B (eds) Proceedings of the 17th Nordic conference on secure IT systems. Springer, Berlin, pp 47–60. doi:10.1007/978-3-642-34210-3_4

    Chapter  Google Scholar 

  • Stake RE (1995) The art of case study research. Sage, Thousand Oaks

    Google Scholar 

  • Thorstensson M (2012) Supporting observers in the field to perform model based data collection. In: Rothkrantz L, Ristvej J, Franco Z (eds) P ISCRAM 9. Simon Fraser University, Vancouver, Canada

    Google Scholar 

  • Tyworth M, Giacobe NA, Mancuso V, Dancy C (2012) The distributed nature of cyber situation awareness. In: P CogSIMA 2. IEEE, New Orleans, pp 174–178. doi:10.1109/CogSIMA.2012.6188375

    Google Scholar 

  • Wildman JL, Salas E, Scott CPR (2013) Measuring cognition in teams: a cross-domain review. Human Factors 56:911–941. doi:10.1177/0018720813515907

    Article  Google Scholar 

  • Yin RK (2009) Case study research: design and methods, 4th edn. Sage, Thousand Oaks

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division for Information- and Aeronautical Systems, Swedish Defense Research Agency, Box 1165, 581 11, Linköping, Sweden

    Magdalena Granåsen & Dennis Andersson

Authors
  1. Magdalena Granåsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dennis Andersson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dennis Andersson.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Granåsen, M., Andersson, D. Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study. Cogn Tech Work 18, 121–143 (2016). https://doi.org/10.1007/s10111-015-0350-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-015-0350-2

Keywords

  • Cyber-defense exercise
  • Cyber SA
  • Performance assessment
  • Team cognition
  • Team effectiveness