Advertisement

Cognition, Technology & Work

, Volume 9, Issue 2, pp 99–108 | Cite as

Time windows-based team performance measures: a framework to measure team performance in dynamic environments

  • Hari Thiruvengada
  • Ling Rothrock
Original Article

Abstract

In this paper, we present a framework for implementing Team Performance Measures based on a temporal accuracy measure called the relative accuracy index (RAI) to evaluate and compare team performance in a command-and-control human-in-the-loop simulation. The framework allows researchers to collect and analyze team outcomes in an unbiased objective manner based on a temporal performance measure known as a Time Window. Our framework provides experimenters and subject matter experts the necessary tools to evaluate performance in terms of task demands. We also provide a sample analysis of individual and team performance using the RAI.

Keywords

Command and control Inter-rater bias Simulation Team performance measurement framework Teamwork dimensions Time window 

Notes

Acknowledgments

This research was supported by the Office of Naval Research under Contract No. N61339-05–C–0031. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the Office of Naval Research. We wish to thank Gwendolyn Campbell, Amy Bolton, Wendi Vanbuskirk and Damodar Bhandarkar for their valuable insights, which enabled the successful development of our framework.

References

  1. Bhandarkar D, Thiruvengada H, Rothrock L, Campbell GE, Bolton AE (2004) TASP: a toolkit to analyze team performance in a complex task environment. In: Paper presented at the annual conference of the Institute of Industrial Engineers, HoustonGoogle Scholar
  2. Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison-Wesley, ReadingGoogle Scholar
  3. Bolton AE, Campbell GE, Buff WL, Rothrock L, Bhandarkar D, Thiruvengada H, Kukreja U (2004) TASP: A flexible alternative for team training and performance research. In: Paper presented at the interservice/industry training, simulation and education conference (I/ITSEC), OrlandoGoogle Scholar
  4. Brannick M, Prince C (1997) An overview of team performance measurement. In: Brannick MT, Salas E, Prince C (eds). Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, Mahwah, pp 3–16Google Scholar
  5. Brannick MT, Salas E, Prince C (eds) (1997) Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, MahwahGoogle Scholar
  6. Brehmer B (1990) Strategies in real-time dynamic decision making. In: Hogarth RM (ed) Insights in decision making: a tribute to Hillel J. Einhorn. University of Chicago Press, Chicago, pp 262–279Google Scholar
  7. Brehmer B, Allard R (1991) Dynamic decision making: the effects of task complexity and feedback delays. In: Rasmussen J, Brehmer B, Leplat J (eds). Distributed decision making: cognitive models for cooperative work. Wiley, New York, pp 319–334Google Scholar
  8. Briggs GE, Johnston WA (1967) Team training (NTDC technical report No. 1327–4). Naval Training Device Center, OrlandoGoogle Scholar
  9. Cannon-Bowers JA, Salas E (1997a) Teamwork competencies: the intersection of team member knowledge, skills, and attitudes. In: O’Neil HF (ed) Workforce readiness: competencies and assessment. Lawrence Erlbaum, Hillsdale, pp 151–174Google Scholar
  10. Cannon-Bowers JA, Salas E (1997b) A framework for developing team performance measures in training. In: Brannick MT, Salas E, Prince C (eds). Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, Mahwah, pp 45–62Google Scholar
  11. Cannon-Bowers JA, Tannenbaum SI, Salas E, Volpe CE (1995) Defining competencies and establishing team training requirements. In: Guzzo RA, Salas E (eds). Team effectiveness and decision making in organizations. Jossey-Bass publishers, San Francisco, pp 333–380Google Scholar
  12. Dickinson TL, McIntyre RM (1998) A conceptual framework for teamwork measurement. In: Brannick MT, Salas E, Prince C (eds). Team performance measurement and assessment: theory, methods and applications. Lawrence Erlbaum, Mahwah, pp 19–43Google Scholar
  13. Green DM, Swets JA (1988) Signal detection theory and psychophysics. Peninsula publishing, Los AltosGoogle Scholar
  14. Hobson CJ, Mendel RW, Gibson FW (1981) Clarifying performance appraisal criteria. Organ Behav Hum Perform 28(2):164–188CrossRefGoogle Scholar
  15. Kirlik A, Fisk AD, Walker N, Rothrock L (1998) Feedback augmentation and part-task practice in training dynamic decision-making skills. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: Implications for individual and team training. American Psychological Association, Washington, pp 91–113CrossRefGoogle Scholar
  16. McEvoy G, Beatty R, Bernardin J (1987) Unanswered questions in assessment center research. J Bus Psychol 2:97–111CrossRefGoogle Scholar
  17. McIntyre RM, Salas E (1995) Measuring and managing for team performance: emerging principles from complex environments. In: Guzzo RA, Salas E (eds). Team effectiveness and decision making in organizations. Jossey-Bass publishers, San Francisco, pp 149–203Google Scholar
  18. Mosier KL (1997) Myths of expert decision making and automated decision making aids. In: Zsambok C, Klein G (eds). Naturalistic Decision Making. Lawrence Erlbaum, Hillsdale, pp 319–330Google Scholar
  19. Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81CrossRefGoogle Scholar
  20. Rothrock L (2001) Using time windows to evaluate operator performance. Int J Cogn Ergon, 5(1):1–21CrossRefGoogle Scholar
  21. Rothrock L, Harvey C, Burns J (2005) A theoretical framework and quantitative architecture to assess team task complexity in dynamic environments. Theoret Issue Ergon Sci 6(2):157–171CrossRefGoogle Scholar
  22. Sackett PR, Dreher GF (1982) Constructs and assessment center dimensions: Some troubling empirical findings. J Appl Psychol 67:401–410CrossRefGoogle Scholar
  23. Smith-Jentsch KA, Johnston JH, Payne SC (1998a) Measuring team-related expertise in complex environments. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: implications for individual and team training. American Psychological Association, Washington, pp 61–87CrossRefGoogle Scholar
  24. Smith-Jentsch KA, Zeisig RL, Acton B, McPherson JA (1998b) Team dimensional training: a strategy for guided team self-correction. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: implications for individual and team training. American Psychological Association, Washington, pp 271–297CrossRefGoogle Scholar
  25. Swets JA (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum, MahwahMATHGoogle Scholar
  26. Thiruvengada H, Bhandarkar DN, Kukreja U, Rothrock L (2004) TASP: a toolkit for human experimentation in a synthetic environment with a hybrid human-agent team. In: Paper presented at behavior research in modeling and simulation (BRIMS) Conference, Crystal CityGoogle Scholar
  27. Thiruvengada H, Bhandarkar DN, Rothrock L (2006) Time windows-based team performance measures: design and implementation. In: Paper to be presented at 2006 IEEE international conference on systems, man, and cybernetics, TaipeiGoogle Scholar
  28. Thiruvengada H, Rothrock L (2006) Team training. In: Nembhard D (ed) Workforce cross training handbook. CRC, Boca Raton (in press)Google Scholar

Copyright information

© Springer-Verlag London Limited 2006

Authors and Affiliations

  1. 1.Department of Industrial and Manufacturing EngineeringPennsylvania State UniversityUniversity ParkUSA

Personalised recommendations