Skip to main content
SpringerLink
Log in

Performance Measurement and Evaluation in Human-in-the-Loop Simulations

  • Chapter
  • First Online:
Human-in-the-Loop Simulations

Abstract

A prerequisite for designers of the complex systems is a proper understanding of human performance characteristics. While human factor texts provide some insights into basic performance issues, the emergence of highly automated computing systems have fundamentally altered the way humans work. The purpose of this paper is to present an approach to quantify and analyze human performance in human-in-the-loop simulations based on over ten years of research experience. The approach is centered on a measurement construct, called a time window, which enables a functional relationship between constraints on operator activities and time availability. A blackboard model is presented as the mechanism to generate, maintain, and complete time windows. To demonstrate the utility of time windows, an existing implementation in a real-time human-in-the-loop simulation is also described. An extension of time windows to measure team performance is also discussed. Using time window outcomes, samples of previous analyses are presented to exhibit the potential of the construct.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Adeli H, Yu G (1995) An integrated computing environment for solution of complex engineering problems using the object-oriented programming paradigm and a blackboard architecture. Comput Struct 54(2):255–265

    Article  Google Scholar 

  • Aksin O, Armony M, Mehrotra V (2007) The modern call-center: a multi-disciplinary perspective on operations management research. Prod Operat Manag 16(6):655–688

    Google Scholar 

  • Allen JF (1983) Maintaining Knowledge about Temporal Intervals. Commun ACM 26(11):832–843

    Article  MATH  Google Scholar 

  • Anton A (1997) Call center management: by the numbers. Purdue University Press, West Lafayette

    Google Scholar 

  • Atlason J, Epelman MA, Henderson SG (2004) Call center staffing with simulation and cutting plane methods. Ann Oper Res 127:333–358

    Article  MathSciNet  MATH  Google Scholar 

  • Avramidis A, Chan W, L’Ecuyer P (2009) Staffing multi-skill call centers via search methods and a performance approximation. IIE Trans 41(6):483–497

    Article  Google Scholar 

  • Beach LR, Lipshitz R (1993) Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok C (eds) Decision making in action: models and methods. Ablex Publishing Corp, Norwood, pp 21–35

    Google Scholar 

  • Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison-Wesley, Reading

    Google Scholar 

  • Caldeira JD (1980) Parametric assumptions of some “nonparametric” measures of sensory efficiency. Hum Factors 22(1):119–120

    Google Scholar 

  • Cannon-Bowers JA, Salas E (1997) A framework for developing team performance measures in training. In: Brannick M, Salas E, Prince C (eds) Team performance assessment and measurement: theory, methods, and applications. Erlbaum, Hillsdale, pp 45–62

    Google Scholar 

  • Cattell RB (1966) The meaning and strategic use of factor analysis. In: Cattell RB (ed) Handbook of multivariate experimental psychology. Rand McNally, Chicago

    Google Scholar 

  • Cezik MT, L’Ecuyer P (2008) Staffing Multiskill Call Centers via Linear Programming and Simulation. Manag Sci 54(2):310–323

    Article  Google Scholar 

  • Chesbrough H, Spohrer J (2006) A research manifesto for services science. Commun ACM 49(7):35–40

    Article  Google Scholar 

  • Chu RW, Jones PM, Mitchell CM (1991) The Georgia tech payload operations control center simulation: design and implementation in C++. In: Proceedings of the object-oriented simulation track of the 1991 society for computer simulation multiconference, Anaheim, CA

    Google Scholar 

  • Coovert MD, Cannon-Bowers JA, Salas E (1990) Applying mathematical modeling technology to the study of team training and performance. Paper presented at the 12th annual interservice/industry training systems conference

    Google Scholar 

  • Craig A (1979) Nonparametric measures of sensory efficiency for sustained monitoring tasks. Hum Factors 21(1):69–78

    MathSciNet  Google Scholar 

  • Endsley MR (2006) Situation awareness. In: Salvendy G (ed) Handbook of human factors and ergonomics, 3rd edn. John Wiley and Sons, Inc, New York, pp 528–542

    Chapter  Google Scholar 

  • Erman DL, Hayes-Roth F, Lesser VR, Reddy D (1980) The HEARSAY-II speech understanding system: integrating knowledge to resolve uncertainty. ACM Comput Surv 12:213–253

    Article  Google Scholar 

  • Fleming JH, Coffman C, Harter JK (2005) Manage your human sigma. Harv Bus Rev 83(7/8):106–114

    Google Scholar 

  • Gabbay DM, Hodkinson I, Reynolds M (1994) Temporal logic: mathematical foundations and computational aspects. Oxford University Press, Oxford

    MATH  Google Scholar 

  • Gans N, Koole G, Mandelbaum A (2003) Telephone call centers: Tutorial, review, and research prospects. Manuf Serv Oper Manag 5(2):79–142

    Article  Google Scholar 

  • Garnet O, Mandelbaum A, Reiman M (2002) Designing a call center with impatient customers. Manuf Serv Oper Manag 4(3):208–227

    Article  Google Scholar 

  • Gilmore A, Moreland L (2000) Call centres: how can service quality be managed? Ir Market Rev 13(1):3–11

    Google Scholar 

  • Green PE (1978) Analyzing multivariate data. Dryden Press, Hinsdale

    Google Scholar 

  • Green DM, Swets JA (1966) Signal detection theory and psychophysics. Wiley, New York

    Google Scholar 

  • Greeno JG (1994) Gibson’s affordances. Psychol Rev 101(2):336–342

    Article  Google Scholar 

  • Greeno JG (1998) The situativity of knowing, learning, and research. Am Psychol 53(1):5–26

    Article  Google Scholar 

  • Greeno JG, Moore JL (1993) Situativity and symbols: response to Vera and Simon. Cogn Sci 17:49–60

    Article  Google Scholar 

  • Hammond KR (1986) Generalization in operational contexts: what does it mean? can it be done? IEEE Trans Syst Man Cybern, SMC 16(3):428–433

    Article  MathSciNet  Google Scholar 

  • Howie DE, Vicente KJ (1998) Measures of operator performance in complex, dynamic microworlds: advancing the state of the art. Ergonomics 41(4):485–500

    Article  Google Scholar 

  • Jones PM, Chu RW, Mitchell CM (1995) A methodology for human-machine systems research: knowledge engineering, modeling, and simulation. IEEE Trans Syst Man Cybern 25(7)

    Google Scholar 

  • Kirlik A (1995) Requirements for psychological models to support design: toward ecological task analysis. In: Flach JM, Hancock P, Caird J, Vicente KJ (eds) Global perspectives on the ecology of human-machine systems. Erlbaum, Hillsdale, pp 68–120

    Google Scholar 

  • Laudeman IV, Palmer EA (1995) The quantitative measurement of observed workload in the analysis of aircrew performance. Int J Aviat Psychol 5(2):187–197

    Article  Google Scholar 

  • Ma J, Kim N, Rothrock L (2011) Performance assessment in an interactive call center workforce simulation. Simul Model Pract Theory 19:227–238

    Article  Google Scholar 

  • Marks MA, Sabella MJ, Burke CS, Zaccaro SJ (2002) The impact of cross-training on team effectiveness. J Appl Psychol 87(1):3–13

    Article  Google Scholar 

  • Mehrotra V, Fama J (2003) Call center simulation modeling: methods, challenges, and opportunities. Paper presented at the 2003 winter simulation conference

    Google Scholar 

  • Militello LG, Hutton RJB (1998) Applied cognitive task analysis (acta): a practitioner’s toolkit for understanding task demands. Ergonomics 41(11):1618–1641

    Article  Google Scholar 

  • Moray N, Dessouky MI, Kijowski BA, Adapathya R (1991) Strategic behavior, workload, and performance in task scheduling. Hum Factors 33(6):607–629

    Google Scholar 

  • Nii HP (1986) Blackboard systems: the blackboard model of problem solving and the evolution of blackboard architectures. Part I. AI Mag 7:38–53

    Google Scholar 

  • Raby M, Wickens CD (1994) Strategic workload management and decision biases in aviation. Int J Aviat Psychol 4(3):211–240

    Article  Google Scholar 

  • Rothrock L (2001) Using time windows to evaluate operator performance. Int J Cogn Ergonomics 5(1):1–21

    Article  Google Scholar 

  • Rothrock L, Cohen A, Yin J, Thiruvengada H, Nahum-Shani I (2009) Analyses of team performance in a dynamic task environment. Appl Ergonomics 40(4):699–706

    Article  Google Scholar 

  • Rubin KS, Jones PM, Mitchell CM (1988) OFMspert: inference of operator intentions in supervisory control using a blackboard architecture. IEEE Trans Syst Man Cybern SMC 18(4):618–637

    Article  Google Scholar 

  • Sanderson PM, Verhage AG, Fuld RB (1989) State-space and verbal protocol methods for studying the human operator in process control. Ergonomics 32:1343–1372

    Article  Google Scholar 

  • Sharp D (2003) Call center operation: design. Digital Science Operation, and Maintenance, New York

    Google Scholar 

  • Smith-Jentsch KA, Johnston JH, Payne SC (1998) Measuring team-related expertise in complex environments. In: Cannon-Bowers JA, Salas E (eds) Making decisions under stress: implications for individual and team training. APA, Washington, D.C., pp. 61–87

    Chapter  Google Scholar 

  • Smith-Jentsch KA, Zeisig RL, Acton B, McPherson JA (1998) Team dimensional training: a strategy for guided team self-correction. In: Cannon-Bowers JA, Salas E (eds) Making decisions under stress: implications for individual and team training. APA, Washington, D.C., pp 271–297

    Chapter  Google Scholar 

  • Suchman LA (1987) Plans and situated actions: the problem of human machine communication. Cambridge University Press, Cambridge

    Google Scholar 

  • Swets JA (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum Assoc, Mahwah

    MATH  Google Scholar 

  • Tanir O, Booth RJ (1999) Call center simulation in Bell Canada. Paper presented at the 1999 winter simulation conference. Phoenix, AZ

    Google Scholar 

  • Thiruvengada H, Rothrock L (2007) Time window-based performance measures: a framework to measure team performance in dynamic environments. Cogn Technol Work 9(2):99–108

    Article  Google Scholar 

  • Vicente KJ, Rasmussen JR (1992) Ecological interface design: theoretical foundations. IEEE Trans Syst, Man Cybern 22(4):589–606

    Article  Google Scholar 

  • Vranes S, Lucin M, Stanojevic M, Stevanovic V, Subasic P (1991) Blackboard metaphor in tactical decision making. Eur J Oper Res 61:86–97

    Article  Google Scholar 

  • Wallace CM, Eagleson G, Waldersee R (2000) The sacrificial HR strategy in call centres. Int J Serv Ind Manag 11(2):174–184

    Article  Google Scholar 

  • Wickens CD, Hollands JG (2000) Engineering psychology and human performance, 3rd edn. Prentice Hall, Upper Saddle River

    Google Scholar 

  • Wickens CD, Kessel C (1979) The effects of participatory mode and task workload on the detection of dynamic system failures. IEEE Trans Syst Man Cybern 9(1):24–34

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Harold and Inge Marcus Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, Leonhard Building 210, State College, University Park, PA, 16802, USA

    Ling Rothrock

Authors
  1. Ling Rothrock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ling Rothrock .

Editor information

Editors and Affiliations

  1. , The Harold and Inge Marcus Department of, The Pennsylvania State University, Leonhard Building 210, State College, 16802, Pennsylvania, USA

    Ling Rothrock

  2. , 405 Russ Engineering Center, Wright State University, Col. Glenn Hwy 3640, Dayton, 45435, Ohio, USA

    S. Narayanan

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag London Limited

About this chapter

Cite this chapter

Rothrock, L. (2011). Performance Measurement and Evaluation in Human-in-the-Loop Simulations. In: Rothrock, L., Narayanan, S. (eds) Human-in-the-Loop Simulations. Springer, London. https://doi.org/10.1007/978-0-85729-883-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-0-85729-883-6_2

  • Published:

  • Publisher Name: Springer, London

  • Print ISBN: 978-0-85729-882-9

  • Online ISBN: 978-0-85729-883-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation