Skip to main content
SpringerLink
Log in

Assessing Cognitive Readiness in a Simulation-Based Training Environment

  • Chapter
  • First Online:
Teaching and Measuring Cognitive Readiness

Abstract

This chapter describes progress toward developing and validating an assessment of cognitive readiness for Tactical Action Officer performance in a simulated Combat Information Center. In this chapter we describe the simulation environment and scenarios, the constructs defined to measure cognitive readiness, an approach to measurement, and a computer-based assessment tool that operationalizes the measures and supports data collection and reporting. The preliminary results provide evidence that the measures are meaningful indicators of the constructs, but the confirmatory factor analysis and convergent validity results were mixed, depending on the nature of the scenario. The chapter closes with a discussion of the implications for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Achille, L. B., Schulze, K. G., & Schmidt-Nielsen, A. (1995). An analysis of communication and the use of military terms in Navy team training. Military Psychology: Special Issue: Team Processes, Training, and Performance, 7, 95–107.

    Google Scholar 

  • Bentler, P. M. (2007). EQS 6.1 for windows [computer software]. Encino, CA: Multivariate Software.

    Google Scholar 

  • Blandford, A. (2004). Situation awareness in emergency medical dispatch. International Journal of Human Computer Studies, 61(4), 421–452.

    Article  Google Scholar 

  • Bruce, J. B. (2008). Making analysis more reliable: Why epistemology matters to intelligence. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 171–190). Washington, DC: Georgetown University Press.

    Google Scholar 

  • Bruce, J. B., & Bennett, M. (2008). Foreign denial and deception: Analytical imperatives. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 122–137). Washington, DC: Georgetown University Press.

    Google Scholar 

  • Cannon-Bowers, J. A., & Salas, E. (Eds.). (1998). Making decisions under stress: Implications for individual and team training. Washington, DC: American Psychological Association.

    Google Scholar 

  • Choi, K. C., Kang, T., & Delacruz, G. C. (2009, July). Intersection of psychometrics with games. Presentation at the Psychometrics of Simulation Conference, Redondo Beach, CA.

    Google Scholar 

  • CIA Directorate of Intelligence. (1997). A compendium of analytic tradecraft notes: Volume 1, notes 3 and 4. Retrieved from http://www.au.af.mil/au/awc/awcgate/cia/tradecraft_notes/contents.htm#contents

  • Davis, J. (2008). Why bad things happen to good analysts. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 157–170). Washington, DC: Georgetown University Press.

    Google Scholar 

  • Durso, F. T., & Sethumadhavan, A. (2008). Situation awareness: Understanding dynamic environments. Human Factors, 50(3), 442–448.

    Article  Google Scholar 

  • Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society 32nd Annual Meeting (Vol. 1, pp. 97–101). Santa Monica, CA: Human Factors Society.

    Google Scholar 

  • Endsley, M. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.

    Article  Google Scholar 

  • Endsley, M. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.

    Article  Google Scholar 

  • Endsley, M. R. (1997). The role of situation awareness in naturalistic decision making. In C. E. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 269–283). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement (pp. 3–32). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Endsley, M. R. (2004). Situation awareness: Progress and directions. In S. Banbury & S. Tremblay (Eds.), A cognitive approach to situation awareness: Theory and application (pp. 317–341). Aldershot: Ashgate Publishing.

    Google Scholar 

  • Fletcher, J. D. (2004). Cognitive readiness: Preparing for the unexpected (IDA Document D-3061). Alexandria, VA: Institute for Defense Analyses.

    Google Scholar 

  • Gaba, D. M., Howard, S. K., & Small, S. D. (1995). Situation awareness in anesthesiology. Human Factors, 37, 20–31.

    Article  Google Scholar 

  • George, R. Z. (2004). Fixing the problem of analytical mind-sets: Alternative analysis. International Journal of Intelligence and Counterintelligence, 17, 385–404.

    Article  Google Scholar 

  • Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation awareness in decentralized command and control environments. Ergonomics, 49, 1312–1325.

    Article  Google Scholar 

  • Graham, S. E., & Matthews, M. D. (2000). Modeling and measuring situation awareness. In J. H. Hiller & R. L. Wampler (Eds.), Workshop on assessing and measuring training performance effectiveness (Tech Report 1116) (pp. 14–24). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

    Google Scholar 

  • Groopman, J. (2007). How doctors think. Boston: Houghton Mifflin.

    Google Scholar 

  • Hair, J. F. J., Anderson, R. E., Tatham, R. L., & Black, W. C. (1995). Multivariate data analysis (4th ed.). Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Harwell, M. R., & Gatti, G. G. (2001). Rescaling ordinal data to interval data in educational research. Review of Educational Research, 71(1), 105–131.

    Article  Google Scholar 

  • Heuer, R. J. (1999). Psychology of intelligence analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.

    Google Scholar 

  • Jones, D. G., & Endsley, M. R. (2004). Use of real-time probes for measuring situation awareness. The International Journal of Aviation Psychology, 14(4), 343–367.

    Article  Google Scholar 

  • Kerbel, J. (2004). Thinking straight: Cognitive bias in the US debate about China. Studies in Intelligence, 48(3), 385–404. Retrieved from http://www.au.af.mil/au/awc/awcgate/cia/think_cog_bias.htm

  • Klein, G. (1999). Sources of power: How people make decisions. Cambridge, MA: MIT Press.

    Google Scholar 

  • Klein, G. (2008). Naturalistic decision making. Human Factors, 50(3), 456–460.

    Article  Google Scholar 

  • Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70–73.

    Article  Google Scholar 

  • Leedom, D. K. (2001). Final report: Sensemaking symposium. Vienna, VA: Command and Control Research Program, Office of the Assistant Secretary of Defense for Command, Control, Communications and Intelligence.

    Google Scholar 

  • Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

    Google Scholar 

  • McCarthy, G. (2006). Situation awareness in medicine. Quality & Safety in Health Care, 15, 384.

    Article  Google Scholar 

  • Morrison, J. E., & Fletcher, J. D. (2001). Cognitive readiness (IDA Paper P-3735). Alexandria, VA: Institute for Defense Analyses.

    Google Scholar 

  • Morrison, J. G., Marshall, S. P., Kelly, R. T., & Moore, R. A. (1997). Eye tracking in tactical decision-making environments: Implications for decision support. In Proceedings of the 1997 Command and Control Research and Technology Symposium (pp. 355–364). Washington, DC: National Defense University.

    Google Scholar 

  • Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach (studentth ed.). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org

  • Radtke, P., Johnston, J. H., Biddle, E., & Carolan, T. F. (2007, December). Integrating and presenting performance information in simulation-based air warfare scenarios. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference, Orlando, FL.

    Google Scholar 

  • Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Smith-Jentsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-related expertise in complex environments. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress (pp. 61–87). Washington, DC: American Psychological Association.

    Google Scholar 

  • Tenney, Y. J., & Pew, R. W. (2006). Situation awareness catches on: What? So what? Now what? In R. C. Williges (Ed.), Reviews of human factors and ergonomics (Vol. 2, pp. 1–34). Santa Monica, CA: Human Factors and Ergonomics Society.

    Google Scholar 

  • U.S. Navy, Surface Warfare Officers School Command. (2012). Multi-Mission Tactical Trainer (MMTT), Tactical Action Officer (TAO) Intelligent Tutoring System (ITS). Retrieved from https://www.netc.navy.mil/centers/swos/Simulators.aspx?ID=2

  • Weick, K. E. (1995). Sensemaking in organizations. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Weick, W. R., Klein, G., Peluso, D. A., Smith, J. L., & Harris-Thompson, D. (2007). FOCUS: A model of sensemaking (Technical Report 1200). Arlington, VA: United States Army Research Institute for the Behavioral and Social Sciences.

    Google Scholar 

  • West, P., Rutstein, D. W., Mislevy, R. J., Liu, J., Levy, R., DiCerbo, K. E., et al. (2009, June). A Bayes net approach to modeling learning progressions and task performances. Paper presented at the Learning Progressions in Science (LeaPS) Conference, Iowa City, IA.

    Google Scholar 

  • Wickens, C. (2008). Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Human Factors, 50(3), 397–403.

    Article  Google Scholar 

Download references

Acknowledgments

The work reported herein was supported by grant number N00014-06-1-0711 from the U.S. Navy, Office of Naval Research with funding to the National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of the U.S. Navy, Office of Naval Research.

Author information

Authors and Affiliations

  1. National Center for Research on Evaluation, Standards, and Student Testing (CRESST)/UCLA, Los Angeles, CA, USA

    William L. Bewley, John J. Lee & Barbara Jones

  2. Guangdong University of Foreign Studies, Guangzhou, People’s Republic of China

    Hongwen Cai

Authors
  1. William L. Bewley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John J. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Barbara Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongwen Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William L. Bewley .

Editor information

Editors and Affiliations

  1. , National Center for Research on Evaluati, University of Southern California, Longbow Drive 15366, Los Angeles, 91403, California, USA

    Harold F. O'Neil

  2. Office of Naval Research, Inglewood Ct. 2431, Arlington, 22043, Virgin Islands, USA

    Ray S. Perez

  3. National Center for Research on Evaluati, Standards, and Student Testing, CRESST/UCLA, Charles E. Young Drive North 301, Los Angeles, 90095-1521, California, USA

    Eva L. Baker

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this chapter

Cite this chapter

Bewley, W.L., Lee, J.J., Jones, B., Cai, H. (2014). Assessing Cognitive Readiness in a Simulation-Based Training Environment. In: O'Neil, H., Perez, R., Baker, E. (eds) Teaching and Measuring Cognitive Readiness. Springer, Boston, MA. https://doi.org/10.1007/978-1-4614-7579-8_14

Download citation

Publish with us

Policies and ethics

Search

Navigation