Assessing Cognitive Readiness in a Simulation-Based Training Environment

  • William L. Bewley
  • John J. Lee
  • Barbara Jones
  • Hongwen Cai
Chapter

Abstract

This chapter describes progress toward developing and validating an assessment of cognitive readiness for Tactical Action Officer performance in a simulated Combat Information Center. In this chapter we describe the simulation environment and scenarios, the constructs defined to measure cognitive readiness, an approach to measurement, and a computer-based assessment tool that operationalizes the measures and supports data collection and reporting. The preliminary results provide evidence that the measures are meaningful indicators of the constructs, but the confirmatory factor analysis and convergent validity results were mixed, depending on the nature of the scenario. The chapter closes with a discussion of the implications for future research.

Keywords

Radar Kelly Defend 

Notes

Acknowledgments

The work reported herein was supported by grant number N00014-06-1-0711 from the U.S. Navy, Office of Naval Research with funding to the National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of the U.S. Navy, Office of Naval Research.

References

  1. Achille, L. B., Schulze, K. G., & Schmidt-Nielsen, A. (1995). An analysis of communication and the use of military terms in Navy team training. Military Psychology: Special Issue: Team Processes, Training, and Performance, 7, 95–107.Google Scholar
  2. Bentler, P. M. (2007). EQS 6.1 for windows [computer software]. Encino, CA: Multivariate Software.Google Scholar
  3. Blandford, A. (2004). Situation awareness in emergency medical dispatch. International Journal of Human Computer Studies, 61(4), 421–452.CrossRefGoogle Scholar
  4. Bruce, J. B. (2008). Making analysis more reliable: Why epistemology matters to intelligence. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 171–190). Washington, DC: Georgetown University Press.Google Scholar
  5. Bruce, J. B., & Bennett, M. (2008). Foreign denial and deception: Analytical imperatives. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 122–137). Washington, DC: Georgetown University Press.Google Scholar
  6. Cannon-Bowers, J. A., & Salas, E. (Eds.). (1998). Making decisions under stress: Implications for individual and team training. Washington, DC: American Psychological Association.Google Scholar
  7. Choi, K. C., Kang, T., & Delacruz, G. C. (2009, July). Intersection of psychometrics with games. Presentation at the Psychometrics of Simulation Conference, Redondo Beach, CA.Google Scholar
  8. CIA Directorate of Intelligence. (1997). A compendium of analytic tradecraft notes: Volume 1, notes 3 and 4. Retrieved from http://www.au.af.mil/au/awc/awcgate/cia/tradecraft_notes/contents.htm#contents
  9. Davis, J. (2008). Why bad things happen to good analysts. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 157–170). Washington, DC: Georgetown University Press.Google Scholar
  10. Durso, F. T., & Sethumadhavan, A. (2008). Situation awareness: Understanding dynamic environments. Human Factors, 50(3), 442–448.CrossRefGoogle Scholar
  11. Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.Google Scholar
  12. Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society 32nd Annual Meeting (Vol. 1, pp. 97–101). Santa Monica, CA: Human Factors Society.Google Scholar
  13. Endsley, M. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.CrossRefGoogle Scholar
  14. Endsley, M. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.CrossRefGoogle Scholar
  15. Endsley, M. R. (1997). The role of situation awareness in naturalistic decision making. In C. E. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 269–283). Mahwah, NJ: Erlbaum.Google Scholar
  16. Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement (pp. 3–32). Mahwah, NJ: Erlbaum.Google Scholar
  17. Endsley, M. R. (2004). Situation awareness: Progress and directions. In S. Banbury & S. Tremblay (Eds.), A cognitive approach to situation awareness: Theory and application (pp. 317–341). Aldershot: Ashgate Publishing.Google Scholar
  18. Fletcher, J. D. (2004). Cognitive readiness: Preparing for the unexpected (IDA Document D-3061). Alexandria, VA: Institute for Defense Analyses.Google Scholar
  19. Gaba, D. M., Howard, S. K., & Small, S. D. (1995). Situation awareness in anesthesiology. Human Factors, 37, 20–31.CrossRefGoogle Scholar
  20. George, R. Z. (2004). Fixing the problem of analytical mind-sets: Alternative analysis. International Journal of Intelligence and Counterintelligence, 17, 385–404.CrossRefGoogle Scholar
  21. Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation awareness in decentralized command and control environments. Ergonomics, 49, 1312–1325.CrossRefGoogle Scholar
  22. Graham, S. E., & Matthews, M. D. (2000). Modeling and measuring situation awareness. In J. H. Hiller & R. L. Wampler (Eds.), Workshop on assessing and measuring training performance effectiveness (Tech Report 1116) (pp. 14–24). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.Google Scholar
  23. Groopman, J. (2007). How doctors think. Boston: Houghton Mifflin.Google Scholar
  24. Hair, J. F. J., Anderson, R. E., Tatham, R. L., & Black, W. C. (1995). Multivariate data analysis (4th ed.). Upper Saddle River, NJ: Prentice Hall.Google Scholar
  25. Harwell, M. R., & Gatti, G. G. (2001). Rescaling ordinal data to interval data in educational research. Review of Educational Research, 71(1), 105–131.CrossRefGoogle Scholar
  26. Heuer, R. J. (1999). Psychology of intelligence analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.Google Scholar
  27. Jones, D. G., & Endsley, M. R. (2004). Use of real-time probes for measuring situation awareness. The International Journal of Aviation Psychology, 14(4), 343–367.CrossRefGoogle Scholar
  28. Kerbel, J. (2004). Thinking straight: Cognitive bias in the US debate about China. Studies in Intelligence, 48(3), 385–404. Retrieved from http://www.au.af.mil/au/awc/awcgate/cia/think_cog_bias.htm
  29. Klein, G. (1999). Sources of power: How people make decisions. Cambridge, MA: MIT Press.Google Scholar
  30. Klein, G. (2008). Naturalistic decision making. Human Factors, 50(3), 456–460.CrossRefGoogle Scholar
  31. Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70–73.CrossRefGoogle Scholar
  32. Leedom, D. K. (2001). Final report: Sensemaking symposium. Vienna, VA: Command and Control Research Program, Office of the Assistant Secretary of Defense for Command, Control, Communications and Intelligence.Google Scholar
  33. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
  34. McCarthy, G. (2006). Situation awareness in medicine. Quality & Safety in Health Care, 15, 384.CrossRefGoogle Scholar
  35. Morrison, J. E., & Fletcher, J. D. (2001). Cognitive readiness (IDA Paper P-3735). Alexandria, VA: Institute for Defense Analyses.Google Scholar
  36. Morrison, J. G., Marshall, S. P., Kelly, R. T., & Moore, R. A. (1997). Eye tracking in tactical decision-making environments: Implications for decision support. In Proceedings of the 1997 Command and Control Research and Technology Symposium (pp. 355–364). Washington, DC: National Defense University.Google Scholar
  37. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach (studentth ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  38. R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
  39. Radtke, P., Johnston, J. H., Biddle, E., & Carolan, T. F. (2007, December). Integrating and presenting performance information in simulation-based air warfare scenarios. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference, Orlando, FL.Google Scholar
  40. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). Thousand Oaks, CA: Sage.Google Scholar
  41. Smith-Jentsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-related expertise in complex environments. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress (pp. 61–87). Washington, DC: American Psychological Association.Google Scholar
  42. Tenney, Y. J., & Pew, R. W. (2006). Situation awareness catches on: What? So what? Now what? In R. C. Williges (Ed.), Reviews of human factors and ergonomics (Vol. 2, pp. 1–34). Santa Monica, CA: Human Factors and Ergonomics Society.Google Scholar
  43. U.S. Navy, Surface Warfare Officers School Command. (2012). Multi-Mission Tactical Trainer (MMTT), Tactical Action Officer (TAO) Intelligent Tutoring System (ITS). Retrieved from https://www.netc.navy.mil/centers/swos/Simulators.aspx?ID=2
  44. Weick, K. E. (1995). Sensemaking in organizations. Thousand Oaks, CA: Sage.Google Scholar
  45. Weick, W. R., Klein, G., Peluso, D. A., Smith, J. L., & Harris-Thompson, D. (2007). FOCUS: A model of sensemaking (Technical Report 1200). Arlington, VA: United States Army Research Institute for the Behavioral and Social Sciences.Google Scholar
  46. West, P., Rutstein, D. W., Mislevy, R. J., Liu, J., Levy, R., DiCerbo, K. E., et al. (2009, June). A Bayes net approach to modeling learning progressions and task performances. Paper presented at the Learning Progressions in Science (LeaPS) Conference, Iowa City, IA.Google Scholar
  47. Wickens, C. (2008). Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Human Factors, 50(3), 397–403.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • William L. Bewley
    • 1
  • John J. Lee
    • 1
  • Barbara Jones
    • 1
  • Hongwen Cai
    • 2
  1. 1.National Center for Research on Evaluation, Standards, and Student Testing (CRESST)/UCLALos AngelesUSA
  2. 2.Guangdong University of Foreign StudiesGuangzhouPeople’s Republic of China

Personalised recommendations