Tracing Cognition with Assessment Center Simulations: Using Technology to See in the Dark
Shifts in the global economy have placed more pressure on the decisions of employees, managers, and CEOs than ever before. At the same time, the proliferation of technology in the workplace has become ubiquitous. Fortunately, the assessment center method is evolving alongside these other trends. With just a few modifications to existing assessment center simulations, we can use newer technology to capture previously difficult to observe behaviors that tap directly into candidate’s decision making processes. By combining passive data logging of key strokes and mouse clicks, eye tracking, and physiological responses such as skin conductivity, we are able to capture behaviors in real time that can be used to supplement traditional assessor ratings. Doing so allows assessors to compile a more accurate and holistic summary of a candidate’s performance. This wealth of new behavioral information has implications for high stakes hiring decisions as well as targeted training and development.
KeywordsAssessment centers Assessment center method Judgment and decision making Process tracing Selection Simulation Technology Training Development Leadership Simulations
This research was supported by the SIOP Foundation’s Douglas W. Bray and Ann Howard Research Grant. The authors would like to thank Carolyn Jagacinski and James LeBreton for helpful feedback.
- Abelson, R. P., & Black, J. B. (1986). Introduction. In J. A. Galambos, R. P. Abelson & J. B. Black (Eds.), Knowledge structures (pp. 1–18). Hillsdale: Lawrence Erlbaum Associates.Google Scholar
- Corporate Leadership Council (2005). Trends in developmental assessment centers. http://www.scribd.com/doc/57017184/Trends-in-Developmental-Assessment-Centers. Accessed 1 Dec 2011.
- Dean, M. A., Roth, P. L., & Bobko, P. (2008). Ethnic and gender subgroup differences in assessment center ratings: A meta-analysis. Journal of Applied Psychology, 93(3), 685.Google Scholar
- Driskell, J. E., Salas, E., & Hall, J. K. (1994). The effect of vigilant and hypervigilant decision training on performance. In Annual Meeting of the Society of Industrial and Organizational Psychology, Nashville.Google Scholar
- Duchowski, A. (2007). Eye tracking methodology: Theory and practice (2nd Edition). London: Springer.Google Scholar
- Evans, M. (2012). The key lessons from RIM’s struggles. Forbes. From http://www.forbes.com/sites/markevans/2012/07/04/key-lessons-from-rim/. Accessed 1 Mar 2013.
- Ferreira, M. B., Garcia-Marques, L., Sherman, S. J., & Sherman, J. W. (2006).Automatic and controlled components of judgment and decision making. Journal of Personality and Social Psychology, 91(5), 797.Google Scholar
- Figner, B., & Murphy, R. O. (2011). Using skin conductance in judgment and decision making research. In M. Schulte-Mecklenbeck, A. Kuehberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 163–184). New York: Psychology Press.Google Scholar
- Gibbons, A. M., Hughes, D., Riley, P., Thornton, G. C., & Sanchez, D. (2013). Is the future here? Current uses and perceived benefits of technology in operational assessment centers Unpublished Manuscript.Google Scholar
- Hammond, K. R. (1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable injustice. USA: Oxford University Press.Google Scholar
- Hammond, K. R., Hamm, R. M., Grassia, J., & Pearson, T. (1997). Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment. In W. M. Goldstein & R. M. Hogarth (Eds.), Research on judgment and decision making: Currents, connections, and controversies (pp. 144–180). Cambridge series on judgment and decision making. Cambridge, UK: Cambridge University Press.Google Scholar
- Hayes-Roth, B. (1977).Evolution of cognitive structures and processes. Psychological Review, 84(3), 260.Google Scholar
- Hughes, D., Riley, P., Shalfrooshan, A., Gibbons, A., Thornton, G. (2012). A global survey of assessment center practices. A research report by a & dc and Colorado State University.Google Scholar
- Koop, G. J., & Johnson, J. G. (2011). Response dynamics: A new window in the decision process. Judgment and Decision Making, 6(8), 750–758.Google Scholar
- Lievens, F., & Schollaert, E. (2011). Reshaping exercise design in assessment centers: Theory, practice, and research. In N. Povah & G. C. Thornton III (Eds.), Assessment centres and global talent management (pp. 47–60). Surrey, UK: Gower.Google Scholar
- Mann, L., Harmoni, R., & Power, C. (1991). The GOFER course in decision making. In J. Baron & R. V. Brown (Eds.), Teaching decision making to adolescents (pp. 61–78). Hillsdale: Lawrence Erlbaum Associates, Inc.Google Scholar
- McGregor, J. (2012). In cruise ship sinking, leadership failures from captain to Carnival CEO. The Washington Post Online. http://www.washingtonpost.com/blogs/post-leadership/post/in-cruise-ship-sinking-leadership-failures-from-captain-to-carnival-ceo/2011/04/01/gIQACNtkAQ_blog.html. Accessed 1 Feb 2012.
- Miller, C. C., & Kopytoff, V. G. (2011). Once a leader, Yahoo now struggles to find its way. The New York Times Online. http://www.nytimes.com/2011/09/08/technology/once-a-leader-yahoo-now-struggles-to-find-its-way.html?pagewanted=all&_r=0. Accessed 1 Mar 2012.
- National Bureau of Economic Research. (2008). Determination of the December 2007 peak in economic activity. National Bureau of Economic Research. From http://www.nber.org/cycles/dec2008.pdf. Accessed 1 Nov 2011.
- Nisbett, R. E., & Wilson, T. D. (1977).Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231.Google Scholar
- Povah, N., & Thornton, G. C. III. (2011). Assessment centers and global talent management. Surrey, UK: Gower Publishing Limited.Google Scholar
- Reynolds, D. H., & Dickter, D. N. (2010). Technology and employee selection. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection. New York: Psychology Press.Google Scholar
- Reynolds, D. H., & Rupp, D. E. (2010). Advances in technology-facilitated assessment. In J. C. Scott & D. H. Reynolds (Eds.), Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent (pp. 609–641). San Francisco: Jossey-Bass.Google Scholar
- Russo, J. E. (2011). Eye fixation as a process trace. In M. Schulte-Mecklenbeck, A. Kühberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 43–64). New York: Psychology Press.Google Scholar
- Simon, Z. (2012). Malev airline stops services after 66 years as Hungary moves to cut losses. Bloomberg Online. From http://www.bloomberg.com/news/2012–02-03/malev-stops-flying-after-66-years-as-hungary-cuts-its-losses.html. Accessed 1 Mar 2012.
- Sims, H. P., & Gioia, D. A. (1986). The thinking organization. San Francisco: Jossey-Bass.Google Scholar
- Soll, J. B., & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions. Journal of Experimental Psychology: Learning, Memory,and Cognition, 35(3), 780.Google Scholar
- Spector, M., & Trachtenberg, J. A. (2011). Borders forced to liquidate, close all stores. The Wall Street Journal Online. http://online.wsj.com/article/SB10001424052702303661904576454353768550280.html. Accessed 1 Jan 2013.
- Starbuck, W. H., & Milliken, F. (1988). Executives’ perceptual filters: What they notice and how they make sense. In D. Hambrick (Ed.), The executive effect: Concepts and methods of studying top managers (pp. 35–65). Greenwich, CT: JAI Press.Google Scholar
- Tetlock, P. E., & Boettger, R. (1989). Accountability: A social magnifier of the dilution effect. Journal of Personality and Social Psychology, 57(3), 388.Google Scholar
- Thornton, G. C. III, & Mueller-Hanson, R. (2004). Developing organizational simulations: A guide for practitioners and students. Series in applied psychology. Mahwah: Lawrence Erlbaum Associates.Google Scholar
- Thornton, G. C. III, & Rupp, D. E. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis, and development. Mahawah: Lawrence Erlbaum Associates, Inc.Google Scholar
- Wang, J. T. Y. (2011). Pupil dilation and eye tracking. In M. Schulte-Mecklenbeck, A. Kühberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 185–204). New York: Psychology Press.Google Scholar
- Worthen, B. (2012). H-P, Dell struggle as buyers shun PCs. The Wall Street Journal Online. http://online.wsj.com/article/SB10000872396390444812704577605703329715394.html. Accessed 1 Mar 2012.
- Zukier, H. (1982).The dilution effect: The role of the correlation and the dispersion of predictor variables in the use of nondiagnostic information. Journal of Personality and Social Psychology, 43(6), 1163.Google Scholar