Tracing Cognition with Assessment Center Simulations: Using Technology to See in the Dark

Chapter

Abstract

Shifts in the global economy have placed more pressure on the decisions of employees, managers, and CEOs than ever before. At the same time, the proliferation of technology in the workplace has become ubiquitous. Fortunately, the assessment center method is evolving alongside these other trends. With just a few modifications to existing assessment center simulations, we can use newer technology to capture previously difficult to observe behaviors that tap directly into candidate’s decision making processes. By combining passive data logging of key strokes and mouse clicks, eye tracking, and physiological responses such as skin conductivity, we are able to capture behaviors in real time that can be used to supplement traditional assessor ratings. Doing so allows assessors to compile a more accurate and holistic summary of a candidate’s performance. This wealth of new behavioral information has implications for high stakes hiring decisions as well as targeted training and development.

Keywords

Assessment centers Assessment center method Judgment and decision making Process tracing Selection Simulation Technology Training Development Leadership Simulations 

Notes

Acknowledgments

This research was supported by the SIOP Foundation’s Douglas W. Bray and Ann Howard Research Grant. The authors would like to thank Carolyn Jagacinski and James LeBreton for helpful feedback.

References

  1. Abelson, R. P., & Black, J. B. (1986). Introduction. In J. A. Galambos, R. P. Abelson & J. B. Black (Eds.), Knowledge structures (pp. 1–18). Hillsdale: Lawrence Erlbaum Associates.Google Scholar
  2. Bettman, J. R., & Park, C. W. (1980). Effects of prior knowledge and experience and phase of the choice process on consumer decision processes: A protocol analysis. Journal of Consumer Research, 7(3), 234–248.CrossRefGoogle Scholar
  3. Bruine de Bruin, W., Parker, A. M., & Fischoff, B. (2007). Individual differences in adult decision-making competence. Journal of Personality and Social Psychology, 92, 938–956.PubMedCrossRefGoogle Scholar
  4. Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4, 55–81.CrossRefGoogle Scholar
  5. Cook, G. J., & Swain, M. R. (1993). A computerized approach to decision process tracing for decision support systems design. Decision Science, 24, 683–697.CrossRefGoogle Scholar
  6. Corporate Leadership Council (2005). Trends in developmental assessment centers. http://www.scribd.com/doc/57017184/Trends-in-Developmental-Assessment-Centers. Accessed 1 Dec 2011.
  7. Dean, M. A., Roth, P. L., & Bobko, P. (2008). Ethnic and gender subgroup differences in assessment center ratings: A meta-analysis. Journal of Applied Psychology, 93(3), 685.Google Scholar
  8. Dilchert, S., & Ones, D. S. (2009). Assessment center dimensions: Individual differences correlates and meta-analytic incremental validity. International Journal of Selection and Assessment, 17(3), 254–270.CrossRefGoogle Scholar
  9. Driskell, J. E., Salas, E., & Hall, J. K. (1994). The effect of vigilant and hypervigilant decision training on performance. In Annual Meeting of the Society of Industrial and Organizational Psychology, Nashville.Google Scholar
  10. Duchowski, A. (2007). Eye tracking methodology: Theory and practice (2nd Edition). London: Springer.Google Scholar
  11. Edwards, W., & Fasolo, B. (2001). Decision technology. Annual Review of Psychology, 52(1), 581–606.PubMedCrossRefGoogle Scholar
  12. Einhorn, H. J., & Hogarth, R. M. (1981). Behavioral decision theory: Processes of judgment and choice. Annual Review of Psychology, 32, 53–88.CrossRefGoogle Scholar
  13. Evans, M. (2012). The key lessons from RIM’s struggles. Forbes. From http://www.forbes.com/sites/markevans/2012/07/04/key-lessons-from-rim/. Accessed 1 Mar 2013.
  14. Farr, J. L. (1973). Response requirements and primacy-recency effects in a simulated selection interview. Journal of Applied Psychology, 57(3), 228–232.CrossRefGoogle Scholar
  15. Ferreira, M. B., Garcia-Marques, L., Sherman, S. J., & Sherman, J. W. (2006).Automatic and controlled components of judgment and decision making. Journal of Personality and Social Psychology, 91(5), 797.Google Scholar
  16. Figner, B., & Murphy, R. O. (2011). Using skin conductance in judgment and decision making research. In M. Schulte-Mecklenbeck, A. Kuehberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 163–184). New York: Psychology Press.Google Scholar
  17. Ford, J. K., Schmitt, N., Schechtman, S. L., Hults, B. M., & Doherty, M. L. (1989). Process tracing methods: Contributions, problems, and neglected research questions. Organizational Behavior and Human Decision Processes, 43, 75–117.CrossRefGoogle Scholar
  18. Gibbons, A. M., Hughes, D., Riley, P., Thornton, G. C., & Sanchez, D. (2013). Is the future here? Current uses and perceived benefits of technology in operational assessment centers Unpublished Manuscript.Google Scholar
  19. Hammond, K. R. (1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable injustice. USA: Oxford University Press.Google Scholar
  20. Hammond, K. R., Hamm, R. M., Grassia, J., & Pearson, T. (1997). Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment. In W. M. Goldstein & R. M. Hogarth (Eds.), Research on judgment and decision making: Currents, connections, and controversies (pp. 144–180). Cambridge series on judgment and decision making. Cambridge, UK: Cambridge University Press.Google Scholar
  21. Hayes-Roth, B. (1977).Evolution of cognitive structures and processes. Psychological Review, 84(3), 260.Google Scholar
  22. Highhouse, S. (1997). Understanding and improving job-finalist choice: The relevance of behavioral decision research. Human Resource Management Review, 7(4), 449–470.CrossRefGoogle Scholar
  23. Huber, J., Payne, J. W., & Puto, C. (1982). Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis. Journal of Consumer Research, 9, 90–98.CrossRefGoogle Scholar
  24. Hughes, D., Riley, P., Shalfrooshan, A., Gibbons, A., Thornton, G. (2012). A global survey of assessment center practices. A research report by a & dc and Colorado State University.Google Scholar
  25. International Task Force on Assessment Center Guidelines (2009). Guidelines and ethical considerations for assessment center operations. International Journal of Selection and Assessment, 17, 243–253.CrossRefGoogle Scholar
  26. Jacoby, J., Mazursky, D., Troutman, T., Kuss, A. (1984). When feedback is ignored: Disutility of outcome feedback. Journal of Applied Psychology, 69(3), 531–545.CrossRefGoogle Scholar
  27. Jasper, J. D., & Shapiro, J. (2002). MouseTrace: A better mousetrap for catching decision processes. Behavior Research Methods, Instruments, & Computers, 34, 364–374.CrossRefGoogle Scholar
  28. Johnston, J. H., Driskell, J. E., & Salas, E. (1997). Vigilant and hypervigilant decision making. Journal of Applied Psychology, 82(4), 614–622.PubMedCrossRefGoogle Scholar
  29. Kirschenbaum, S. S. (1992). Influence of experience on information-gathering strategies. Journal of Applied Psychology, 77(3), 343–352.CrossRefGoogle Scholar
  30. Klein, J. D. (1997). ETR & D-Development: An analysis of content and survey of future direction. Educational Technology Research and Development, 45(3), 57–62.CrossRefGoogle Scholar
  31. Koop, G. J., & Johnson, J. G. (2011). Response dynamics: A new window in the decision process. Judgment and Decision Making, 6(8), 750–758.Google Scholar
  32. Lievens, F. (2008). What does exercise-based assessment really mean? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1(1), 112–115.CrossRefGoogle Scholar
  33. Lievens, F., & Schollaert, E. (2011). Reshaping exercise design in assessment centers: Theory, practice, and research. In N. Povah & G. C. Thornton III (Eds.), Assessment centres and global talent management (pp. 47–60). Surrey, UK: Gower.Google Scholar
  34. Lievens, F., Tett, R. P., & Schleicher, D. J. (2009). Assessment centers at the crossroads: Toward a reconceptualization of assessment center exercises. Research in Personnel and Human Resources Management, 28, 99–152.CrossRefGoogle Scholar
  35. Mann, L., Harmoni, R., & Power, C. (1991). The GOFER course in decision making. In J. Baron & R. V. Brown (Eds.), Teaching decision making to adolescents (pp. 61–78). Hillsdale: Lawrence Erlbaum Associates, Inc.Google Scholar
  36. McGregor, J. (2012). In cruise ship sinking, leadership failures from captain to Carnival CEO. The Washington Post Online. http://www.washingtonpost.com/blogs/post-leadership/post/in-cruise-ship-sinking-leadership-failures-from-captain-to-carnival-ceo/2011/04/01/gIQACNtkAQ_blog.html. Accessed 1 Feb 2012.
  37. Miller, C. C., & Kopytoff, V. G. (2011). Once a leader, Yahoo now struggles to find its way. The New York Times Online. http://www.nytimes.com/2011/09/08/technology/once-a-leader-yahoo-now-struggles-to-find-its-way.html?pagewanted=all&_r=0. Accessed 1 Mar 2012.
  38. Miller, J. W., & Rowe, P. M. (1967). Influence of favorable and unfavorable information upon assessment decisions. Journal of Applied Psychology, 51(5), 432–435.PubMedCrossRefGoogle Scholar
  39. National Bureau of Economic Research. (2008). Determination of the December 2007 peak in economic activity. National Bureau of Economic Research. From http://www.nber.org/cycles/dec2008.pdf. Accessed 1 Nov 2011.
  40. Nisbett, R. E., & Wilson, T. D. (1977).Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231.Google Scholar
  41. Nisbett, R. E., Zukier, H., & Lemley, R. E. (1981). The dilution effect: Nondiagnostic information weakens the implications of diagnostic information. Cognitive Psychology, 13(2), 248–277.CrossRefGoogle Scholar
  42. Onken, J., Hastie, R., & Revelle, W. (1985). Individual differences in the use of simplification strategies in a complex decision-making task. Journal of Experimental Psychology: Human Perception and Performance, 11(1), 14–27.PubMedCrossRefGoogle Scholar
  43. Patrick, J., & James, N. (2004). Process tracing of complex cognitive work tasks. Journal of Occupational and Organizational Psychology, 77, 259–280.CrossRefGoogle Scholar
  44. Payne, J. W. (1976). Task complexity and contingent processing in decision making: An informational search and protocol analysis. Organizational Behavior and Human Performance, 16, 366–387.CrossRefGoogle Scholar
  45. Payne, J. W. (1982). Contingent decision behavior. Psychological Bulletin, 92(2), 382–402.CrossRefGoogle Scholar
  46. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  47. Payne, J. W., Braunstein, M. L., & Carroll, J. S. (1978). Exploring predecisional behavior: An alternative approach to decision research. Organizational Behavior and Human Performance, 22(1), 17–44.CrossRefGoogle Scholar
  48. Perrin, B. M., Barnett, B. J., Walrath, L., & Grossman, J. D. (2001). Information order and outcome framing: An assessment of judgment bias in a naturalistic decision-making context. Human Factors: The Journal of the Human Factors and Ergonomics Society, 43(2), 227–238.CrossRefGoogle Scholar
  49. Povah, N., & Thornton, G. C. III. (2011). Assessment centers and global talent management. Surrey, UK: Gower Publishing Limited.Google Scholar
  50. Reynolds, D. H., & Dickter, D. N. (2010). Technology and employee selection. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection. New York: Psychology Press.Google Scholar
  51. Reynolds, D. H., & Rupp, D. E. (2010). Advances in technology-facilitated assessment. In J. C. Scott & D. H. Reynolds (Eds.), Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent (pp. 609–641). San Francisco: Jossey-Bass.Google Scholar
  52. Riedl, R., Brandstätter, E., & Roithmayr, R. (2008). Identifying decision strategies: A process- and outcome-based classification method. Behavioral Research Methods, 40(3), 795–807.CrossRefGoogle Scholar
  53. Rigney, J. W., & DeBow, C. H. (1967). Multidimensional scaling analysis of decision strategies in threat evaluation. Journal of Applied Psychology, 51(4), 305–310.PubMedCrossRefGoogle Scholar
  54. Rowe, P. M. (1967). Order effects in assessment decisions. Journal of Applied Psychology, 51(2), 170–173.PubMedCrossRefGoogle Scholar
  55. Rupp, D. E., Gibbons, A. M., & Snyder, L. A. (2008). The role of technology in enabling third-generation training and development. Industrial and Organizational Psychology, 1(4), 496–500.CrossRefGoogle Scholar
  56. Russo, J. E. (2011). Eye fixation as a process trace. In M. Schulte-Mecklenbeck, A. Kühberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 43–64). New York: Psychology Press.Google Scholar
  57. Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for Frame-of-Reference Training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87(4), 735–746.PubMedCrossRefGoogle Scholar
  58. Sherer, P. D., Schwab, D. P., & Heneman, H. G. III (1987). Managerial salary-raise decisions: A policy-capturing approach. Personnel Psychology, 40, 27–38.CrossRefGoogle Scholar
  59. Simon, Z. (2012). Malev airline stops services after 66 years as Hungary moves to cut losses. Bloomberg Online. From http://www.bloomberg.com/news/2012–02-03/malev-stops-flying-after-66-years-as-hungary-cuts-its-losses.html. Accessed 1 Mar 2012.
  60. Simonson, I. (1989). Choice based on reasons: The case of attraction and compromise effects. Journal of Consumer Research, 16, 158–174.CrossRefGoogle Scholar
  61. Sims, H. P., & Gioia, D. A. (1986). The thinking organization. San Francisco: Jossey-Bass.Google Scholar
  62. Soll, J. B., & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions. Journal of Experimental Psychology: Learning, Memory,and Cognition, 35(3), 780.Google Scholar
  63. Spector, M., & Trachtenberg, J. A. (2011). Borders forced to liquidate, close all stores. The Wall Street Journal Online. http://online.wsj.com/article/SB10001424052702303661904576454353768550280.html. Accessed 1 Jan 2013.
  64. Starbuck, W. H., & Milliken, F. (1988). Executives’ perceptual filters: What they notice and how they make sense. In D. Hambrick (Ed.), The executive effect: Concepts and methods of studying top managers (pp. 35–65). Greenwich, CT: JAI Press.Google Scholar
  65. Svenson, O. (1979). Process descriptions of decision making. Organizational Behavior and Human Performance, 23, 86–112.CrossRefGoogle Scholar
  66. Tabatabai, M. (1998). Investigation of decision making process: A hypermedia approach. Interacting With Computers, 9, 385–396.CrossRefGoogle Scholar
  67. Tetlock, P. E., & Boettger, R. (1989). Accountability: A social magnifier of the dilution effect. Journal of Personality and Social Psychology, 57(3), 388.Google Scholar
  68. Thorngate, W. (1980). Efficient decision heuristics. Behavioral Science, 25(3), 219–225.CrossRefGoogle Scholar
  69. Thornton, G. C. III, & Mueller-Hanson, R. (2004). Developing organizational simulations: A guide for practitioners and students. Series in applied psychology. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  70. Thornton, G. C. III, & Rupp, D. E. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis, and development. Mahawah: Lawrence Erlbaum Associates, Inc.Google Scholar
  71. Walsh, J. P. (1995). Managerial and organizational cognition: Notes from a trip down memory lane. Organizational Science, 6(3), 280–321.CrossRefGoogle Scholar
  72. Wang, J. T. Y. (2011). Pupil dilation and eye tracking. In M. Schulte-Mecklenbeck, A. Kühberger & R. Ranyard (Eds.), A handbook of process tracing methods for decision research: A critical review and user’s guide (pp. 185–204). New York: Psychology Press.Google Scholar
  73. Worthen, B. (2012). H-P, Dell struggle as buyers shun PCs. The Wall Street Journal Online. http://online.wsj.com/article/SB10000872396390444812704577605703329715394.html. Accessed 1 Mar 2012.
  74. Zakay, D., & Wooler, S. (1984). Time pressure, training and decision effectiveness. Ergonomics, 27(3), 273–284.PubMedCrossRefGoogle Scholar
  75. Zukier, H. (1982).The dilution effect: The role of the correlation and the dispersion of predictor variables in the use of nondiagnostic information. Journal of Personality and Social Psychology, 43(6), 1163.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Brett W. Guidry
    • 1
  • Deborah E. Rupp
    • 1
  • Martin Lanik
    • 2
  1. 1.Purdue UniversityINUSA
  2. 2.Global Assessor PoolDenverUSA

Personalised recommendations