Advances in Health Sciences Education

, Volume 18, Issue 2, pp 291–303 | Cite as

Exploring the impact of mental workload on rater-based assessments

Review

Abstract

When appraising the performance of others, assessors must acquire relevant information and process it in a meaningful way in order to translate it effectively into ratings, comments, or judgments about how well the performance meets appropriate standards. Rater-based assessment strategies in health professional education, including scale and faculty development strategies aimed at improving them have generally been implemented with limited consideration of human cognitive and perceptual limitations. However, the extent to which the task assigned to raters aligns with their cognitive and perceptual capacities will determine the extent to which reliance on human judgment threatens assessment quality. It is well recognized in medical decision making that, as the amount of information to be processed increases, judges may engage mental shortcuts through the application of schemas, heuristics, or the adoption of solutions that satisfy rather than optimize the judge’s needs. Further, these shortcuts may fundamentally limit/bias the information perceived or processed. Thinking of the challenges inherent in rater-based assessments in an analogous way may yield novel insights regarding the limits of rater-based assessment and may point to greater understanding of ways in which raters can be supported to facilitate sound judgment. This paper presents an initial exploration of various cognitive and perceptual limitations associated with rater-based assessment tasks. We hope to highlight how the inherent cognitive architecture of raters might beneficially be taken into account when designing rater-based assessment protocols.

Keywords

Assessment Clinical performance Competence Rater cognition Mental workload Performance assessment 

References

  1. Alexander, A. L., Nygren, T. E., & Vidulich, M. A. (2000). Examining the relationship between mental workload and situation awareness in a simulated air combat task AFRL-HE-WPTR-2000-0094. Air Force Research Laboratory, Human Effectiveness Directorate, Wright-Patterson AFB, OH.Google Scholar
  2. Baddeley, A. D. (2007). Working memory, thought and action. Oxford: Oxford University Press.CrossRefGoogle Scholar
  3. Bannert, M. (2002). Managing cognitive load-recent trends in cognitive load theory. Learning and Instruction, 12(1), 139–146.CrossRefGoogle Scholar
  4. Beckmann, J. F. (2010). Taming a beast of burden-On some issues with the conceptualisation and operationalisation of cognitive load. Learning and Instruction, 20(3), 250–264.CrossRefGoogle Scholar
  5. Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3), 417–428.Google Scholar
  6. Borman, W. C. (1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63(2), 135–144.CrossRefGoogle Scholar
  7. Botvinick, M. M., & Rosen, Z. B. (2009). Anticipation of cognitive demand during decision-making. Psychological Research, 73(6), 835–842.CrossRefGoogle Scholar
  8. Byrne, A., Oliver, M., Bodger, O., Barnett, W., Williams, D., Jones, H., et al. (2010). Novel method of measuring the mental workload of anaesthetists during clinical practice. British Journal of Anaesthesia, 105(6), 767.CrossRefGoogle Scholar
  9. Cook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010). Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Advances in Health Sciences Education, 15(5), 633–645.CrossRefGoogle Scholar
  10. Cooper, W. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218–244.CrossRefGoogle Scholar
  11. Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: construct alignment improves the performance of workplace based assessment scales. Medical education, 45(6), 560–569.Google Scholar
  12. Crossley, J., & Jolly, B. (2012). Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28–37.CrossRefGoogle Scholar
  13. Davis, D., Oliver, M., & Byrne, A. (2009). A novel method of measuring the mental workload of anaesthetists during simulated practice. British Journal of Anaesthesia, 103(5), 665.CrossRefGoogle Scholar
  14. DeNisi, A., Cafferty, T., & Meglino, B. (1984). A cognitive view of the performance appraisal process: A model and research propositions* 1. Organizational Behavior and Human Performance, 33(3), 360–396.CrossRefGoogle Scholar
  15. Dore, K. L., Hanson, M., Reiter, H. I., Blanchard, M., Deeth, K., & Eva, K. W. (2006). Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Academic Medicine, 81(10), S70.CrossRefGoogle Scholar
  16. Downing, S. (2005). Threats to the validity of clinical teaching assessments: What about rater error? Medical Education, 39(4), 353–355.CrossRefGoogle Scholar
  17. Eva, K. W. (2008). On the limits of systematicity. Medical Education, 42(9), 852–853.CrossRefGoogle Scholar
  18. Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004). An admissions OSCE: The multiple mini interview. Medical Education, 38(3), 314–326.CrossRefGoogle Scholar
  19. Feldman, J. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66(2), 127–148.CrossRefGoogle Scholar
  20. Gaugler, B., & Thornton, G. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74(4), 611.CrossRefGoogle Scholar
  21. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780.CrossRefGoogle Scholar
  22. Gopher, D., & Braune, R. (1984). On the psychophysics of workload: Why bother with subjective measures? Human Factors, 26, 519–532.Google Scholar
  23. Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.Google Scholar
  24. Govaerts, M., van der Vleuten, C., Schuwirth, L., & Muijtjens, A. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.CrossRefGoogle Scholar
  25. Green, M. L., & Holmboe, E. (2010). Perspective: The ACGME toolbox: Half empty or half full? Academic Medicine, 85(5), 787.CrossRefGoogle Scholar
  26. Haber, R., & Avins, A. (1994). Do ratings on the American board of internal medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9(3), 140–145.CrossRefGoogle Scholar
  27. Herbers, J., Noel, G., Cooper, G., Harvey, J., Pangaro, L., & Weaver, M. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4(3), 202–208.CrossRefGoogle Scholar
  28. Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43.CrossRefGoogle Scholar
  29. Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 74(10), 1129.CrossRefGoogle Scholar
  30. Holmboe, E. (2004). Faculty and the observation of trainees’ clinical skills: Problems and opportunities. Academic Medicine, 79(1), 16.CrossRefGoogle Scholar
  31. Huwendiek, S., Mennin, S., Dern, P., Friedman Ben-David, M., Van Der Fleuten, C., Tonshoff, B., et al. (2010). Expertise, needs and challanges of medical educators: Results of an international web survey. Medical Teacher, 32(11), 912–918.CrossRefGoogle Scholar
  32. Ilgen, D. (1983). & Feldman, J. Performance appraisal: A process focus. Research in Organizational Behavior.Google Scholar
  33. Ilgen, D., Barnes-Farrell, J., & McKellin, D. (1993). Performance appraisal process research in the 1980s: what has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321.CrossRefGoogle Scholar
  34. James, W. (1890). The Principles of Psychology. The Principles of Psychology. New York: Holt.CrossRefGoogle Scholar
  35. Kalet, A., Earp, J., & Kowlowitz, V. (1992). How well do faculty evaluate the interviewing skills of medical students? Journal of General Internal Medicine, 7(5), 499–505.CrossRefGoogle Scholar
  36. Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. Journals of American Medical Association, 302(12), 1316–1326.CrossRefGoogle Scholar
  37. Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139(4), 665.CrossRefGoogle Scholar
  38. LaMantia, J., Rennie, W., Risucci, D., Cydulka, R., Spillane, L., Graff, L., et al. (1999). Interobserver variability among faculty in evaluations of residents clinical skills. Academic Emergency Medicine, 6(1), 38–44.CrossRefGoogle Scholar
  39. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.CrossRefGoogle Scholar
  40. Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451.CrossRefGoogle Scholar
  41. Levitin, D. J. (2002). Foundations of cognitive psychology: core readings: The MIT Press, Cambridge.Google Scholar
  42. Lord, R. G., & Maher, K. J. (1990). Alternative information-processing models and their implications for theory, research, and practice. The Academy of Management Review, 15(1), 9–28.Google Scholar
  43. Lurie, S., Mooney, C., & Lyness, J. (2009). Measurement of the general competencies of the accreditation council for graduate medical education: A systematic review. Academic Medicine, 84(3), 301.CrossRefGoogle Scholar
  44. Mack, A. (2003). Inattentional blindness. Current Directions in Psychological Science, 12(5), 180.CrossRefGoogle Scholar
  45. Mack, A., & Rock, I. (1998). Inattentional Blindness. Cambridge, MA: MIT Press.Google Scholar
  46. Margolis, M., Clauser, B., Cuddy, M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10), S56.CrossRefGoogle Scholar
  47. Marois, R., & Ivanoff, J. (2005). Capacity limits of information processing in the brain. Trends in Cognitive Sciences, 9(6), 296–305.CrossRefGoogle Scholar
  48. Mayer, R. E. (2010). Applying the science of learning to medical education. Medical Education, 44(6), 543–549.CrossRefGoogle Scholar
  49. Melchers, K. G., Kleinmann, M., & Prinz, M. A. (2010). Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality. International Journal of Selection and Assessment, 18(3), 329–341.CrossRefGoogle Scholar
  50. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.CrossRefGoogle Scholar
  51. Morgeson, F., & Campion, M. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82(5), 627–654.CrossRefGoogle Scholar
  52. Noel, G., Herbers, J., Caplow, M., Cooper, G., Pangaro, L., & Harvey, J. (1992). How well do internal medicine faculty members evaluate the clinical skills of residents? Annals of Internal Medicine, 117(9), 757.Google Scholar
  53. Norcini, J., Blank, L., Duffy, F., & Fortna, G. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.Google Scholar
  54. Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist, 38(1), 63–71.CrossRefGoogle Scholar
  55. Pashler, H. (Ed.). (1998). Attention. East Sussex, UK: Psychology Press, Ltd.Google Scholar
  56. Payne, J. W. (1980). Information processing theory: Some concepts and methods applied to decision research. Cognitive processes in choice and decision behavior, 95, 115.Google Scholar
  57. Pérez Moreno, E., Conchillo, Á., & Recarte, M. A. (2011). The Role of Mental Load in Inattentional Blindness. Psicológica: Revista de metodología y psicología experimental, 32(2), 255–278.Google Scholar
  58. Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2), 327–344.CrossRefGoogle Scholar
  59. Schulz, C. M., Skrzypczak, M., Schneider, E., Hapfelmeier, A., Martin, J., Kochs, E. F., et al. (2011). Assessment of subjective workload in an anaesthesia simulator environment: reliability and validity. European Journal of Anaesthesiology, 28(7), 502.CrossRefGoogle Scholar
  60. Shah, A. K., & Oppenheimer, D. M. (2008). Heuristics made easy: An effort-reduction framework. Psychological Bulletin, 134(2), 207.CrossRefGoogle Scholar
  61. Simons, D. (2000). Attentional capture and inattentional blindness. Trends in Cognitive Sciences, 4(4), 147–155.CrossRefGoogle Scholar
  62. Thompson, W. G., Lipkin, M., Jr, Gilbert, D. A., Guzzo, R. A., & Roberson, L. (1990). Evaluating evaluation: assessment of the American board of internal medicine resident evaluation form. Journal of General Internal Medicine, 5(3), 214–217.CrossRefGoogle Scholar
  63. Tsang, P. S., & Vidulich, M. A. (2003). Principles and practice of aviation psychology Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  64. Tsang, P. S., & Vidulich, M. A. (2006). Mental workload and situation awareness. Hoboken, NJ: Wiley.Google Scholar
  65. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124.CrossRefGoogle Scholar
  66. Van Merriënboer, J. J. G., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85–93.CrossRefGoogle Scholar
  67. Wickens, C. D., & Carswell, C. (2006). Handbook of human factors and ergonomics (3rd ed.). Hoboken, NJ: Wiley.Google Scholar
  68. Williams, R., Klamen, D., & McGaghie, W. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.CrossRefGoogle Scholar
  69. Young, G., Zavelina, L., & Hooper, V. (2008). Assessment of workload using NASA task load index in perianesthesia nursing. Journal of PeriAnesthesia Nursing, 23(2), 102–110.CrossRefGoogle Scholar
  70. Yurko, Y. Y., Scerbo, M. W., Prabhu, A. S., Acker, C. E., & Stefanidis, D. (2010). Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare, 5(5), 267.CrossRefGoogle Scholar
  71. Zheng, B., Cassera, M. A., Martinec, D. V., Spaun, G. O., & Swanström, L. L. (2010). Measuring mental workload during the performance of advanced laparoscopic tasks. Surgical Endoscopy, 24(1), 45–50.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.School of Community and Health StudiesCentennial CollegeTorontoCanada
  2. 2. Wilson Centre for Health Professions Education ResearchUniversity of TorontoTorontoCanada
  3. 3.Centre for Health Education Scholarship, Department of MedicineUniversity of British ColumbiaVancouverCanada

Personalised recommendations