Skip to main content

Advertisement

Log in

Exploring the impact of mental workload on rater-based assessments

  • Review
  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

A Commentary to this article was published on 11 August 2012

Abstract

When appraising the performance of others, assessors must acquire relevant information and process it in a meaningful way in order to translate it effectively into ratings, comments, or judgments about how well the performance meets appropriate standards. Rater-based assessment strategies in health professional education, including scale and faculty development strategies aimed at improving them have generally been implemented with limited consideration of human cognitive and perceptual limitations. However, the extent to which the task assigned to raters aligns with their cognitive and perceptual capacities will determine the extent to which reliance on human judgment threatens assessment quality. It is well recognized in medical decision making that, as the amount of information to be processed increases, judges may engage mental shortcuts through the application of schemas, heuristics, or the adoption of solutions that satisfy rather than optimize the judge’s needs. Further, these shortcuts may fundamentally limit/bias the information perceived or processed. Thinking of the challenges inherent in rater-based assessments in an analogous way may yield novel insights regarding the limits of rater-based assessment and may point to greater understanding of ways in which raters can be supported to facilitate sound judgment. This paper presents an initial exploration of various cognitive and perceptual limitations associated with rater-based assessment tasks. We hope to highlight how the inherent cognitive architecture of raters might beneficially be taken into account when designing rater-based assessment protocols.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alexander, A. L., Nygren, T. E., & Vidulich, M. A. (2000). Examining the relationship between mental workload and situation awareness in a simulated air combat task AFRL-HE-WPTR-2000-0094. Air Force Research Laboratory, Human Effectiveness Directorate, Wright-Patterson AFB, OH.

  • Baddeley, A. D. (2007). Working memory, thought and action. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Bannert, M. (2002). Managing cognitive load-recent trends in cognitive load theory. Learning and Instruction, 12(1), 139–146.

    Article  Google Scholar 

  • Beckmann, J. F. (2010). Taming a beast of burden-On some issues with the conceptualisation and operationalisation of cognitive load. Learning and Instruction, 20(3), 250–264.

    Article  Google Scholar 

  • Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3), 417–428.

    Google Scholar 

  • Borman, W. C. (1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63(2), 135–144.

    Article  Google Scholar 

  • Botvinick, M. M., & Rosen, Z. B. (2009). Anticipation of cognitive demand during decision-making. Psychological Research, 73(6), 835–842.

    Article  Google Scholar 

  • Byrne, A., Oliver, M., Bodger, O., Barnett, W., Williams, D., Jones, H., et al. (2010). Novel method of measuring the mental workload of anaesthetists during clinical practice. British Journal of Anaesthesia, 105(6), 767.

    Article  Google Scholar 

  • Cook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010). Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Advances in Health Sciences Education, 15(5), 633–645.

    Article  Google Scholar 

  • Cooper, W. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218–244.

    Article  Google Scholar 

  • Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: construct alignment improves the performance of workplace based assessment scales. Medical education, 45(6), 560–569.

    Google Scholar 

  • Crossley, J., & Jolly, B. (2012). Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28–37.

    Article  Google Scholar 

  • Davis, D., Oliver, M., & Byrne, A. (2009). A novel method of measuring the mental workload of anaesthetists during simulated practice. British Journal of Anaesthesia, 103(5), 665.

    Article  Google Scholar 

  • DeNisi, A., Cafferty, T., & Meglino, B. (1984). A cognitive view of the performance appraisal process: A model and research propositions* 1. Organizational Behavior and Human Performance, 33(3), 360–396.

    Article  Google Scholar 

  • Dore, K. L., Hanson, M., Reiter, H. I., Blanchard, M., Deeth, K., & Eva, K. W. (2006). Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Academic Medicine, 81(10), S70.

    Article  Google Scholar 

  • Downing, S. (2005). Threats to the validity of clinical teaching assessments: What about rater error? Medical Education, 39(4), 353–355.

    Article  Google Scholar 

  • Eva, K. W. (2008). On the limits of systematicity. Medical Education, 42(9), 852–853.

    Article  Google Scholar 

  • Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004). An admissions OSCE: The multiple mini interview. Medical Education, 38(3), 314–326.

    Article  Google Scholar 

  • Feldman, J. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66(2), 127–148.

    Article  Google Scholar 

  • Gaugler, B., & Thornton, G. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74(4), 611.

    Article  Google Scholar 

  • Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780.

    Article  Google Scholar 

  • Gopher, D., & Braune, R. (1984). On the psychophysics of workload: Why bother with subjective measures? Human Factors, 26, 519–532.

    Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.

    Google Scholar 

  • Govaerts, M., van der Vleuten, C., Schuwirth, L., & Muijtjens, A. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.

    Article  Google Scholar 

  • Green, M. L., & Holmboe, E. (2010). Perspective: The ACGME toolbox: Half empty or half full? Academic Medicine, 85(5), 787.

    Article  Google Scholar 

  • Haber, R., & Avins, A. (1994). Do ratings on the American board of internal medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9(3), 140–145.

    Article  Google Scholar 

  • Herbers, J., Noel, G., Cooper, G., Harvey, J., Pangaro, L., & Weaver, M. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4(3), 202–208.

    Article  Google Scholar 

  • Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43.

    Article  Google Scholar 

  • Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 74(10), 1129.

    Article  Google Scholar 

  • Holmboe, E. (2004). Faculty and the observation of trainees’ clinical skills: Problems and opportunities. Academic Medicine, 79(1), 16.

    Article  Google Scholar 

  • Huwendiek, S., Mennin, S., Dern, P., Friedman Ben-David, M., Van Der Fleuten, C., Tonshoff, B., et al. (2010). Expertise, needs and challanges of medical educators: Results of an international web survey. Medical Teacher, 32(11), 912–918.

    Article  Google Scholar 

  • Ilgen, D. (1983). & Feldman, J. Performance appraisal: A process focus. Research in Organizational Behavior.

    Google Scholar 

  • Ilgen, D., Barnes-Farrell, J., & McKellin, D. (1993). Performance appraisal process research in the 1980s: what has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321.

    Article  Google Scholar 

  • James, W. (1890). The Principles of Psychology. The Principles of Psychology. New York: Holt.

    Book  Google Scholar 

  • Kalet, A., Earp, J., & Kowlowitz, V. (1992). How well do faculty evaluate the interviewing skills of medical students? Journal of General Internal Medicine, 7(5), 499–505.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. Journals of American Medical Association, 302(12), 1316–1326.

    Article  Google Scholar 

  • Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139(4), 665.

    Article  Google Scholar 

  • LaMantia, J., Rennie, W., Risucci, D., Cydulka, R., Spillane, L., Graff, L., et al. (1999). Interobserver variability among faculty in evaluations of residents clinical skills. Academic Emergency Medicine, 6(1), 38–44.

    Article  Google Scholar 

  • Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.

    Article  Google Scholar 

  • Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451.

    Article  Google Scholar 

  • Levitin, D. J. (2002). Foundations of cognitive psychology: core readings: The MIT Press, Cambridge.

  • Lord, R. G., & Maher, K. J. (1990). Alternative information-processing models and their implications for theory, research, and practice. The Academy of Management Review, 15(1), 9–28.

    Google Scholar 

  • Lurie, S., Mooney, C., & Lyness, J. (2009). Measurement of the general competencies of the accreditation council for graduate medical education: A systematic review. Academic Medicine, 84(3), 301.

    Article  Google Scholar 

  • Mack, A. (2003). Inattentional blindness. Current Directions in Psychological Science, 12(5), 180.

    Article  Google Scholar 

  • Mack, A., & Rock, I. (1998). Inattentional Blindness. Cambridge, MA: MIT Press.

    Google Scholar 

  • Margolis, M., Clauser, B., Cuddy, M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10), S56.

    Article  Google Scholar 

  • Marois, R., & Ivanoff, J. (2005). Capacity limits of information processing in the brain. Trends in Cognitive Sciences, 9(6), 296–305.

    Article  Google Scholar 

  • Mayer, R. E. (2010). Applying the science of learning to medical education. Medical Education, 44(6), 543–549.

    Article  Google Scholar 

  • Melchers, K. G., Kleinmann, M., & Prinz, M. A. (2010). Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality. International Journal of Selection and Assessment, 18(3), 329–341.

    Article  Google Scholar 

  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.

    Article  Google Scholar 

  • Morgeson, F., & Campion, M. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82(5), 627–654.

    Article  Google Scholar 

  • Noel, G., Herbers, J., Caplow, M., Cooper, G., Pangaro, L., & Harvey, J. (1992). How well do internal medicine faculty members evaluate the clinical skills of residents? Annals of Internal Medicine, 117(9), 757.

    Google Scholar 

  • Norcini, J., Blank, L., Duffy, F., & Fortna, G. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

    Google Scholar 

  • Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist, 38(1), 63–71.

    Article  Google Scholar 

  • Pashler, H. (Ed.). (1998). Attention. East Sussex, UK: Psychology Press, Ltd.

    Google Scholar 

  • Payne, J. W. (1980). Information processing theory: Some concepts and methods applied to decision research. Cognitive processes in choice and decision behavior, 95, 115.

    Google Scholar 

  • Pérez Moreno, E., Conchillo, Á., & Recarte, M. A. (2011). The Role of Mental Load in Inattentional Blindness. Psicológica: Revista de metodología y psicología experimental, 32(2), 255–278.

    Google Scholar 

  • Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2), 327–344.

    Article  Google Scholar 

  • Schulz, C. M., Skrzypczak, M., Schneider, E., Hapfelmeier, A., Martin, J., Kochs, E. F., et al. (2011). Assessment of subjective workload in an anaesthesia simulator environment: reliability and validity. European Journal of Anaesthesiology, 28(7), 502.

    Article  Google Scholar 

  • Shah, A. K., & Oppenheimer, D. M. (2008). Heuristics made easy: An effort-reduction framework. Psychological Bulletin, 134(2), 207.

    Article  Google Scholar 

  • Simons, D. (2000). Attentional capture and inattentional blindness. Trends in Cognitive Sciences, 4(4), 147–155.

    Article  Google Scholar 

  • Thompson, W. G., Lipkin, M., Jr, Gilbert, D. A., Guzzo, R. A., & Roberson, L. (1990). Evaluating evaluation: assessment of the American board of internal medicine resident evaluation form. Journal of General Internal Medicine, 5(3), 214–217.

    Article  Google Scholar 

  • Tsang, P. S., & Vidulich, M. A. (2003). Principles and practice of aviation psychology Mahwah, NJ: Lawrence Erlbaum Associates.

  • Tsang, P. S., & Vidulich, M. A. (2006). Mental workload and situation awareness. Hoboken, NJ: Wiley.

  • Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124.

    Article  Google Scholar 

  • Van Merriënboer, J. J. G., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85–93.

    Article  Google Scholar 

  • Wickens, C. D., & Carswell, C. (2006). Handbook of human factors and ergonomics (3rd ed.). Hoboken, NJ: Wiley.

  • Williams, R., Klamen, D., & McGaghie, W. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.

    Article  Google Scholar 

  • Young, G., Zavelina, L., & Hooper, V. (2008). Assessment of workload using NASA task load index in perianesthesia nursing. Journal of PeriAnesthesia Nursing, 23(2), 102–110.

    Article  Google Scholar 

  • Yurko, Y. Y., Scerbo, M. W., Prabhu, A. S., Acker, C. E., & Stefanidis, D. (2010). Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare, 5(5), 267.

    Article  Google Scholar 

  • Zheng, B., Cassera, M. A., Martinec, D. V., Spaun, G. O., & Swanström, L. L. (2010). Measuring mental workload during the performance of advanced laparoscopic tasks. Surgical Endoscopy, 24(1), 45–50.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Walter Tavares.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tavares, W., Eva, K.W. Exploring the impact of mental workload on rater-based assessments. Adv in Health Sci Educ 18, 291–303 (2013). https://doi.org/10.1007/s10459-012-9370-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-012-9370-3

Keywords

Navigation