From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements

Abstract

The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. Bandiera, G., & Lendrum, D. (2008). Daily encounter cards facilitate competency-based feedback while leniency bias persists. Canadian Journal of Emergency Medicine, 10(1), 44–50.

    Google Scholar 

  2. Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors’ perspectives. Advances in Health Sciences Education, 18(4), 559–571.

    Article  Google Scholar 

  3. Bredo, E. (1994). Reconstructing educational psychology: Situated cognition and Deweyian pragmatism. Educational Psychology, 29(1), 23–35.

    Article  Google Scholar 

  4. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.

    Article  Google Scholar 

  5. Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of Mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.

    Article  Google Scholar 

  6. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: SAGE Publications Inc.

    Google Scholar 

  7. De Vaus, D. (2014). Surveys in social research (6th ed.). Abingdon-on-Thames: Routledge.

    Google Scholar 

  8. Driessen, E., & Scheele, F. (2013). What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research. Medical Teacher, 35(7), 569–574.

    Article  Google Scholar 

  9. Durning, S. J., & Artino, A. R. (2011). Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical Teacher, 33(3), 188–199.

    Article  Google Scholar 

  10. Durning, S. J., Artino, A. R., Boulet, J. R., Dorrance, K., van der Vleuten, C., & Schuwirth, L. (2012). The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in Health Sciences Education, 17(1), 65–79.

    Article  Google Scholar 

  11. Durning, S. J., Artino, A. R., Jr., Pangaro, L. N., van der Vleuten, C., & Schuwirth, L. (2010). Perspective: Redefining context in the clinical encounter: Implications for research and training in medical education. Academic Medicine, 85(5), 894–901.

    Article  Google Scholar 

  12. Durning, S., Artino, A. R., Pangaro, L., van der Vleuten, C., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45(9), 927–938.

    Article  Google Scholar 

  13. Eva, K. W. (2018). Cognitive influences on complex performance assessment: Lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7(2), 177–188.

    Article  Google Scholar 

  14. Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522.

    Article  Google Scholar 

  15. Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014a). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.

    Article  Google Scholar 

  16. Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10 Suppl), S1–S7.

    Article  Google Scholar 

  17. Gingerich, A., van der Vleuten, C. P. M., Eva, K. W., & Regehr, G. (2014b). More consensus than idiosyncrasy: Categorizing social judgments to examine variability in Mini-CEX ratings. Academic Medicine, 89(11), 1–10.

    Article  Google Scholar 

  18. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  19. Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.

    Article  Google Scholar 

  20. Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396.

    Article  Google Scholar 

  21. Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.

    Article  Google Scholar 

  22. Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72(9), 861–871.

    Article  Google Scholar 

  23. Hall, K. H. (2002). Reviewing intuitive decision-making and uncertainty: The implications for medical education. Medical Education, 36(3), 216–224.

    Article  Google Scholar 

  24. Hill, Faith, Kathleen, Kendall, Galbraith, Kevin, & Crossley, J. (2009). Implemanting the undergradate Mini-CEX: A tailored approach at Southmpatom University. Medical Education, 43(4), 326–334.

    Article  Google Scholar 

  25. Hoffman, K. G., & Donaldson, J. F. (2004). Contextual tensions of the clinical environment and their influence on teaching and learning. Medical Education, 38(4), 448–454.

    Article  Google Scholar 

  26. Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004). Effects of training in direct observation of medical residents’ clinical competence: A randomized trial. Annals of Internal Medicine, 140(11), 874–881.

    Article  Google Scholar 

  27. Huber, V. L., Neale, M. A., & Nofthcraft, G. B. (1987). Judgment by heuristics: Effects of ratee and rater characteristics and performance standards on performance-related judgments. Organizational Behavior and Human Decision Processes, 40(2), 149–169.

    Article  Google Scholar 

  28. Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.

    Article  Google Scholar 

  29. Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.

    Article  Google Scholar 

  30. Kogan, J. R., Conforti, L. N., Bernabeo, E., Iobst, W., & Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: A qualitative study. Medical Education, 49(7), 692–708.

    Article  Google Scholar 

  31. Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721–727.

    Article  Google Scholar 

  32. Lavelle, E., Vuk, J., & Barber, C. (2013). Twelve tips for getting started using mixed methods in medical education research. Medical Teacher, 35, 272–276.

    Article  Google Scholar 

  33. Lee, V., Brain, K., & Martin, J. (2017). Factors influencing Mini-CEX rater judgments and their practical implications. Academic Medicine, 92(6), 880–887.

    Article  Google Scholar 

  34. Maudsley, G. (2011). Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review). Medical Teacher, 33(2), e92–e104.

    Article  Google Scholar 

  35. Mertens, F., de Groot, E., Meijer, L., Wens, J., Gemma Cherry, M., Deveugele, M., et al. (2017). Workplace learning through collaboration in primary healthcare: A BEME realist review of what works, for whom and in what circumstances: BEME Guide No. 46. Medical Teacher, 40(2), 117–134.

    Article  Google Scholar 

  36. Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The Mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.

    Article  Google Scholar 

  37. Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The Mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

    Article  Google Scholar 

  38. Pangaro, L., & ten Cate, O. (2013). Frameworks for learner assessment in medicine: AMEE Guide No. 78. Medical Teacher, 35(6), e1197–e1210.

    Article  Google Scholar 

  39. Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604–612.

    Article  Google Scholar 

  40. Ramani, S., Könings, K. D., Ginsburg, S., & van der Vleuten, C. P. M. (2018). Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships. Medical Teacher, 7, 1–7. https://doi.org/10.1080/0142159X.2018.1432850.

    Article  Google Scholar 

  41. Rogausch, A., Beyeler, C., Montagne, S., Jucker-Kupper, P., Berendonk, C., Huwendiek, S., et al. (2015). The influence of students’ prior clinical skills and context characteristics on Mini-CEX scores in clerkships—a multilevel analysis. BMC Medical Education, 15(1), 208.

    Article  Google Scholar 

  42. Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: How trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education, 22(4), 931–949.

    Article  Google Scholar 

  43. Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical education: Basic guidelines for researchers. Medical Education, 43(7), 637–644.

    Article  Google Scholar 

  44. Schuh, L. A., London, Z., Neel, R., Brock, C., Kissela, B. M., Schultz, L., et al. (2009). Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology, 73(11), 904–908.

    Article  Google Scholar 

  45. Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011a). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485.

    Article  Google Scholar 

  46. Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011b). General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher, 33(10), 783–797.

    Article  Google Scholar 

  47. Sheehan, D., Jowsey, T., Parwaiz, M., Birch, M., Seaton, P., Shaw, S., et al. (2017). Clinical learning environments: Place, artefacts and rhythm. Medical Education, 51(10), 1049–1060.

    Article  Google Scholar 

  48. Stalmeijer, R. E., Mcnaughton, N., & Van Mook, W. N. K. A. (2014). Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher, 36(11), 923–939.

    Article  Google Scholar 

  49. Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18(2), 291–303.

    Article  Google Scholar 

  50. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

    Article  Google Scholar 

  51. Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.

    Article  Google Scholar 

  52. Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37.

    Article  Google Scholar 

  53. Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education, 18(3), 523–525.

    Article  Google Scholar 

  54. Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.

    Article  Google Scholar 

  55. Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: Contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909–919.

    Article  Google Scholar 

  56. Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013a). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325–341.

    Article  Google Scholar 

  57. Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013b). “You”re certainly relatively competent: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Marjan Govaerts and Andrea Gingerich for their time and expertise in providing feedback regarding our survey and focus group questions. We would also like to thank staff at Monash University Eastern Health Clinical School for their assistance, the Australasian College for Emergency Medicine for permitting us to conduct this research and the Fellows of the College who participated.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Victor Lee.

Ethics declarations

Conflict of interest

K Brain and J. Martin declare they have no conflicts of interest. V. Lee was chair of the Workplace-based Assessment working group during the period of this research, but this was a voluntary and unpaid position for the Australasian College for Emergency Medicine. This research was conducted independently without funding or coercion as part of his Master of Health Professional Education thesis at Monash University.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 86 kb)

Supplementary material 2 (DOCX 24 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, V., Brain, K. & Martin, J. From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements. Adv in Health Sci Educ 24, 85–102 (2019). https://doi.org/10.1007/s10459-018-9851-0

Download citation

Keywords

  • Assessment
  • Assessment of clinical competence
  • Faculty development
  • Mini-CEX
  • Performance assessment
  • Postgraduate training
  • Rater cognition
  • Workplace-based assessment