Advances in Health Sciences Education

, Volume 24, Issue 1, pp 85–102 | Cite as

From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements

  • Victor LeeEmail author
  • Keira Brain
  • Jenepher Martin


The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.


Assessment Assessment of clinical competence Faculty development Mini-CEX Performance assessment Postgraduate training Rater cognition Workplace-based assessment 



The authors would like to thank Marjan Govaerts and Andrea Gingerich for their time and expertise in providing feedback regarding our survey and focus group questions. We would also like to thank staff at Monash University Eastern Health Clinical School for their assistance, the Australasian College for Emergency Medicine for permitting us to conduct this research and the Fellows of the College who participated.

Compliance with ethical standards

Conflict of interest

K Brain and J. Martin declare they have no conflicts of interest. V. Lee was chair of the Workplace-based Assessment working group during the period of this research, but this was a voluntary and unpaid position for the Australasian College for Emergency Medicine. This research was conducted independently without funding or coercion as part of his Master of Health Professional Education thesis at Monash University.

Supplementary material

10459_2018_9851_MOESM1_ESM.docx (87 kb)
Supplementary material 1 (DOCX 86 kb)
10459_2018_9851_MOESM2_ESM.docx (24 kb)
Supplementary material 2 (DOCX 24 kb)


  1. Bandiera, G., & Lendrum, D. (2008). Daily encounter cards facilitate competency-based feedback while leniency bias persists. Canadian Journal of Emergency Medicine, 10(1), 44–50.Google Scholar
  2. Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors’ perspectives. Advances in Health Sciences Education, 18(4), 559–571.Google Scholar
  3. Bredo, E. (1994). Reconstructing educational psychology: Situated cognition and Deweyian pragmatism. Educational Psychology, 29(1), 23–35.Google Scholar
  4. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.Google Scholar
  5. Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of Mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.Google Scholar
  6. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: SAGE Publications Inc.Google Scholar
  7. De Vaus, D. (2014). Surveys in social research (6th ed.). Abingdon-on-Thames: Routledge.Google Scholar
  8. Driessen, E., & Scheele, F. (2013). What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research. Medical Teacher, 35(7), 569–574.Google Scholar
  9. Durning, S. J., & Artino, A. R. (2011). Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical Teacher, 33(3), 188–199.Google Scholar
  10. Durning, S. J., Artino, A. R., Boulet, J. R., Dorrance, K., van der Vleuten, C., & Schuwirth, L. (2012). The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in Health Sciences Education, 17(1), 65–79.Google Scholar
  11. Durning, S. J., Artino, A. R., Jr., Pangaro, L. N., van der Vleuten, C., & Schuwirth, L. (2010). Perspective: Redefining context in the clinical encounter: Implications for research and training in medical education. Academic Medicine, 85(5), 894–901.Google Scholar
  12. Durning, S., Artino, A. R., Pangaro, L., van der Vleuten, C., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45(9), 927–938.Google Scholar
  13. Eva, K. W. (2018). Cognitive influences on complex performance assessment: Lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7(2), 177–188.Google Scholar
  14. Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522.Google Scholar
  15. Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014a). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.Google Scholar
  16. Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10 Suppl), S1–S7.Google Scholar
  17. Gingerich, A., van der Vleuten, C. P. M., Eva, K. W., & Regehr, G. (2014b). More consensus than idiosyncrasy: Categorizing social judgments to examine variability in Mini-CEX ratings. Academic Medicine, 89(11), 1–10.Google Scholar
  18. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.Google Scholar
  19. Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.Google Scholar
  20. Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396.Google Scholar
  21. Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.Google Scholar
  22. Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72(9), 861–871.Google Scholar
  23. Hall, K. H. (2002). Reviewing intuitive decision-making and uncertainty: The implications for medical education. Medical Education, 36(3), 216–224.Google Scholar
  24. Hill, Faith, Kathleen, Kendall, Galbraith, Kevin, & Crossley, J. (2009). Implemanting the undergradate Mini-CEX: A tailored approach at Southmpatom University. Medical Education, 43(4), 326–334.Google Scholar
  25. Hoffman, K. G., & Donaldson, J. F. (2004). Contextual tensions of the clinical environment and their influence on teaching and learning. Medical Education, 38(4), 448–454.Google Scholar
  26. Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004). Effects of training in direct observation of medical residents’ clinical competence: A randomized trial. Annals of Internal Medicine, 140(11), 874–881.Google Scholar
  27. Huber, V. L., Neale, M. A., & Nofthcraft, G. B. (1987). Judgment by heuristics: Effects of ratee and rater characteristics and performance standards on performance-related judgments. Organizational Behavior and Human Decision Processes, 40(2), 149–169.Google Scholar
  28. Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.Google Scholar
  29. Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.Google Scholar
  30. Kogan, J. R., Conforti, L. N., Bernabeo, E., Iobst, W., & Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: A qualitative study. Medical Education, 49(7), 692–708.Google Scholar
  31. Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721–727.Google Scholar
  32. Lavelle, E., Vuk, J., & Barber, C. (2013). Twelve tips for getting started using mixed methods in medical education research. Medical Teacher, 35, 272–276.Google Scholar
  33. Lee, V., Brain, K., & Martin, J. (2017). Factors influencing Mini-CEX rater judgments and their practical implications. Academic Medicine, 92(6), 880–887.Google Scholar
  34. Maudsley, G. (2011). Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review). Medical Teacher, 33(2), e92–e104.Google Scholar
  35. Mertens, F., de Groot, E., Meijer, L., Wens, J., Gemma Cherry, M., Deveugele, M., et al. (2017). Workplace learning through collaboration in primary healthcare: A BEME realist review of what works, for whom and in what circumstances: BEME Guide No. 46. Medical Teacher, 40(2), 117–134.Google Scholar
  36. Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The Mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.Google Scholar
  37. Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The Mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.Google Scholar
  38. Pangaro, L., & ten Cate, O. (2013). Frameworks for learner assessment in medicine: AMEE Guide No. 78. Medical Teacher, 35(6), e1197–e1210.Google Scholar
  39. Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604–612.Google Scholar
  40. Ramani, S., Könings, K. D., Ginsburg, S., & van der Vleuten, C. P. M. (2018). Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships. Medical Teacher, 7, 1–7. Scholar
  41. Rogausch, A., Beyeler, C., Montagne, S., Jucker-Kupper, P., Berendonk, C., Huwendiek, S., et al. (2015). The influence of students’ prior clinical skills and context characteristics on Mini-CEX scores in clerkships—a multilevel analysis. BMC Medical Education, 15(1), 208.Google Scholar
  42. Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: How trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education, 22(4), 931–949.Google Scholar
  43. Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical education: Basic guidelines for researchers. Medical Education, 43(7), 637–644.Google Scholar
  44. Schuh, L. A., London, Z., Neel, R., Brock, C., Kissela, B. M., Schultz, L., et al. (2009). Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology, 73(11), 904–908.Google Scholar
  45. Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011a). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485.Google Scholar
  46. Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011b). General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher, 33(10), 783–797.Google Scholar
  47. Sheehan, D., Jowsey, T., Parwaiz, M., Birch, M., Seaton, P., Shaw, S., et al. (2017). Clinical learning environments: Place, artefacts and rhythm. Medical Education, 51(10), 1049–1060.Google Scholar
  48. Stalmeijer, R. E., Mcnaughton, N., & Van Mook, W. N. K. A. (2014). Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher, 36(11), 923–939.Google Scholar
  49. Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18(2), 291–303.Google Scholar
  50. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.Google Scholar
  51. Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.Google Scholar
  52. Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37.Google Scholar
  53. Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education, 18(3), 523–525.Google Scholar
  54. Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.Google Scholar
  55. Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: Contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909–919.Google Scholar
  56. Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013a). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325–341.Google Scholar
  57. Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013b). “You”re certainly relatively competent: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Department of Emergency MedicineAustin HealthHeidelbergAustralia
  2. 2.Western HospitalFootscrayAustralia
  3. 3.Eastern Health Clinical SchoolMonash University and Deakin UniversityBox HillAustralia

Personalised recommendations