Skip to main content

Advertisement

SpringerLink
Log in

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Performance-based assessment (PBA) is a valued assessment approach in medical education, be it in a clerkship, residency, or practice context. Raters are intrinsic to PBA and the increased use of PBA has lead to an increased interest in rater cognition. Although several researchers have tackled factors that may influence the variability in rater judgment, the critical examination of rater observation of performance and the translation of that data into judgements are being investigated. The purpose of this study was to qualitatively investigate the cognitive processes of raters, and to create a framework that conceptualizes those processes when raters assess a complex performance. We conducted semi-structured interviews with 11 faculty members (nominated as excellent assessors) from a Department of Medicine to investigate how raters observe, interpret, and translate performance into judgments. The transcribed verbal protocols were analyzed using Constructivist Grounded Theory in order to develop a theoretical model of raters’ assessment processes. Several themes emerged from the data and were grouped according to three macro-level themes describing how the raters balance two sources of data [(1) external sources of information and (2) internal/personal sources of information] by relying on specific cognitive processes to assess an examinee performance. The results from our study demonstrate that assessment is a difficult cognitive task that involves nuance using specific cognitive processes to weigh external and internal data against each other. Our data clearly draws attention to the constant struggle between objectivity and subjectivity that is observed in assessment as illustrated by the importance given to nuancing the examinee’s observed performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Barrouillet, P. (2011). Dual-process theories and cognitive development: Advances and challenges. Developmental Review, 31(2), 79–95.

    Article  Google Scholar 

  • Beck, D. E., O’Sullivan, P. S., & Boh, L. E. (1995). Increasing the accuracy of observer ratings by enhancing cognitive processing skills. American Journal of Pharmaceutical Education, 59, 228–235.

    Google Scholar 

  • Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors’ perspectives. Advances in Health Sciences Education, 18(4), 559–571.

    Article  Google Scholar 

  • Byrne, A., Tweed, N., & Halligan, C. (2014). A pilot study of the mental workload of objective structured clinical examination examiners. Medical Education, 48(3), 262–267.

    Article  Google Scholar 

  • Cadwell, J., & Jenkins, J. (1986). Teachers’ judgments about their students: The effect of cognitive simplification strategies on the rating process. American Educational Research Journal, 23(3), 460–475.

    Article  Google Scholar 

  • Chambers, D. W. (2004). Do repeat clinical competency ratings stereotype students? Journal of Dental Education, 68(12), 1220–1227.

    Google Scholar 

  • Charmaz, K. (2005). Grounded theory in the 21st century: Applications for advancing social justice studies. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (Vol. 3, pp. 507–535). Thousand Oaks, California: Sage Publications.

    Google Scholar 

  • Cross, V., Hicks, C., & Barwell, F. (2001). Exploring the gap between evidence and judgement: Using video vignettes for practice-based assessment of physiotherapy undergraduates. Assessment and Evaluation in Higher Education, 26(3), 189–212.

    Article  Google Scholar 

  • De Neys, W. (2006). Dual processing in reasonning: Two systems but one reasoner. Psychological Science, 17(5), 428–433.

    Article  Google Scholar 

  • Downing, S. M., & Haladyna, T. M. (2004). Validity threats: Overcoming interference with proposed interpretations of assessment data. Medical Education, 38(3), 327–333.

    Article  Google Scholar 

  • Eva, K., Brooks, L., & Norman, G. (2001) Forward reasoning as a hallmark of expertise in medicine: Logical, psychological, phenomenological inconsistencies. In S.P. Shohov (Ed.), Advances in psychology research (Vol. 8). New York: Nova Science Publishers, Inc.

  • Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278.

    Article  Google Scholar 

  • Gallagher, P. (2010). The role of the assessor in the assessment of practice: An alternative view. Medical Teacher, 32(10), E413–E416.

    Article  Google Scholar 

  • Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the “black box”differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.

    Article  Google Scholar 

  • Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10), S1–S7.

    Article  Google Scholar 

  • Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  • Ginsburg, S., Regehr, G., & Mylopoulos, M. (2009). From behaviours to attributions: Further concerns regarding the evaluation of professionalism. Medical Education, 43(5), 414–425.

    Article  Google Scholar 

  • Gödert, H. W., Gamer, M., Rill, H.-G., & Vossel, G. (2005). Statement validity assessment: Inter-rater reliability of criteria-based content analysis in the mock-crime paradigm. Legal and Criminological Psychology, 10(2), 225–245. doi:10.1348/135532505X52680.

    Article  Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16, 151–165.

    Article  Google Scholar 

  • Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396.

    Article  Google Scholar 

  • Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12, 239–260.

    Article  Google Scholar 

  • Hogan, E. A. (1987). Effects of prior expectations on performance ratings: A longitudinal study. Academy of Management Journal, 30(2), 354–368.

    Article  Google Scholar 

  • Hoppmman, T. K. (2009). Examining the ‘point of rustration’. The think-aloud method applied to online search tasks. Quality and Quantity, 43, 211–224.

    Article  Google Scholar 

  • Howley, L. D., & Wilson, W. G. (2004). Direct observation of students during clerkship rotations: A multiyear descriptive study. Academic Medicine, 79(3), 276–280.

    Article  Google Scholar 

  • Judge, T. A., & Ferris, G. R. (1993). Social context of performance evaluation decisions. The Academy of Management Journal, 36(1), 80–105.

    Article  Google Scholar 

  • Kishor, N. (1994). Teachers’ judgements of students’ performance: Use of consensus, consistency and distinctiveness. Educational Psychology, 14(2), 233–247.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45, 1048–1060. doi:10.1111/j.1365-2923.2011.04025.x.

    Article  Google Scholar 

  • Kogan, J. R., Hess, B. J., Conforti, L. N., & Holmboe, E. S. (2010). What drives faculty ratings of residents’ clinical skills? The impact of faculty’s own clinical skills. Academic Medicine, 85(10), S25–S28.

    Article  Google Scholar 

  • Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.

    Article  Google Scholar 

  • Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.

    Article  Google Scholar 

  • Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9–10), 855–871.

    Article  Google Scholar 

  • Norman, G. (2009). Dual processing and diagnostic errors. Advances in Health Sciences Education, 14(1), 37–49.

    Article  Google Scholar 

  • Palmer, D. J., Stough, L. M., Burdenski, T. K, Jr, & Gonzales, M. (2005). Identifying teacher expertise: An examination of researchers’ decision making. Educational Psychologist, 40(1), 13–25.

    Article  Google Scholar 

  • Ross, K. G., Shafer, J. L., & Klein, G. (2006). Professional judgments and “naturalistic decision making”. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 403–420). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Schwandt, T. A. (2007). The SAGE dictionary of qualitative inquiry (3rd ed.). Thousand Oaks, CA: SAGE Publications.

    Book  Google Scholar 

  • Smith, E. V., & Kulikowich, J. M. (2004). An application of generalizability theory and many-faceted Rasch measurement using a complex problem-solving skills assessment. Educational and Psychological Measurement, 64(4), 617–639.

    Article  Google Scholar 

  • St-Onge, C., Chamberland, M., Lévesque, A., & Varpio, L. (2014). The role of the assessor: Exploring the clinical supervisor’s skill set. The Clinical Teacher, 11(3), 209–213.

    Article  Google Scholar 

  • Swanson, D. B., Norman, G. R., & Linn, R. L. (1995). Performance-based assessment: Lessons from the health professions. Educational Researcher, 24(5), 5–11.

    Article  Google Scholar 

  • Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18(2), 291–303.

    Article  Google Scholar 

  • Tavares, W., & Eva, K. (2014). Impact of rating demands on rater-based assessments of clinical competence. Education for General Practice, 25(6), 308–318.

    Google Scholar 

  • Turban, D. B., & Jones, A. P. (1988). Supervisor-subordinate similarity: Types, effects, and mechanisms. Journal of Applied Psychology, 73(2), 228–234.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1974). Judgement under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

    Article  Google Scholar 

  • Tweed, M., & Ingham, C. (2009). How do assessors make decisions on marking and standard setting for observed consultation? Focus on Health Professional Education, 10(3), 39–50.

    Google Scholar 

  • Van der Vleuten, C. (2000). Validity of final examinations in undergraduate medical training. British Medical Journal, 321, 1217–1219.

    Article  Google Scholar 

  • Van der Vleuten, C. P. M., Scherpbier, A. J. J. A., Dolmans, D. H. J. M., Schuwirth, L. W. T., Verwijnen, G. M., & Worlfhagen, H. A. P. (2000). Clerkship assessment assessed. Medical Teacher, 22(6), 592–600.

    Article  Google Scholar 

  • van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press.

    Google Scholar 

  • Voss, J. F., Tyler, S. W., & Yengo, L. A. (1983). Individual differences in the solving of social science problems. In R. Dillon & R. Schmeck (Eds.), Individual differences in cognition. New York: Academic Press.

    Google Scholar 

  • Waldman, D. A., & Avolio, B. J. (1991). Race effects in performance evaluation: Controlling for ability, education and experience. Journal of Applied Psychology, 76(6), 897–901.

    Article  Google Scholar 

  • Weiss, D. J., Shanteau, J., & Harries, P. (2006). People who judge people. Journal of Behavioral Decision Making, 19, 441–454.

    Article  Google Scholar 

  • Williams, R. G., Klamen, D. A., & McGaghie, W. (2003). Cognitive, social, and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.

    Article  Google Scholar 

  • Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.

    Article  Google Scholar 

  • Yaphe, J., & Street, S. (2003). How do examiners decide?: A qualitative study of the process of decision making in the oral examination component of the MRCGP examination. Medical Education, 37, 764–771.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2012). Effect of exposure to good vs poor medical trainee performance on attending physician rating of subsequent performances. JAMA: Journal of the American Medical Association, 308(21), 2226–2232.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325–341.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculté de médecine et des sciences de la santé, Université de Sherbrooke, Sherbrooke, QC, Canada

    Christina St-Onge, Martine Chamberland & Annie Lévesque

  2. Uniformed Services University of the Health Sciences, Bethesda, MD, USA

    Lara Varpio

Authors
  1. Christina St-Onge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martine Chamberland
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Annie Lévesque
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lara Varpio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christina St-Onge.

Additional information

Disclaimer Preliminary analysis of these data was presented at the American Education Research Association, April 2013. Although LV is employed by the Department of Defense the views expressed herein are those of the authors and do not necessarily reflect those of the Department of Defense or other federal agencies.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

St-Onge, C., Chamberland, M., Lévesque, A. et al. Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance. Adv in Health Sci Educ 21, 627–642 (2016). https://doi.org/10.1007/s10459-015-9656-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-015-9656-3

Keywords

Search

Navigation