Group Assessments of Resident Physicians Improve Reliability and Decrease Halo Error
Purchase on Springer.com
$39.95 / €34.95 / £29.95*
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.
Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.
We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.
This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.
Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.
Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785–0.866 vs. 0.749, 0.686–0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.
This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
- ACGME. Program Director Guide to the Common Requirements. 2011 [cited 2011 January 13]; Available from: http://www.acgme.org/acWebsite/home/home.asp.
- Chaudhry SI, Holmboe E, Beasley BW. The state of evaluation in internal medicine residency. J Gen Intern Med. 2008;23(7):1010–5. CrossRef
- Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387–96. CrossRef
- Gray JD. Global rating scales in residency education. Acad Med. 1996;71(1 Suppl):S55–63. CrossRef
- Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings' post-rotation performance ratings detect residents' clinical performance deficiencies? Acad Med. 2004;79(5):453–7. CrossRef
- Levine HG, Yunker R, Bee D. Pediatric resident performance. The reliability and validity of rating forms. Eval Health Prof. 1986;9(1):62–74. CrossRef
- Marienfeld RD, Reid JC. Six-year documentation of the easy grader in the medical clerkship setting. J Med Educ. 1984;59(7):589–91.
- Ryan JG, Mandel FS, Sama A, Ward MF. Reliability of faculty clinical evaluations of non-emergency medicine residents during emergency department rotations. Acad Emerg Med. 1996;3(12):1124–30. CrossRef
- Thorndike EL. A constant error in psychological ratings. J Appl Psychol. 1920;4:25–9. CrossRef
- McKinstry BH, Cameron HS, Elton RA, Riley SC. Leniency and halo effects in marking undergraduate short research projects. BMC Med Educ. 2004;4(1):28. CrossRef
- Hemmer PA, Pangaro L. The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med. 1997;72(7):641–3. CrossRef
- Hemmer PA, Grau T, Pangaro LN. Assessing the effectiveness of combining evaluation methods for the early identification of students with inadequate knowledge during a clerkship. Med Teach. 2001;23(6):580–4. CrossRef
- Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students' professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75(2):167–73. CrossRef
- Albritton TA, Fincher RM, Work JA. Group evaluation of student performance in a clerkship. Acad Med. 1996;71(5):551–2. CrossRef
- Beckman TJ, Mandrekar JN. The interpersonal, cognitive and efficiency domains of clinical teaching: construct validity of a multi-dimensional scale. Med Educ. 2005;39(12):1221–9. CrossRef
- Beckman TJ, Mandrekar JN, Engstler GJ, Ficalora RD. Determining reliability of clinical assessment scores in real time. Teach Learn Med. 2009;21(3):188–94. CrossRef
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. CrossRef
- Risucci DA, Lutsky L, Rosati RJ, Tortolani AJ. Reliability and accuracy of resident evaluations of surgical faculty. Eval Health Prof. 1992;15(3):313–24. CrossRef
- Jacobs R, Kozlowski SW. A closer look at halo error in performance ratings. Academy of Management Journal. 1985;28(1):pp.
- Murphy KR, Jako RA, Anhalt RL. Nature and consequences of halo error: a critical analysis. J Appl Psychol. 1993;78(2):218–25. CrossRef
- Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med. 2004;79(4):343–6. CrossRef
- Iglehart JK. Revisiting duty-hour limits–IOM recommendations for patient safety and resident education. N Engl J Med. 2008;359(25):2633–5. CrossRef
- Holmboe ES, Rodak W, Mills G, McFarlane MJ, Schultz HJ. Outcomes-based evaluation in resident education: creating systems and structured portfolios. Am J Med. 2006;119(8):708–14. CrossRef
- Silber CG, Nasca TJ, Paskin DL, Eiger G, Robeson M, Veloski JJ. Do global rating forms enable program directors to assess the ACGME competencies? Acad Med. 2004;79(6):549–56. CrossRef
- Holmboe E, Hawkins RE. Practical Guide to the Evaluation of Clinical Competence. Philadelphia: Mosby; 2008.
- Speer AJ, Solomon DJ, Ainsworth MA. An innovative evaluation method in an internal medicine clerkship. Acad Med. 1996;71(1 Suppl):S76–8. CrossRef
- Hemmer PA, Pangaro L. Using formal evaluation sessions for case-based faculty development during clinical clerkships. Acad Med. 2000;75(12):1216–21. CrossRef
- Battistone MJ, Milne C, Sande MA, Pangaro LN, Hemmer PA, Shomaker TS. The feasibility and acceptability of implementing formal evaluation sessions and using descriptive vocabulary to assess student performance on a clinical clerkship. Teach Learn Med. 2002;14(1):5–10. CrossRef
- Williams RG, Schwind CJ, Dunnington GL, Fortune J, Rogers D, Boehler M. The effects of group dynamics on resident progress committee deliberations. Teach Learn Med. 2005;17(2):96–100. CrossRef
- Kisiel JB, Bundrick JB, Beckman TJ. Resident physicians' perspectives on effective outpatient teaching: a qualitative study. Adv Health Sci Educ Theory Pract. 2010;15(3):357–68. CrossRef
- Group Assessments of Resident Physicians Improve Reliability and Decrease Halo Error
Journal of General Internal Medicine
Volume 26, Issue 7 , pp 759-764
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links
- graduate medical education
- professional competence
- educational measurement
- formal evaluation sessions
- Industry Sectors
- Author Affiliations
- 1. Division of Primary Care Internal Medicine, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
- 2. Division of General Internal Medicine, Section of Medical Education, Mayo Clinic, Rochester, MN, USA
- 3. Department of Biomedical Statistics and Informatics, Rochester, MN, USA