Journal of General Internal Medicine

, 24:74

Effect of Rater Training on Reliability and Accuracy of Mini-CEX Scores: A Randomized, Controlled Trial

  • David A. Cook
  • Denise M. Dupras
  • Thomas J. Beckman
  • Kris G. Thomas
  • V. Shane Pankratz
Original Article

DOI: 10.1007/s11606-008-0842-3

Cite this article as:
Cook, D.A., Dupras, D.M., Beckman, T.J. et al. J GEN INTERN MED (2009) 24: 74. doi:10.1007/s11606-008-0842-3

Abstract

Background

Mini-CEX scores assess resident competence. Rater training might improve mini-CEX score interrater reliability, but evidence is lacking.

Objective

Evaluate a rater training workshop using interrater reliability and accuracy.

Design

Randomized trial (immediate versus delayed workshop) and single-group pre/post study (randomized groups combined).

Setting

Academic medical center.

Participants

Fifty-two internal medicine clinic preceptors (31 randomized and 21 additional workshop attendees).

Intervention

The workshop included rater error training, performance dimension training, behavioral observation training, and frame of reference training using lecture, video, and facilitated discussion. Delayed group received no intervention until after posttest.

Measurements

Mini-CEX ratings at baseline (just before workshop for workshop group), and four weeks later using videotaped resident–patient encounters; mini-CEX ratings of live resident–patient encounters one year preceding and one year following the workshop; rater confidence using mini-CEX.

Results

Among 31 randomized participants, interrater reliabilities in the delayed group (baseline intraclass correlation coefficient [ICC] 0.43, follow-up 0.53) and workshop group (baseline 0.40, follow-up 0.43) were not significantly different (p = 0.19). Mean ratings were similar at baseline (delayed 4.9 [95% confidence interval 4.6–5.2], workshop 4.8 [4.5–5.1]) and follow-up (delayed 5.4 [5.0–5.7], workshop 5.3 [5.0–5.6]; p = 0.88 for interaction). For the entire cohort, rater confidence (1 = not confident, 6 = very confident) improved from mean (SD) 3.8 (1.4) to 4.4 (1.0), p = 0.018. Interrater reliability for ratings of live encounters (entire cohort) was higher after the workshop (ICC 0.34) than before (ICC 0.18) but the standard error of measurement was similar for both periods.

Conclusions

Rater training did not improve interrater reliability or accuracy of mini-CEX scores.

Clinical trials registration

clinicaltrials.gov identifier NCT00667940

KEY WORDS

medical educationfaculty developmentrater trainingclinical competenceassessmentrandomized trial

Supplementary material

11606_2008_842_MOESM1_ESM.doc (128 kb)
AppendixCook et al, CEX Rater Training (DOC 129kb)

Copyright information

© Society of General Internal Medicine 2008

Authors and Affiliations

  • David A. Cook
    • 1
    • 2
  • Denise M. Dupras
    • 3
  • Thomas J. Beckman
    • 1
    • 2
  • Kris G. Thomas
    • 3
  • V. Shane Pankratz
    • 4
  1. 1.Office of Education Research, College of MedicineMayo ClinicRochesterUSA
  2. 2.Division of General Internal Medicine, College of MedicineMayo ClinicRochesterUSA
  3. 3.Division of Primary Care Internal Medicine, College of MedicineMayo ClinicRochesterUSA
  4. 4.Division of Biostatistics, College of MedicineMayo ClinicRochesterUSA