Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX

Article

DOI: 10.1007/s10459-008-9147-x

Cite this article as:
Cook, D.A. & Beckman, T.J. Adv in Health Sci Educ (2009) 14: 655. doi:10.1007/s10459-008-9147-x

Abstract

Educators must often decide how many points to use in a rating scale. No studies have compared interrater reliability for different-length scales, and few have evaluated accuracy. This study sought to evaluate the interrater reliability and accuracy of mini-clinical evaluation exercise (mini-CEX) scores, comparing the traditional mini-CEX nine-point scale to a five-point scale. Methods: The authors conducted a validity study in an academic internal medicine residency program. Fifty-two program faculty participated. Participants rated videotaped resident-patient encounters using the mini-CEX with both a nine-point scale and a five-point scale. Some cases were scripted to reflect a specific level of competence (unsatisfactory, satisfactory, superior). Outcome measures included mini-CEX scores, accuracy (scores compared to scripted competence level), interrater reliability, and domain intercorrelation. Results: Interviewing, exam, counseling, and overall ratings varied significantly across levels of competence (P < .0001). Nine-point scale scores accurately classified competence more often (391/720 [54%] for overall ratings) than five-point scores (316/723 [44%], P < .0001). Interrater reliability was similar for scores from the nine- and five-point scales (0.43 and 0.40, respectively, for overall ratings). With the exception of correlation between exam and counseling scores using the five-point scale (r = 0.38, P = .13), score correlations among all domain combinations were high (r = 0.46–0.89) and statistically significant (P ≤ .015) for both scales. Conclusions: Mini-CEX scores demonstrated modest interrater reliability and accuracy. Although interrater reliability is similar for nine- and five-point scales, nine-point scales appear to provide more accurate scores. This has implications for many educational assessments.

Keywords

Medical education Educational measurement Clinical competence Assessment Reproducibility of results Psychometrics Interrater reliability Accuracy 

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  1. 1.Division of General Internal Medicine and Office of Education ResearchMayo Clinic College of MedicineRochesterUSA

Personalised recommendations