Do ratings on the american board of internal medicine resident evaluation form detect differences in clinical competence?
- 92 Downloads
Objective: To determine whether raters using the American Board of Internal Medicine (ABIM) Resident Evaluation Form can detect differences among residents in clinical competence.
Design: Cross-sectional study.
Setting: Inpatient general medicine service in a university-affiliated public hospital.
Participants: University-based internal medicine (UCIM) residents (ABIM certifying examination pass rate, 91%; mean score, 95th percentile), community hospital-based internal medicine (CHIM) residents (ABIM examination pass rate, 68%; mean score, 42nd percentile), and residents from three university-based non-internal medicine (UC non-IM) programs all assigned to the same inpatient general medicine service over a three-year period. Four hundred eighty-nine evaluations of 110 postgraduate-year-one residents were analyzed.
Measurements and main results: Mean ratings for the UCIM residents were significantly higher than those for the CHIM or UC non-IM residents (analysis of variance [ANOVA], p<0.05). Variance was smallest for the UCIM residents (F test, p<0.01), and only the UCIM residents’ mean scores were in the “superior” range (7–9) in all evaluated categories. The mean ratings for the CHIM residents while at the university-affiliated hospital were not significantly different from the ratings of the same residents at their home hospital. The ratings for the CHIM residents at either site were significantly lower than those for the UCIM residents in all categories (ANOVA, p<0.05). Factor analysis revealed a single factor accounting for 76% of the variance among the ratings with all dimensions loading high on that factor (0.75–0.95), providing evidence for a “halo” effect. Mean interrater agreement over all variables was 0.87, indicating good consistency among raters.
Conclusions: Ratings on the ABIM Resident Evaluation Form detect global differences among residents in clinical competence in the expected direction based on type of training program and performance on the ABIM certification examination, but fail to differentiate among the nine evaluated dimensions of clinical care. This rating method may be valid for assessing overall clinical performance, but is less useful for providing feedback in specific areas to individual residents.
Key wordsclinical competence educational measurement evaluation studies internal medicine internship and residency ABIM Resident Evaluation Form
Unable to display preview. Download preview PDF.
- 1.The American Board of Internal Medicine. Guide to Evaluation of Residents in Internal Medicine 1988–1989. Portland, OR: Office of the President, American Board of Internal Medicine, 1988.Google Scholar
- 4.Program Director’s Report A. American Board of Internal Medicine, 1986–1991. In:Google Scholar
- 5.Snedecor G, Cochran W. Statistical Methods. 8th ed. Ames, IA: Iowa State University Press, 1989.Google Scholar
- 6.Sokal RR, Rohlf FJ. Biometry, 2nd ed. New York: W. H. Freeman, 1981;242–62.Google Scholar
- 7.Stevens J. Applied Multivariate Statistics for the Social Sciences. Hillsdale, NJ: Lawrence Erlbaum, 1986.Google Scholar
- 9.Neufeld VR An introduction to measurement properties. In: Neufeld VR, Normal GR (eds). Assessing Clinical Competence. Springer Series on Medical Education, vol. 7. New York: Springer Publishing, 1985;7:39–50.Google Scholar
- 11.Noricini JJ Jr, Webster GD, Grosso LJ, Blank LL, Benson JA Jr. Ratings of residents’ clinical competence and performance on certification examination. J Med Educ. 1987;62:457–62.Google Scholar
- 13.Streiner DL. Global rating scales. In: Neufeld VR, Norman GR (eds). Assessing Clinical Competence. Springer Series on Medical Education, vol. 7. New York: Springer Publishing, 1985;7:119–41.Google Scholar
- 24.Cranton PA, Dauphinee WD, McQueen MM, Smith LP. The reliability and validity of in-training evaluation reports in obstetrics and gynecology. Proceedings, 23rd Annual Conference on Research in Medical Education. Washington, DC: Association of American Medical Colleges, 1984;59–64.Google Scholar
- 26.Pierleoni RG, Dudding BA, Clark GM. An analysis of pediatric clerkship performance in a multicomponent evaluation system. Proceedings, 17th Annual Conference on Research in Medical Education. Washington, DC: Association of American Medical Colleges, 1978;31–6.Google Scholar