Abstract
The American Board of Internal Medicine suggests use of a standard form to rate residents on nine dimensions (such as clinical judgment and overall clinical competence) on a scale of 1 to 9. The authors examined the psychometric evidence for reliability and validity of 1,039 ratings of 85 residents by 135 attendings in a single internal medicine residency program. Of these ratings, 95.6% were from 6 to 9. Factor analysis revealed that high correlations among the nine dimensions (r ranged from 0.72 to 0.92) resulted from a single global factor accounting for 86% of the variance. The study also examined whether the form reliably distinguishes among residents scoring between 6 and 9. Agreement among attendings rating the same individual was weak (average reliability=0.64, by the method of James). The rating method fails to discriminate dimensions of clinical care and has low reliability for distinguishing among competent residents.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
The American Board of Internal Medicine. The role of the attending physician in evaluating residents. Philadelphia: American Board of Internal Medicine, 1987.
James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. J Applied Psychol. 1984;69:85–98.
Murphy KR, Balzer WK. Rater errors and rating accuracy. J Applied Psychol. 1989;74:619–24.
Bernardin HJ, Beatty RW. Performance appraisal: assessing human behavior at work. Boston: Kent Publishing, 1984.
Nunnally JC. Psychometric theory. New York: McGraw-Hill, 1967.
Levine HG, McGuire CH. Rating habitual performance in graduate medical education. J Med Educ. 1971;46:306–11.
American Board of Internal Medicine. Clinical competence in internal medicine. Ann Intern Med. 1979;90:402–11.
Cranton PA, Dauphinee WD, McQueen MM, Smith LP. The reliability and validity of in-training evaluation reports in obstetrics and gynecology. Research in medical education: proceedings of the twenty-third annual conference, 59–64. Washington, DC, Association of American Medical Colleges.
Williams SV. Assessment of procedures used to substantiate clinical competence for ABIM certifying examination. Abstract presented to Robert Wood Johnson Clinical Scholars Program, 1977.
Banks CG, Roberson L. Performance appraisers as test developers. Acad Management Rev. 1985;10:128–42.
Wigton RS. Factors important in the evaluation of clinical performance of internal medicine residents. J Med Educ. 1980;55:206–8.
Quarrick EA, Sloop EW. A method for identifying the criteria of good performance in a medical clerkship program. J Med Educ. 1972;47:188–97.
Landy FJ, Farr J. Performance rating. Psychol Bull. 1980;87:72–107.
Allen MJ, Yen WM. Introduction to measurement theory. Belmont, CA: Wadsworth, 1979.
Stillman PL, Gillers MA. Clinical performance evaluation in medicine and law. In: Berk RA, ed. Performance assessment: methods and applications. Baltimore: Johns Hopkins University Press, 1986.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Thompson, W.G., Lipkin, M., Gilbert, D.A. et al. Evaluating evaluation. J Gen Intern Med 5, 214–217 (1990). https://doi.org/10.1007/BF02600537
Issue Date:
DOI: https://doi.org/10.1007/BF02600537