Skip to main content
SpringerLink
Log in

Effectiveness of a focused educational intervention on resident evaluations from faculty

A randomized controlled trial

  • Original Articles
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

OBJECTIVE: To improve the quality and specificity of written evaluations by faculty attendings of internal medicine residents during inpatient rotations.

DESIGN: Prospective randomized controlled trial.

SETTING: Four hospitals: tertiary care university hospital, Veterans’ Administration hospital, and two community hospitals.

PARTICIPANTS: Eighty-eight faculty and 157 residents from categorical and primary-care internal medicine residency training programs rotating on inpatient general medicine teams.

INTERVENTION: Focused 20-minute educational session on evaluation and feedback, accompanied by 3 by 5 reminder card and diary, given to faculty at the start of their attending month.

MEASUREMENTS AND MAIN RESULTS: Primary outcomes: 1) number of written comments from faculty specific to unique, preselected dimensions of competence; 2) number of written comments from faculty describing a specific resident behavior or providing a recommendation; and 3) resident Likert-scale ratings of the quantity and effect of feedback received from faculty. Faculty in the intervention group provided more written comments specific to defined dimensions of competence, a median of three comments per evaluation form versus two in the control group, but when adjusted for clustering by faculty, the difference was not statistically significant (P=.09). Regarding feedback, residents in the intervention group rated the quantity significantly higher (P=.04) and were significantly more likely to make changes in clinical management of patients than residents in the control group (P=.04).

CONCLUSIONS: A brief, focused educational intervention delivered to faculty prior to the start of a ward rotation appears to have a modest effect on faculty behavior for written evaluations and promoted higher quality feedback given to house staff.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Eisenberg JM. Evaluating internists’ clinical competence. J Gen Intern Med. 1989;4:139–43.

    Article  PubMed  CAS  Google Scholar 

  2. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(suppl):63–7.

    Article  Google Scholar 

  3. American Board of Internal Medicine. Project Professionalism. Philadelphia: American Board of Internal Medicine; 1994.

    Google Scholar 

  4. Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. J Gen Intern Med. 1998;13:317–23.

    Article  PubMed  CAS  Google Scholar 

  5. American Board of Internal Medicine. Attending Physicians: Your Role in Evaluating Residents. Philadelphia: American Board of Internal Medicine; 1995.

    Google Scholar 

  6. American Board of Internal Medicine. Guide to the Evaluation of Residents in Internal Medicine 1990–92. Philadelphia: American Board of Internal Medicine; 1992.

    Google Scholar 

  7. Gray JD. Globa rating scales in residency education. Acad Med. 1996;71:S55–63.

    Article  PubMed  CAS  Google Scholar 

  8. Thompson WG, Lipkin M, Gilbert DA, Guzzo RA, Roberson L. Evaluating evaluation: assessment of the American Board of Internal Medicine resident evaluation form. J Gen Intern Med. 1990;5:214–7.

    Article  PubMed  CAS  Google Scholar 

  9. Haber RJ, Avins AL. Do ratings on the American Board of Internal Medicine resident evaluation form detect differences in clinical competence? J Gen Intern Med. 1994;9:140–5.

    Article  PubMed  CAS  Google Scholar 

  10. Borman WC. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. J Appl Psychol. 1975;60:556–60.

    Article  Google Scholar 

  11. Ende J. Feedback in clinical medical education. JAMA. 1983;250:777–81.

    Article  PubMed  CAS  Google Scholar 

  12. Ende J. The evaluation product: putting it to use. In: Lloyd JS, Langley DG, eds. How to Evaluate Residents. Chicago: American Board of Medical Specialties; 1986:99–116.

    Google Scholar 

  13. Weinholtz D. Teaching during attending rounds: a manual for attending physicians. Des Moines, Iowa: University of Iowa; 1987.

    Google Scholar 

  14. Skeff K, Stratos G. Feedback. From the Standford Faculty Development Program, Clinical Teaching. Standford, Calif: Stanford University; 1996.

    Google Scholar 

  15. Neufeld V, Norman GR, eds. Assessing Clinical Competence. New York: Springer; 1985.

    Google Scholar 

  16. Huber PJ. The behavior of maximum likelihood estimates under non-standard conditions. Proceedings of the Fifth Berkeley Symposium in Mathematical Probability. Berkeley, Calif: University of California Press; 1967.

    Google Scholar 

  17. White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica. 1980;48:817–30.

    Article  Google Scholar 

  18. Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in Internal Medicine: a review. Ann Intern Med. 1998;129:42–8.

    PubMed  CAS  Google Scholar 

  19. Veet LL, Shea JA, Ende J. Our continuing interest in manuscripts about education. J Gen Intern Med. 1997;12:583–5.

    Article  PubMed  CAS  Google Scholar 

  20. Education Group for Guidelines on Evaluation. Guideline for evaluating papers on educational interventions. BMJ. 1999;318:1265–7.

    Google Scholar 

  21. Davis JK, Inamdar S, Stone RK. Interrater agreement and predictive validity of faculty ratings of pediatric residents. J Med Educ. 1986;61:901–5.

    PubMed  CAS  Google Scholar 

  22. Maxim BR, Dielman TE. Dimensionality, internal consistency, and interrater reliability of clinical performance ratings. Med Educ. 1996;21:130–7.

    Article  Google Scholar 

  23. Carline JD, Paauw DS, Thiede KW, Ramsey PG. Factors affecting the reliability of ratings of students’ clinical skills in a medicine clerkship. J Gen Intern Med. 1992;7:506–10.

    Article  PubMed  CAS  Google Scholar 

  24. Norcini JJ. Standards and reliability in evaluation: when rules of thumb don’t apply. Acad Med. 1999;74:1088–90.

    Article  PubMed  Google Scholar 

  25. Kassirer JP. Performance results on the 1995 Internal Medicine certifying examination. ABIM News Update. 1995;Fall/Winter:2–4.

  26. Noel GL, Herbers JE, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do Internal Medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992;117:757–65.

    PubMed  CAS  Google Scholar 

  27. Herbers JE, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4:202–8.

    Article  PubMed  Google Scholar 

  28. Kroboth FJ, Hanusa BH, Parker S, et al. The inter-rater reliability and internal consistency of a clinical examination exercise. J Gen Intern Med. 1992;7:174–9.

    Article  PubMed  CAS  Google Scholar 

  29. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123:795–9.

    PubMed  CAS  Google Scholar 

  30. Kalet A, Earp J, Kowlowitz V. How well do the faculty evaluate the interviewing skills of medical students? J Gen Intern Med. 1992;7:499–505.

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Received from the Yale University Primary Care Internal Medicine Residency Program, New Haven, Conn

    Eric S. Holmboe MD (Director), Leslie A. Galaty MAT & Stephen Huot MD, PhD

  2. Johns Hopkins General Internal Medicine Residency Program, Johns Hopkins Bayview Medical Center, Baltimore, MD

    Nicholas H. Fiebach MD, MPH

Authors
  1. Eric S. Holmboe MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas H. Fiebach MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leslie A. Galaty MAT
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephen Huot MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric S. Holmboe MD.

Additional information

Portions of this work were previously presented at the 22nd Annual Meeting of the Society of General Internal Medicine, San Francisco, Calif, April 29–May 1, 1999.

Dr. Holmboe was a Robert Wood Johnson Clinical Scholar during completion of this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Holmboe, E.S., Fiebach, N.H., Galaty, L.A. et al. Effectiveness of a focused educational intervention on resident evaluations from faculty. J GEN INTERN MED 16, 427–434 (2001). https://doi.org/10.1046/j.1525-1497.2001.016007427.x

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1046/j.1525-1497.2001.016007427.x

Key Words

Search

Navigation