Skip to main content

Advertisement

SpringerLink
Log in

Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education

  • Education (G Badalato, Section Editor)
  • Published:
Current Urology Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees.

Recent Findings

In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education.

Summary

No one “best approach” exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Academic medicine : Journal of the Association of American Medical Colleges. 2012;87(10):1401–7. https://doi.org/10.1097/ACM.0b013e3182677805.

    Article  Google Scholar 

  2. Merriam-Webster.com Dictionary: Merriam-Webster Evaluation.

  3. Gibbs T, Brigden D, Hellenberg D. Assessment and evaluation in medical education. S Afr Fam Pract. 2006;48(1):5–7. https://doi.org/10.1080/20786204.2006.10873311.

    Article  Google Scholar 

  4. Guyot WM. Summative and formative evaluation. The Journal of Business Education. 1978;54(3):127–9. https://doi.org/10.1080/00219444.1978.10534702.

    Article  Google Scholar 

  5. Merriam-Webster.com. Dictionary: Merriam-Webster. Feedback.

  6. Lefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015;4(6):284–99. https://doi.org/10.1007/s40037-015-0231-7.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Carr BM, O’Neil A, Lohse C, Heller S, Colletti JE. Bridging the gap to effective feedback in residency training: perceptions of trainees and teachers. BMC Med Educ. 2018;18(1):225. https://doi.org/10.1186/s12909-018-1333-9.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Holmboe ES, Edgar L, Hamstra S. The Milestone Guidebook2016.

  9. Education ACfGM. What we do. 2020. .

  10. •• Education ACfGM. Surgery Milestones2019. Defines current surgery Milestones from ACGME. Explains structure and elaborates on how programs should utilize the Milestones.

  11. Accreditation Council for Graduate Medical Education ABoU. The Urology Milestone Project 2016.

  12. Bartlett KW, Whicker SA, Bookman J, Narayan AP, Staples BB, Hering H, et al. Milestone-based assessments are superior to Likert-type assessments in illustrating trainee progression. Journal of graduate medical education. 2015;7(1):75–80. https://doi.org/10.4300/JGME-D-14-00389.1.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Beeson MS, Holmboe ES, Korte RC, Nasca TJ, Brigham T, Russ CM, et al. Initial validity analysis of the emergency medicine milestones. Acad Emerg Med. 2015;22(7):838–44. https://doi.org/10.1111/acem.12697.

    Article  PubMed  Google Scholar 

  14. • Goldman RH, Tuomala RE, Bengtson JM, Stagg AR. How effective are new milestones assessments at demonstrating resident growth? 1 year of data. J Surg Educ. 2017;74(1):68–73. https://doi.org/10.1016/j.jsurg.2016.06.009Study explores the development of assessment tools based on the milestone system. Authors were able to develop an assessment tool that demonstrated growth of residents overtime and captured differences in skills that are quantifiable between PGY class.

    Article  PubMed  Google Scholar 

  15. Gardner AK, AbdelFattah K. Comparison of simulation-based assessments and faculty ratings for general surgery resident milestone evaluation: are they telling the same story? Am J Surg. 2017;214(3):547–53. https://doi.org/10.1016/j.amjsurg.2016.07.025.

    Article  PubMed  Google Scholar 

  16. • Tekian A, Borhani M, Tilton S, Abasolo E, Park YS. What do quantitative ratings and qualitative comments tell us about general surgery residents’ progress toward independent practice? Evidence from a 5-year longitudinal cohort. Am J Surg. 2019;217(2):288–95. https://doi.org/10.1016/j.amjsurg.2018.09.031Quantitative evaluations with area for comments are commonly used for resident feedback and evaluation; this long-term study demonstrates how qualitative feedback correlated with quantitative evaluation of residents.

    Article  PubMed  Google Scholar 

  17. •• Clinial competency committees: a guidebook for programs. Accredidation Council for Graduate Medical Education2017 September 23, 2017. Most current ACGME resource on Clinical Competency Committees which are required for accrediation. Explains that this committee is responsible for evaluating each residents progression using the Milestone system. However, it also highlights that each program is responsible for how assessment tools are administered, giving programs more freedom to utilize new techniques.

  18. Bosse HM, Mohr J, Buss B, Krautter M, Weyrich P, Herzog W, et al. The benefit of repetitive skills training and frequency of expert feedback in the early acquisition of procedural skills. BMC Med Educ. 2015;15:22. https://doi.org/10.1186/s12909-015-0286-5.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H, et al. Facilitated reflective performance feedback: developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Academic medicine : Journal of the Association of American Medical Colleges. 2015;90(12):1698–706. https://doi.org/10.1097/acm.0000000000000809.

    Article  Google Scholar 

  20. Meier AH, Gruessner A, Cooney RN. Using the ACGME milestones for resident self-evaluation and faculty engagement. J Surg Educ. 2016;73(6):e150–e7. https://doi.org/10.1016/j.jsurg.2016.09.001.

    Article  PubMed  Google Scholar 

  21. Kuo LE, Hoffman RL, Morris JB, Williams NN, Malachesky M, Huth LE, et al. A milestone-based evaluation system-the cure for grade inflation? J Surg Educ. 2015;72(6):e218–25. https://doi.org/10.1016/j.jsurg.2015.09.012.

    Article  PubMed  Google Scholar 

  22. • Sebesta EM, Cooper KL, Badalato GM. Program director perceptions of usefulness of the accreditation council for graduate medical education milestones system for urology resident evaluation. Urology. 2019;124:28–32. https://doi.org/10.1016/j.urology.2018.10.042This study evaluated urology Program Directors’ opinions on the ACGME Milestones. 48% of the sample felt that Milestones were somewhat or very unhelpful in evaluation; 58% felt that evaluations were uncorelated with in-service exam scores. This study highlights that the Milestone system is still in its infancy and that program directors perception of its utilization in urology demonstrates room for improvement.

    Article  PubMed  Google Scholar 

  23. Ray JJ, Sznol JA, Teisch LF, Meizoso JP, Allen CJ, Namias N, et al. Association between American Board of Surgery in-training examination scores and resident performance. JAMA surgery. 2016;151(1):26–31. https://doi.org/10.1001/jamasurg.2015.3088.

    Article  PubMed  Google Scholar 

  24. • Edgar L, Roberts S, Holmboe E. Milestones 2.0: A Step Forward. J Graduate Med Educ. 2018;10(3):367–9. https://doi.org/10.4300/JGME-D-18-00372.1Highlights the development of Milestones 2.0 project to better improve the feedback and evaluation of reidents and add clarity to educational goals.

    Article  Google Scholar 

  25. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: where to begin? Surgery. 2004;135(1):6–20. https://doi.org/10.1016/s0039-6060(03)00154-5.

    Article  CAS  PubMed  Google Scholar 

  26. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–8. https://doi.org/10.1046/j.1365-2168.1997.02502.x.

    Article  CAS  PubMed  Google Scholar 

  27. Williams RG, Verhulst S, Mellinger JD, Dunnington GL. Is a single-item operative performance rating sufficient? J Surg Educ. 2015;72(6):e212–7. https://doi.org/10.1016/j.jsurg.2015.05.002.

    Article  PubMed  Google Scholar 

  28. Rekman J, Hamstra SJ, Dudek N, Wood T, Seabrook C, Gofton W. A new instrument for assessing resident competence in surgical clinic: the Ottawa Clinic Assessment Tool. J Surg Educ. 2016;73(4):575–82. https://doi.org/10.1016/j.jsurg.2016.02.003.

    Article  PubMed  Google Scholar 

  29. Stausmire JM, Cashen CP, Myerholtz L, Buderer N. Measuring general surgery residents’ communication skills from the patient’s perspective using the Communication Assessment Tool (CAT). J Surg Educ. 2015;72(1):108–16. https://doi.org/10.1016/j.jsurg.2014.06.021.

    Article  PubMed  Google Scholar 

  30. Falcone JL, Claxton RN, Marshall GT. Communication skills training in surgical residency: a needs assessment and metacognition analysis of a difficult conversation objective structured clinical examination. J Surg Educ. 2014;71(3):309–15. https://doi.org/10.1016/j.jsurg.2013.09.020.

    Article  PubMed  Google Scholar 

  31. Joshi MK, Singh T, Badyal DK. Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students. J Postgrad Med. 2017;63(2):100–5. https://doi.org/10.4103/0022-3859.201411.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Lynch T, Risucci D, Blair P, Sachdeva A. Implementation of the American College of Surgeons (ACS) OSCE to assess the knowledge and competency of entering surgical residents. J Surg Educ - J SURG EDUC. 2008;65:27. https://doi.org/10.1016/j.jsurg.2008.01.045.

    Article  Google Scholar 

  33. Gee DW, Phitayakorn R, Khatri A, Butler K, Mullen JT, Petrusa ER. A pilot study to gauge effectiveness of standardized patient scenarios in assessing general surgery milestones. J Surg Educ. 2016;73(6):e1–8. https://doi.org/10.1016/j.jsurg.2016.08.012.

    Article  PubMed  Google Scholar 

  34. Hochberg MS, Berman RS, Kalet AL, Zabar SR, Gillespie C, Pachter HL, et al. The professionalism curriculum as a cultural change agent in surgical residency education. Am J Surg. 2012;203(1):14–20. https://doi.org/10.1016/j.amjsurg.2011.05.007.

    Article  PubMed  Google Scholar 

  35. Posner G, Nakajima A. Assessing residents’ communication skills: disclosure of an adverse event to a standardized patient. J Obstet Gynaecol Can. 2011;33(3):262–8. https://doi.org/10.1016/s1701-2163(16)34828-9.

    Article  PubMed  Google Scholar 

  36. Larkin AC, Cahan MA, Whalen G, Hatem D, Starr S, Haley H-L, et al. Human Emotion and Response in Surgery (HEARS): a simulation-based curriculum for communication skills, systems-based practice, and professionalism in surgical residency training. J Am Coll Surg. 2010;211(2):285–92. https://doi.org/10.1016/j.jamcollsurg.2010.04.004.

    Article  PubMed  Google Scholar 

  37. Hochberg MS, Kalet A, Zabar S, Kachur E, Gillespie C, Berman RS. Can professionalism be taught? Encouraging evidence. Am J Surg. 2010;199(1):86–93. https://doi.org/10.1016/j.amjsurg.2009.10.002.

    Article  PubMed  Google Scholar 

  38. Nicksa GA, Anderson C, Fidler R, Stewart L. Innovative approach using interprofessional simulation to educate surgical residents in technical and nontechnical skills in high-risk clinical scenarios. JAMA surgery. 2015;150(3):201–7. https://doi.org/10.1001/jamasurg.2014.2235.

    Article  PubMed  Google Scholar 

  39. • Kim SSY, Blankstein U, Ordon M, Pace KT, RJD H, Lee JY, et al. Evaluation of optimal timing of expert feedback in a simulated flexible ureteroscopy course. J Endourol. 2019;33(6):463–7. https://doi.org/10.1089/end.2018.0732Study examining simulation-based training on resident performance of ureteroscopy and optimal timing of feedback. Demonstrates how simulation can improve mean performance scores and that the majority of residents (73%) preferred early feedback from simulation as opposed to delayed feedbak from session.

    Article  PubMed  Google Scholar 

  40. Setia S, Feng C, Coogan C, Vourganti S, Abern M. Urology residents’ experience with simulation: initial evaluation of MRI/US fusion biopsy workshop. Urology. 2019;134:51–5. https://doi.org/10.1016/j.urology.2019.09.004.

    Article  PubMed  Google Scholar 

  41. Kneebone R. Simulation in surgical training: educational issues and practical implications. Med Educ. 2003;37(3):267–77. https://doi.org/10.1046/j.1365-2923.2003.01440.x.

    Article  PubMed  Google Scholar 

  42. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306(9):978–88. https://doi.org/10.1001/jama.2011.1234.

    Article  CAS  PubMed  Google Scholar 

  43. Pena G, Altree M, Babidge W, Field J, Hewett P, Maddern G. Mobile simulation unit: taking simulation to the surgical trainee. ANZ J Surg. 2015;85(5):339–43. https://doi.org/10.1111/ans.12549.

    Article  PubMed  Google Scholar 

  44. Schwartz SI, Galante J, Kaji A, Dolich M, Easter D, Melcher ML, et al. Effect of the 16-hour work limit on general surgery intern operative case volume: a multi-institutional study. JAMA surgery. 2013;148(9):829–33. https://doi.org/10.1001/jamasurg.2013.2677.

    Article  PubMed  Google Scholar 

  45. Kairys JC, McGuire K, Crawford AG, Yeo CJ. Cumulative operative experience is decreasing during general surgery residency: a worrisome trend for surgical trainees? J Am Coll Surg. 2008;206(5):804–13. https://doi.org/10.1016/j.jamcollsurg.2007.12.055.

    Article  PubMed  Google Scholar 

  46. Snyder RA, Phillips SE, Terhune KP. Impact of implementation of a pediatric surgery fellowship on general surgery resident operative volume. Journal of surgical education. 2012;69(6):753–8. https://doi.org/10.1016/j.jsurg.2012.03.007.

    Article  PubMed  Google Scholar 

  47. •• Hu YY, Mazer LM, Yule SJ, Arriaga AF, Greenberg CC, Lipsitz SR, et al. Complementing operating room teaching with video-based coaching. JAMA Surg. 2017;152(4):318–25. https://doi.org/10.1001/jamasurg.2016.4619Development of new approaches to evaluation and feedback is important for improvement of medical education. This study demonstrates the development of novel video-based coaching in the OR and its utility in providing residents and faculty opportunity to discuss more complex topics during feedback sessions.

    Article  PubMed  Google Scholar 

  48. Soucisse ML, Boulva K, Sideris L, Drolet P, Morin M, Dubé P. Video coaching as an efficient teaching method for surgical residents-a randomized controlled trial. Journal of surgical education. 2017;74(2):365–71. https://doi.org/10.1016/j.jsurg.2016.09.002.

    Article  PubMed  Google Scholar 

  49. Hu Y-Y, Peyre SE, Arriaga AF, Osteen RT, Corso KA, Weiser TG, et al. Postgame analysis: using video-based coaching for continuous professional development. J Am Coll Surg. 2012;214(1):115–24. https://doi.org/10.1016/j.jamcollsurg.2011.10.009.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Mazer LM, Hu YY, Arriaga AF, Greenberg CC, Lipsitz SR, Gawande AA, et al. Evaluating surgical coaching: a mixed methods approach reveals more than surveys alone. J Surg Educ. 2018;75(6):1520–5. https://doi.org/10.1016/j.jsurg.2018.03.009.

    Article  PubMed  Google Scholar 

  51. Moore MD, Abelson JS, O’Mahoney P, Bagautdinov I, Yeo H, Watkins AC. Using GoPro to give video-assisted operative feedback for surgery residents: a feasibility and utility assessment. J Surg Educ. 2018;75(2):497–502. https://doi.org/10.1016/j.jsurg.2017.07.024.

    Article  PubMed  Google Scholar 

  52. Bohnen JD, George BC, Williams RG, Schuller MC, DaRosa DA, Torbeck L, et al. The feasibility of real-time intraoperative performance assessment with SIMPL (System for Improving and Measuring Procedural Learning): early experience from a multi-institutional trial. J Surg Educ. 2016;73(6):e118–e30. https://doi.org/10.1016/j.jsurg.2016.08.010.

    Article  PubMed  Google Scholar 

  53. Williams RG, Verhulst S, Colliver JA, Sanfey H, Chen X, Dunnington GL. A template for reliable assessment of resident operative performance: assessment intervals, numbers of cases and raters. Surgery. 2012;152(4):517–24; discussion 24-7. https://doi.org/10.1016/j.surg.2012.07.004.

    Article  PubMed  Google Scholar 

  54. Eaton M, Scully R, Schuller M, Yang A, Smink D, Williams RG, et al. Value and barriers to use of the SIMPL tool for resident feedback. Journal of Surgical Education. 2019;76(3):620–7. https://doi.org/10.1016/j.jsurg.2019.01.012.

    Article  PubMed  Google Scholar 

  55. Harriman D, Singla R, Nguan C. The resident report card: a tool for operative feedback and evaluation of technical skills. J Surg Res. 2019;239:261–8. https://doi.org/10.1016/j.jss.2019.02.006.

    Article  PubMed  Google Scholar 

  56. • Hartranft TH, Yandle K, Graham T, Holden C, Chambers LW. Evaluating surgical residents quickly and easily against the milestones using electronic formative feedback. J Surg Educ. 2017;74(2):237–42. https://doi.org/10.1016/j.jsurg.2016.09.006This paper describes an institution’s implementation of web-based evaluations for surgical residents. This study demonstrates increase in resident and faculty overall satisfaction with web-based evaluation technique.

    Article  PubMed  Google Scholar 

  57. Green JM. An innovative, no-cost, evidence-based smartphone platform for resident evaluation. J Surg Educ. 2016;73(6):e14–e8. https://doi.org/10.1016/j.jsurg.2016.07.016.

    Article  PubMed  Google Scholar 

  58. Wagner JP, Chen DC, Donahue TR, Quach C, Hines OJ, Hiatt JR, et al. Assessment of resident operative performance using a real-time mobile web system: preparing for the milestone age. J Surg Educ. 2014;71(6):e41–6. https://doi.org/10.1016/j.jsurg.2014.06.008.

    Article  PubMed  Google Scholar 

  59. J H. The rise of crowdsourcing. 2006:1–5.

  60. Boyle E, Al-Akash M, Gallagher AG, Traynor O, Hill AD, Neary PC. Optimising surgical training: use of feedback to reduce errors during a simulated surgical procedure. Postgrad Med J. 2011;87(1030):524–8. https://doi.org/10.1136/pgmj.2010.109363.

    Article  PubMed  Google Scholar 

  61. •• Dai JC, Lendvay TS, Sorensen MD. Crowdsourcing in surgical skills acquisition: a developing technology in surgical education. J Graduate Med Educ. 2017;9(6):697–705. https://doi.org/10.4300/jgme-d-17-00322.1Systematic review of crowdsourcing and its utility in surgical training demonstrated that crowdsourced feedback was less expensive than feedback from expert surgeons and was more timely. This paper explores the valididity of crowd sourcings role in evaluation across multiple training levels and skill tasks. This paper provides evidence to support the use this technology in feedback and evaluation of surgical residents.

    Article  Google Scholar 

  62. Vernez SL, Huynh V, Osann K, Okhunov Z, Landman J, Clayman RV. C-SATS: assessing surgical skills among urology residency applicants. J Endourol. 2017;31(S1):S95–s100. https://doi.org/10.1089/end.2016.0569.

    Article  PubMed  Google Scholar 

  63. Aghdasi N, Bly R, White LW, Hannaford B, Moe K, Lendvay TS. Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. J Surg Res. 2015;196(2):302–6. https://doi.org/10.1016/j.jss.2015.03.018.

    Article  PubMed  PubMed Central  Google Scholar 

  64. White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS. Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol. 2015;29(11):1295–301. https://doi.org/10.1089/end.2015.0191.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Henry Ford Health System, 2799 W Grand Blvd, K9, Detroit, MI, 48202, USA

    Sara Q. Perkins, Ali Dabaja & Humphrey Atiemo

Authors
  1. Sara Q. Perkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Dabaja
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Humphrey Atiemo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Humphrey Atiemo.

Ethics declarations

Conflict of Interest

Sara Perkins, Humphrey Atiemo, and Ali Dabaja each declare no potential conflicts of interest.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Education

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Perkins, S.Q., Dabaja, A. & Atiemo, H. Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education. Curr Urol Rep 21, 36 (2020). https://doi.org/10.1007/s11934-020-00991-2

Download citation

  • Published:

  • DOI: https://doi.org/10.1007/s11934-020-00991-2

Keywords

Search

Navigation