Skip to main content
SpringerLink
Log in

Building a System of Competency-Based Assessment

  • Chapter
  • First Online:
Graduate Medical Education in Psychiatry

  • 411 Accesses

Abstract

Competency-based assessment (CBA) for psychiatric graduate medical education must simultaneously optimize three goals: (1) Maximize learning through formative assessment (assessment for learning). (2) Enable robust and trustworthy high stakes decision-making (e.g., promotion or selection) through summative assessment (assessment of learning). (3) Support ongoing improvements in the curriculum. As such, the CBA system must promote both self-regulated growth and clinical competence as judged by a trustworthy process. The key to effectively doing so is situating these activities within a carefully designed system of assessment with several critical components: workplace-based assessment, ongoing faculty development, learning analytics, longitudinal coaching, and fit-for-purpose clinical competency committees.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet (London, England). 2010;376(9756):1923–58.

    Article  Google Scholar 

  2. Lucey CR. Medical education: part of the problem and part of the solution. JAMA Intern Med. 2013;173(17):1639–43.

    Article  PubMed  Google Scholar 

  3. Cooke M, Irby DM, O'Brien BC. Carnegie foundation for the advancement of teaching. In: Educating physicians : a call for reform of medical school and residency. 1st ed. San Francisco: Jossey-Bass; 2010.

    Google Scholar 

  4. Eden J, Berwick DM, Wilensky GR, Institute of Medicine (U.S.). Committee on the Governance and Financing of Graduate Medical Education. Graduate medical education that meets the nation's health needs. Washington, D.C.: The National Academies Press; 2014.

    Google Scholar 

  5. Dyrbye L, Shanafelt T. A narrative review on burnout experienced by medical students and residents. Med Educ. 2016;50(1):132–49.

    Article  PubMed  Google Scholar 

  6. Dyrbye LN, West CP, Satele D, et al. Burnout among U.S. medical students, residents, and early career physicians relative to the general U.S. population. Acad Med. 2014;89(3):443–51.

    Article  PubMed  Google Scholar 

  7. Young JQ, Sugarman R, Schwartz J, Thakker K, O'Sullivan PS. Exploring residents' experience of career development scholarship tracks: a qualitative case study using social cognitive career theory. Teach Learn Med. 2020. https://doi.org/10.1080/10401334.10402020.11751637. Online ahead of print .

  8. Corgan S, Ford Winkel A, Sugarman R, Young JQ. From burnout to wholehearted engagement: a qualitative exploration of psychiatry residents' experience of stress. Acad Med. 2021;96(5):709–17.

    Article  PubMed  Google Scholar 

  9. Young JQ, Schwartz J, Thakker K, O'Sullivan PS, Sugarman R. Where passion meets need: a longitudinal, self-directed program to help residents discover meaning and develop as scholars. Acad Psychiatry. 2020;44(4):455–60.

    Article  PubMed  Google Scholar 

  10. Osseo-Asare A, Balasuriya L, Huot SJ, et al. Minority resident physicians' views on the role of race/ethnicity in their training experiences in the workplace. JAMA Netw Open. 2018;1(5):e182723.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Thibault GE. Reforming health professions education will require culture change and closer ties between classroom and practice. Health Affairs (Project Hope). 2013;32(11):1928–32.

    Article  Google Scholar 

  12. Skochelak SE. A decade of reports calling for change in medical education: what do they say? Acad Med. 2010;85(9 Suppl):S26–33.

    Article  PubMed  Google Scholar 

  13. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77(5):361–7.

    Article  PubMed  Google Scholar 

  14. Leach DC. A model for GME: shifting from process to outcomes. A progress report from the accreditation council for graduate medical education. Med Educ. 2004;38(1):12–4.

    Article  PubMed  Google Scholar 

  15. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45.

    Article  PubMed  Google Scholar 

  16. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. 2012;366(11):1051–6.

    Article  CAS  PubMed  Google Scholar 

  17. Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29(7):642–7.

    Article  PubMed  Google Scholar 

  18. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631–7.

    Article  PubMed  Google Scholar 

  19. Dauphinee WD. Building a core competency assessment program for all stakeholders: the design and building of sailing ships can inform core competency frameworks. Adv Health Sci Educ Theory Pract. 2020;25(1):189–93.

    Article  PubMed  Google Scholar 

  20. Arbuckle MR, Weinberg M, Cabaniss DL, et al. Training psychiatry residents in quality improvement: an integrated, year-long curriculum. Acad Psychiatry. 2013;37(1):42–5.

    Article  PubMed  Google Scholar 

  21. Arbuckle MR, Travis MJ, Eisen J, et al. Transforming psychiatry from the classroom to the clinic: lessons from the National Neuroscience Curriculum Initiative. Acad Psychiatry. 2020;44(1):29–36.

    Article  PubMed  Google Scholar 

  22. Patel E, Muthusamy V, Young JQ. Delivering on the promise of CLER: a patient safety rotation that aligns resident education with hospital processes. Acad Med. 2018;93(6):898–903.

    Article  PubMed  Google Scholar 

  23. Sunderji N, Ion A, Huynh D, Benassi P, Ghavam-Rassoul A, Carvalhal A. Advancing integrated care through psychiatric workforce development: a systematic review of educational interventions to train psychiatrists in integrated care. Can J Psychiatr. 2018;63(8):513–25.

    Article  Google Scholar 

  24. Crawford A, Sunderji N, Lopez J, Soklaridis S. Defining competencies for the practice of telepsychiatry through an assessment of resident learning needs. BMC Med Educ. 2016;16:28.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Hilty D, Chan S, Torous J, Luo J, Boland R. A framework for competencies for the use of mobile technologies in psychiatry and medicine: scoping review. JMIR Mhealth Uhealth. 2020;8(2):e12229.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Shanafelt TD, Dyrbye LN, West CP. Addressing physician burnout: the way forward. JAMA. 2017;317(9):901–2.

    Article  PubMed  Google Scholar 

  27. Goldman ML, Bernstein CA, Konopasek L, Arbuckle M, Mayer LES. An intervention framework for institutions to meet new ACGME common program requirements for physician well-being. Acad Psychiatry. 2018;42(4):542–7.

    Article  PubMed  PubMed Central  Google Scholar 

  28. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45.

    Article  PubMed  Google Scholar 

  29. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Institute of Medicine. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001.

    Google Scholar 

  31. Niven DJ, Rubenfeld GD, Kramer AA, Stelfox HT. Effect of published scientific evidence on glycemic control in adult intensive care units. JAMA Intern Med. 2015;175(5):801–9.

    Article  PubMed  Google Scholar 

  32. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205–14.

    Article  PubMed  Google Scholar 

  33. ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 2007;82(6):542–7.

    Article  PubMed  Google Scholar 

  34. Malone K, Supri S. A critical time for medical education: the perils of competence-based reform of the curriculum. Adv Health Sci Educ Theory Pract. 2012;17(2):241–6.

    Article  PubMed  Google Scholar 

  35. Halman S, Dudek N, Wood T, et al. Direct observation of clinical skills feedback scale: development and validity evidence. Teach Learn Med. 2016;28(4):385–94.

    Article  PubMed  Google Scholar 

  36. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;341:c5064.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Schuwirth LW, Van der Vleuten CP. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–85.

    Article  PubMed  Google Scholar 

  38. Al-Eraky M, Marei H. A fresh look at Miller's pyramid: assessment at the 'Is' and 'Do' levels. Med Educ. 2016;50(12):1253–7.

    Article  PubMed  Google Scholar 

  39. Norcini J, Anderson MB, Bollela V, et al. Consensus framework for good assessment. Med Teach. 2018;2018:1–8.

    Google Scholar 

  40. Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ. 2011;45(6):560–9.

    Article  PubMed  Google Scholar 

  41. Ginsburg S, van der Vleuten CPM, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 2017;92(11):1617–21.

    Article  PubMed  Google Scholar 

  42. Ginsburg S, Regehr G, Lingard L, Eva KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015;49(3):296–306.

    Article  PubMed  Google Scholar 

  43. Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013;88(10):1539–44.

    Article  PubMed  Google Scholar 

  44. Ginsburg S, van der Vleuten CP, Eva KW, Lingard L. Cracking the code: residents' interpretations of written assessment comments. Med Educ. 2017;51(4):401–10.

    Article  PubMed  Google Scholar 

  45. Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisors' completed clinical evaluation reports. Med Educ. 2008;42(8):816–22.

    Article  PubMed  Google Scholar 

  46. Holmboe ES. Work-based assessment and co-production in postgraduate medical training. GMS J Med Educ. 2017;34(5):Doc58.

    Google Scholar 

  47. Ten Cate TJ, Kusurkar RA, Williams GC. How self-determination theory can assist our understanding of the teaching and learning processes in medical education. AMEE guide no. 59. Med Teach. 2011;33(12):961–73.

    Article  PubMed  Google Scholar 

  48. Sandars J, Cleary TJ. Self-regulation theory: applications to medical education: AMEE guide no. 58. Med Teach. 2011;33(11):875–86.

    Article  PubMed  Google Scholar 

  49. Sargeant J, Mann K, van der Vleuten C, Metsemakers J. “Directed” self-assessment: practice and feedback within a social context. J Contin Educ Heal Prof. 2008;28(1):47–54.

    Article  Google Scholar 

  50. Konings KD, van Berlo J, Koopmans R, et al. Using a smartphone app and coaching group sessions to promote residents’ reflection in the workplace. Acad Med. 2016;91(3):365–70. https://doi.org/10.1097/ACM.0000000000000989.

    Article  PubMed  Google Scholar 

  51. LaDonna KA, Ginsburg S, Watling C. “Rising to the level of your incompetence”: what physicians’ self-assessment of their performance reveals about the imposter syndrome in medicine. Acad Med. 2018;93(5):763–8.

    Article  PubMed  Google Scholar 

  52. Schuwirth LW, van der Vleuten CP. Programmatic assessment and Kane's validity perspective. Med Educ. 2012;46(1):38–48.

    Article  PubMed  Google Scholar 

  53. Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. Twelve tips for programmatic assessment. Med Teach. 2015;37(7):641–6.

    Article  Google Scholar 

  54. Ten Cate O. Competency-based education, entrustable professional activities, and the power of language. J Grad Med Educ. 2013;5(1):6–7.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE guide no. 99. Med Teach. 2015;37(11):983–1002.

    Article  PubMed  Google Scholar 

  56. Dagnone D, Stockley D, Flynn L, et al. Delivering on the promise of competency based medical education - an institutional approach. Can Med Educ J. 2019;10(1):e28–38.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Macaluso M, Kinzie M, Cowley D, et al. The psychiatry milestones 2.0: how did we get from 1.0 to 2.0 and what can users expect? Acad Psychiatry. 2020;44(6):779–84.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Thomas CR, Keepers G. The milestones for general psychiatry residency training. Acad Psychiatry. 2014;38(3):255–60.

    Article  PubMed  Google Scholar 

  59. Leipzig RM, Sauvigne K, Granville LJ, et al. What is a geriatrician? American Geriatrics Society and Association of Directors of Geriatric Academic Programs end-of-training entrustable professional activities for geriatric medicine. J Am Geriatr Soc. 2014;62(5):924–9.

    Article  PubMed  Google Scholar 

  60. Wisman-Zwarter N, van der Schaaf M, Ten Cate O, Jonker G, van Klei WA, Hoff RG. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: a Delphi study. Eur J Anaesthesiol. 2016;33(8):559–67.

    Article  PubMed  Google Scholar 

  61. Brown CR Jr, Criscione-Schreiber L, O'Rourke KS, et al. What is a rheumatologist and how do we make one? Arthritis Care Res. 2016;68(8):1166–72.

    Article  Google Scholar 

  62. Rose S, Fix OK, Shah BJ, et al. Entrustable professional activities for gastroenterology fellowship training. Gastrointest Endosc. 2014;80(1):16–27.

    Article  PubMed  Google Scholar 

  63. Weiss A, Ozdoba A, Carroll V, DeJesus F. Entrustable professional activities: enhancing meaningful use of evaluations and milestones in a psychiatry residency program. Acad Psychiatry. 2016;40(5):850–4.

    Article  PubMed  Google Scholar 

  64. Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ. 2011;11:96.

    Article  PubMed  PubMed Central  Google Scholar 

  65. RANZCP. The Royal Australian & New Zealand College of Psychiatrists. EPA Handbook. https://www.ranzcp.org/Files/PreFellowship/2012-Fellowship-Program/EPA-forms/EPA-handbook.aspx. Accessed 22 Jan 2017.

  66. Young JQ, Hasser C, Hung EK, et al. Developing end-of-training entrustable professional activities for psychiatry: results and methodological lessons. Acad Med. 2018;93(7):1048–54.

    Article  PubMed  Google Scholar 

  67. Pinilla S, Lenouvel E, Strik W, Kloppel S, Nissen C, Huwendiek S. Entrustable professional activities in psychiatry: a systematic review. Acad Psychiatry. 2020;44(1):37–45.

    Article  PubMed  Google Scholar 

  68. Hung EK, Jibson M, Sadhu J, et al. Wresting with Implementation: a step-by-step guide to implementing Entrustable Professional Activities (EPAs) in psychiatry residency programs. Acad Psychiatry. 2021;45(2):210–16. https://doi.org/10.1007/s40596-020-01341-7.

  69. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Med Teach. 2007;29(9):855–71.

    Article  PubMed  Google Scholar 

  70. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7.

    Article  PubMed  Google Scholar 

  71. Lockyer JM, Sargeant J, Richards SH, Campbell JL, Rivera LA. Multisource feedback and narrative comments: polarity, specificity, actionability, and CanMEDS roles. J Contin Educ Heal Prof. 2018;38(1):32–40.

    Article  Google Scholar 

  72. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 conference. Med Teach. 2011;33(3):206–14.

    Article  PubMed  Google Scholar 

  73. Grujich NN, Razmy A, Zaretsky A, Styra RG, Sockalingam S. Evaluation of professional role competency during psychiatry residency. Acad Psychiatry. 2012;36(2):126–8.

    Article  PubMed  Google Scholar 

  74. Violato C, Lockyer JM, Fidler H. Assessment of psychiatrists in practice through multisource feedback. Can J Psychiatr. 2008;53(8):525–33.

    Article  Google Scholar 

  75. Vesel TP, O'Brien BC, Henry DM, van Schaik SM. Useful but different: resident physician perceptions of Interprofessional feedback. Teach Learn Med. 2016;28(2):125–34.

    Article  PubMed  Google Scholar 

  76. Philibert I. Using chart review and chart-stimulated recall for resident assessment. J Grad Med Educ. 2018;10(1):95–6.

    Article  PubMed  PubMed Central  Google Scholar 

  77. Young JQ, Lieu S, O'Sullivan P, Tong L. Development and initial testing of a structured clinical observation tool to assess pharmacotherapy competence. Acad Psychiatry. 2011;35(1):27–34.

    Article  PubMed  Google Scholar 

  78. Young JQ, Sugarman R, Schwartz J, O’Sullivan PS. Faculty and Resident Engagement With a Workplace-Based Assessment Tool: Use of Implementation Science to Explore Enablers and Barriers. Acad Med. 2020;95(12):1937–44.

    Google Scholar 

  79. Young JQ, Irby DM, Kusz M, O'Sullivan PS. Performance assessment of pharmacotherapy: results from a content validity survey of the Psychopharmacotherapy-Structured Clinical Observation (P-SCO) tool. Acad Psychiatry. 2018;42(6):765–72.

    Article  PubMed  Google Scholar 

  80. Young JQ, Rasul R, O'Sullivan PS. Evidence for the validity of the psychopharmacotherapy-structured clinical observation tool: results of a factor and time series analysis. Acad Psychiatry. 2018;42(6):759–64.

    Article  PubMed  Google Scholar 

  81. Young JQ, Sugarman R, Holmboe E, O'Sullivan PS. Advancing our understanding of narrative comments generated by direct observation tools: lessons from the psychopharmacotherapy-structured clinical observation. J Grad Med Educ. 2019;11(5):570–9.

    Article  PubMed  PubMed Central  Google Scholar 

  82. Jibson MD, Broquet KE, Anzia JM, et al. Clinical skills verification in general psychiatry: recommendations of the ABPN task force on rater training. Acad Psychiatry. 2012;36(5):363–8.

    Article  PubMed  Google Scholar 

  83. Young JQ, McClure M. Fast, Easy, and Good: Assessing Entrustable Professional Activities in Psychiatry Residents With a Mobile App. Acad Med. 2020;95(10):1546–9.

    Google Scholar 

  84. O'Sullivan PS, Reckase MD, McClain T, Savidge MA, Clardy JA. Demonstration of portfolios to assess competency of residents. Adv Health Sci Educ Theory Pract. 2004;9(4):309–23.

    Article  PubMed  Google Scholar 

  85. Sockalingam S, Stergiopoulos V, Maggi JD, Zaretsky A, Stovel L, Hodges B. Evaluating psychiatry residents as physician-managers: development of an assessment tool. Acad Psychiatry. 2013;37(1):11–7.

    Article  PubMed  Google Scholar 

  86. Schumacher DJ, Holmboe E, Carraccio C, et al. Resident-Sensitive Quality Measures in the Pediatric Emergency Department: Exploring Relationships With Supervisor Entrustment and Patient Acuity and Complexity. Acad Med. 2020;95(8):1256–64.

    Google Scholar 

  87. Schumacher DJ, Holmboe ES, van der Vleuten C, Busari JO, Carraccio C. Developing resident-sensitive quality measures: a model from pediatric emergency medicine. Acad Med. 2018;93(7):1071–8.

    Article  PubMed  Google Scholar 

  88. Schumacher DJ, Martini A, Holmboe E, et al. Initial Implementation of Resident-Sensitive Quality Measures in the Pediatric Emergency Department: A Wide Range of Performance. Acad Med. 2020;95(8):1248–55.

    Google Scholar 

  89. Davis J, Roach C, Elliott C, Mardis M, Justice EM, Riesenberg LA. Feedback and assessment tools for handoffs: a systematic review. J Grad Med Educ. 2017;9(1):18–32.

    Article  PubMed  PubMed Central  Google Scholar 

  90. ten Cate O, Durning SJ. Approaches to assessing the clinical reasoning of preclinical students. In: ten Cate O, Custers E, Durning SJ, editors. Principles and practice of case-based clinical reasoning education: a method for preclinical students. Cham: Springer. Copyright 2018, The Author(s). 2018:65–72.

    Google Scholar 

  91. Cook RJ, Durning SJ. Clinical process modeling: an approach for enhancing the assessment of physicians' clinical reasoning. Acad Med. 2019;94(9):1317–22.

    Article  PubMed  Google Scholar 

  92. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212–20.

    Article  PubMed  Google Scholar 

  93. Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: rater performance and behavior when considering multiple competencies. Teach Learn Med. 2016;28(1):41–51.

    Article  PubMed  Google Scholar 

  94. Lefebvre C, Hiestand B, Glass C, et al. Examining the effects of narrative commentary on evaluators’ summative assessments of resident performance. Eval Health Prof. 2020;43:159. https://doi.org/10.1177/0163278718820415.

    Article  PubMed  Google Scholar 

  95. Young JQ, Sugarman R, Schwartz J, McClure M, O'Sullivan PS. A mobile app to capture EPA assessment data: utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement. Perspect Med Edu. 2020;

    Google Scholar 

  96. Warm EJ, Held JD, Hellmann M, et al. Entrusting observable practice activities and milestones over the 36 months of an internal medicine residency. Acad Med. 2016;91(10):1398–405.

    Article  PubMed  Google Scholar 

  97. Saleem JJ, Herout J, Wilck NR. Function-specific design principles for the electronic health record. Proc Human Factors Ergonom Soc Ann Meet. 2016;60(1):578–82.

    Article  Google Scholar 

  98. Mayer RE, Moreno R. Nine ways to reduce cognitive load in multimedia learning. Educ Psychol. 2003;38(1):43–52.

    Article  Google Scholar 

  99. Massie J, Ali JM. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract. 2016;21(2):455–73.

    Article  PubMed  Google Scholar 

  100. Cheung WJ, Patey AM, Frank JR, Mackay M, Boet S. Barriers and enablers to direct observation of trainees’ clinical performance: a qualitative study using the theoretical domains framework. Acad Med. 2019;94(1):101–14.

    Article  PubMed  Google Scholar 

  101. Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions? Med Educ. 2011;45(1):69–80.

    Article  PubMed  Google Scholar 

  102. Ramani S, Konings KD, Ginsburg S, van der Vleuten CPM. Twelve tips to promote a feedback culture with a growth mind-set: swinging the feedback pendulum from recipes to relationships. Med Teach. 2019;41(6):625–31.

    Article  PubMed  Google Scholar 

  103. Ramani S, Post SE, Konings K, Mann K, Katz JT, van der Vleuten C. “It’s just not the culture”: a qualitative study exploring residents’ perceptions of the impact of institutional culture on feedback. Teach Learn Med. 2017;29(2):153–61.

    Article  PubMed  Google Scholar 

  104. Telio S, Ajjawi R, Regehr G. The “educational alliance” as a framework for reconceptualizing feedback in medical education. Acad Med. 2015;90(5):609–14.

    Article  PubMed  Google Scholar 

  105. Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943–54.

    Article  PubMed  Google Scholar 

  106. Sukhera J, Wodzinski M, Milne A, Teunissen PW, Lingard L, Watling C. Implicit bias and the feedback paradox: exploring how health professionals engage with feedback while questioning its credibility. Acad Med. 2019;94(8):1204–10.

    Article  PubMed  Google Scholar 

  107. Watling C, Driessen E, van der Vleuten CP, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements. Med Educ. 2012;46(2):192–200.

    Article  PubMed  Google Scholar 

  108. Telio S, Regehr G, Ajjawi R. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 2016;50(9):933–42.

    Article  PubMed  Google Scholar 

  109. Bok HG, Teunissen PW, Favier RP, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 2013;13:123.

    Article  PubMed  PubMed Central  Google Scholar 

  110. Harrison C, Wass V. The challenge of changing to an assessment for learning culture. Med Educ. 2016;50(7):704–6.

    Article  PubMed  Google Scholar 

  111. Harrison CJ, Konings KD, Schuwirth LWT, Wass V, van der Vleuten CPM. Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC Med Educ. 2017;17(1):73.

    Article  PubMed  PubMed Central  Google Scholar 

  112. Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. 'Sometimes the work just needs to be done': socio-cultural influences on direct observation in medical training. Med Educ. 2016;50(10):1054–64.

    Article  PubMed  Google Scholar 

  113. LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. Staging a performance: learners' perceptions about direct observation during residency. Med Educ. 2017;51(5):498–510.

    Article  PubMed  Google Scholar 

  114. Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines: the do's, don'ts and don't knows of direct observation of clinical skills in medical education. Perspect Med Educ. 2017;6(5):286–305.

    Article  PubMed  PubMed Central  Google Scholar 

  115. Lorwald AC, Lahner FM, Greif R, Berendonk C, Norcini J, Huwendiek S. Factors influencing the educational impact of mini-CEX and DOPS: a qualitative synthesis. Med Teach. 2018;40(4):414–20.

    Article  PubMed  Google Scholar 

  116. Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. Overcoming the challenges of direct observation and feedback programs: a qualitative exploration of resident and faculty experiences. Teach Learn Med. 2020;32(5):541–51. https://doi.org/10.1080/10401334.2020.1767107.

  117. Kogan JR, Conforti LN, Bernabeo E, Iobst W, Holmboe E. How faculty members experience workplace-based assessment rater training: a qualitative study. Med Educ. 2015;49(7):692–708.

    Article  PubMed  Google Scholar 

  118. van de Ridder JM, McGaghie WC, Stokking KM, ten Cate OT. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ. 2015;49(7):658–73.

    Article  PubMed  Google Scholar 

  119. Sargeant J, Lockyer JM, Mann K, et al. The R2C2 model in residency education: how does it Foster coaching and promote feedback use? Acad Med. 2018;93(7):1055–63.

    Article  PubMed  Google Scholar 

  120. Holmboe EH. Developing Faculty Competencies in Assessment. https://www.acgme.org/Meetings-and-Educational-Activities/Other-Educational-Activities/Courses-and-Workshops/Developing-Faculty-Competencies-in-Assessment. Accessed 20 May 2020.

  121. Iobst WI, Holmboe ES. Programmatic assessment: the secret sauce of effective CBME implementation. J Gradaute Med Educ. In Press.

    Google Scholar 

  122. Eva KW. On the generality of specificity. Med Educ. 2003;37(7):587–8.

    Article  PubMed  Google Scholar 

  123. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005;39(3):309–17.

    Article  PubMed  Google Scholar 

  124. Eva KW, Bordage G, Campbell C, et al. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 2016;21(4):897–913.

    Article  PubMed  Google Scholar 

  125. Thoma B, Bandi V, Carey R, et al. Developing a dashboard to meet competence committee needs: a design-based research project. Can Med Educ J. 2020;11(1):e16–34.

    PubMed  PubMed Central  Google Scholar 

  126. Warm EJ, Kinnear B, Kelleher M, Sall D, Holmboe E. Transforming resident assessment: an analysis using deming's system of profound knowledge. Acad Med. 2018.

    Google Scholar 

  127. Ross S, Binczyk NM, Hamza DM, et al. Association of a competency-based assessment system with identification of and support for medical residents in difficulty. JAMA Netw Open. 2018;1(7):e184581.

    Article  PubMed  PubMed Central  Google Scholar 

  128. Holmboe ES, Yamazaki K, Nasca TJ, Hamstra SJ. Using longitudinal milestones data and learning analytics to facilitate the professional development of residents: early lessons from three specialties. Acad Med. 2020;95(1):97–103.

    Article  PubMed  Google Scholar 

  129. Kelleher M, Kinnear B, Sall DR, et al. Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program. Perspect Med Educ. 2021;

    Google Scholar 

  130. Govaerts M. Workplace-based assessment and assessment for learning: threats to validity. J Grad Med Educ. 2015;7(2):265–7.

    Article  PubMed  PubMed Central  Google Scholar 

  131. Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of engagement: residents’ perceptions of the in-training evaluation process. Acad Med. 2008;83(10 Suppl):S97–100.

    Article  PubMed  Google Scholar 

  132. Nicol DJ, Macfarlane-Dick D. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud High Educ. 2006;31(2):199–218.

    Article  Google Scholar 

  133. Harrison CJ, Konings KD, Molyneux A, Schuwirth LW, Wass V, van der Vleuten CP. Web-based feedback after summative assessment: how do students engage? Med Educ. 2013;47(7):734–44.

    Article  PubMed  Google Scholar 

  134. Young JQ, Sugarman R, Schwartz J, McClure M, O'Sulivan P. A mobile app to capture EPA assessment data: utilizing the consolidated framework for implementation reserach to identify enablers and barriers to engagement. Perspect Med Educ. In Press.

    Google Scholar 

  135. White CB, Gruppen LD, Fantone JC. Self-regulated learning in medical education. In: Swanwick T, ed. Understanding Medical Education. 2013:201–11.

    Google Scholar 

  136. Sargeant J, Lockyer J, Mann K, et al. Facilitated reflective performance feedback: developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Acad Med. 2015;90(12):1698–706.

    Article  PubMed  Google Scholar 

  137. Gawande A. Personal best. Top athletes and singers have coaches. Should you? In. The New Yorker. New York; 2011.

    Google Scholar 

  138. Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90(11):1471–86.

    Article  PubMed  Google Scholar 

  139. Palamara K, Kauffman C, Stone VE, Bazari H, Donelan K. Promoting success: a professional development coaching program for interns in medicine. J Grad Med Educ. 2015;7(4):630–7.

    Article  PubMed  PubMed Central  Google Scholar 

  140. Parks K, Miller J, Westcott A. Coaching in graduate medical education. In: Coaching in medical education: a faculty handbook. American Medical Association. 2017. pp. 50–53.

    Google Scholar 

  141. Sargeant J, Eva KW, Armson H, et al. Features of assessment learners use to make informed self-assessments of clinical performance. Med Educ. 2011;45(6):636–47.

    Article  PubMed  Google Scholar 

  142. Andolesk K, Padmore J, Hauer KE, Ekpenyong A, Edgar L, Holmboe EH. Clinical competency committees: a guidebook for programs. accreditation council for graduate medical education. file:///C:/Users/jyoung9/OneDrive%20-%20Northwell%20Health/Academic/Presentations%20and%20Publications/Psychiatric%20Clinics/System%20Article/ACGMEClinicalCompetencyCommitteeGuidebook%20(1).pdf. Published 2020. Accessed 20 May 2020.

    Google Scholar 

  143. Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment. Acad Med. 2015;90(8):1084–92.

    Article  PubMed  Google Scholar 

  144. Teherani A, Hauer KE, Fernandez A, King TEJ, Lucey C. How small differences in assessed clinical performance amplify to large differences in grades and awards: a Cascade with serious consequences for students underrepresented in medicine. Acad Med. 2018;93(9):1286–92.

    Article  PubMed  Google Scholar 

  145. Klein R, Julian KA, Snyder ED, et al. Gender bias in resident assessment in graduate medical education: review of the literature. J Gen Intern Med. 2019;34(5):712–9.

    Article  PubMed  PubMed Central  Google Scholar 

  146. Rojek AE, Khanna R, Yim JWL, et al. Differences in narrative language in evaluations of medical students by gender and under-represented minority status. J Gen Intern Med. 2019;34(5):684–91.

    Article  PubMed  PubMed Central  Google Scholar 

  147. Donato AA, Alweis R, Wenderoth S. Design of a clinical competency committee to maximize formative feedback. J Commun Hosp Intern Med Perspect. 2016;6(6):33533.

    Article  Google Scholar 

  148. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings’ post-rotation performance ratings detect residents’ clinical performance deficiencies? Acad Med. 2004;79(5):453–7.

    Article  PubMed  Google Scholar 

  149. Pack R, Lingard L, Watling CJ, Chahine S, Cristancho SM. Some assembly required: tracing the interpretative work of clinical competency committees. Med Educ. 2019;53(7):723–34.

    Article  PubMed  Google Scholar 

  150. Hauer KE, Cate OT, Boscardin CK, et al. Ensuring resident competence: a narrative review of the literature on group decision making to inform the work of clinical competency committees. J Grad Med Educ. 2016;8(2):156–64.

    Article  PubMed  PubMed Central  Google Scholar 

  151. Kinnear B, Warm EJ, Hauer KE. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 2018;40(11):1110–5.

    Article  PubMed  Google Scholar 

  152. Birken SA, Powell BJ, Presseau J, et al. Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): a systematic review. Implement Sci. 2017;12(1):2.

    Article  PubMed  PubMed Central  Google Scholar 

  153. Bowe CM, Armstrong E. Assessment for systems learning: a holistic assessment framework to support decision making across the medical education continuum. Acad Med. 2017;92(5):585–92.

    Article  PubMed  Google Scholar 

  154. Warm EJ, Edgar L, Kelleher M, et al. A guidebook for Implementating and changing assessment in the milestone era. Accreditation Council for Graduate Medical Education. https://www.acgme.org/Portals/0/Milestones%20Implementation%202020.pdf?ver=2020-05-11-165927-097. Published 2020. Accessed.

  155. Young JQ, Frank JR, Holmboe ES. Advancing workplace-based assessment in psychiatric education: key design and implementation issues. PsychiatrClin North Am. 2021;44(2):317–32.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA

    John Q. Young

  2. Zucker Hillside Hospital at Northwell Health, Glen Oaks, NY, USA

    John Q. Young

Authors
  1. John Q. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John Q. Young .

Editor information

Editors and Affiliations

  1. Department of Psychiatry and Behavioral Neurobiology, The University of Alabama at Birmingham, Heersink School of Medicine, Birmingham, AL, USA

    Matthew Macaluso

  2. Department of Psychiatry, Neuroscience Institute Southern Illinois University School of Medicine, Springfield, IL, USA

    L. Joy Houston

  3. Department of Psychiatry, Oregon Health and Science University, Portland, OR, USA

    J. Mark Kinzie

  4. Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA

    Deborah S. Cowley

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Young, J.Q. (2022). Building a System of Competency-Based Assessment. In: Macaluso, M., Houston, L.J., Kinzie, J.M., Cowley, D.S. (eds) Graduate Medical Education in Psychiatry. Springer, Cham. https://doi.org/10.1007/978-3-031-00836-8_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-00836-8_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-00835-1

  • Online ISBN: 978-3-031-00836-8

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation