Understanding Assessment Systems for Clinical Competency Committee Decisions: Evidence from a Multisite Study of Psychiatry Residency Training Programs

Abstract

Objective

This multisite study examines how clinical competency committees in Psychiatry synthesize resident assessments to inform milestones decisions to provide guidelines that support their use.

Methods

The study convened training directors and associate training directors from three psychiatry residency programs to examine decision-making processes of clinical competency committees. Annual resident assessments for one second year and one third year resident were used in a mock clinical competency committee format to assign milestones for two consecutive reporting periods. The committees reflected on the process and rated how the assessment tools impacted the assessment of milestones and evaluated the overall process. The authors compared reliability of assessment between the mock committees and examined both reliability of end of rotation assessments and their composite scores when combined with clinical skills evaluations.

Results

End of rotation evaluations were the most informative tool for assigning milestones and clarifying discrepancies in performance. In particular, the patient care and medical knowledge competencies were the easiest to rate, while the systems-based practice and practice-based learning and improvement were the most difficult. Reliability between committees was low although higher number of available evaluations improved reliability in decision-making.

Conclusions

The results indicate that the medical knowledge and patient care competencies are the easiest to rate and informed most by end of rotation evaluations and clinical skills examinations. Other evaluation tools may better capture performance on specific sub-competencies beyond workplace-based assessment, or it may be helpful to reconsider the utility of how individual sub-competencies are evaluated.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

References

  1. 1.

    Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. 2012;366:1051–6.

    CAS  Article  Google Scholar 

  2. 2.

    Swing SR, Cowley DS, Bentman A. Assessing resident performance on the psychiatry milestones. Acad Psychiatry. 2014;38:294–302.

    Article  Google Scholar 

  3. 3.

    Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32:676–82.

    Article  Google Scholar 

  4. 4.

    Guerrero APS, Beresin EV, Balon R, Brenner AM, Louie AK, Coverdale JH, et al. The competency movement in psychiatric education. Acad Psychiatry. 2017;41:312–4.

  5. 5.

    Park YS, Riddle J, Tekian A. Validity evidence of resident competency ratings and the identification of problem residents. Med Educ. 2014;48:614–22.

    Article  Google Scholar 

  6. 6.

    Park YS, Hicks PJ, Carraccio C, Margolis M, Schwartz A. Does incorporating a measure of clinical workload improve workplace-based assessment scores? Insights for measurement precision and longitudinal score growth from ten pediatrics residency programs. Acad Med. 2018;93:S21–s9.

    Article  Google Scholar 

  7. 7.

    Sebesta EM, Cooper KL, Badalato GM. Program director perceptions of usefulness of the accreditation Council for Graduate Medical Education Milestones System for urology resident evaluation. Urology. 2019;124:28–32.

    Article  Google Scholar 

  8. 8.

    Goyal N, Folt J, Jaskulka B, Baliga S, Slezak M, Schultz LR, et al. Assessment methods and resource requirements for milestone reporting by an emergency medicine clinical competency committee. Med Educ Online. 2018;23:1538925.

    Article  Google Scholar 

  9. 9.

    Regan L, Cope L, Omron R, Bright L, Bayram JD. Do end-of-rotation and end-of-shift assessments inform clinical competency committees' (CCC) decisions? West J Emerg Med. 2018;19:121–7.

    Article  Google Scholar 

  10. 10.

    Park YS, Zar FA, Norcini JJ, Tekian A. Competency evaluations in the next accreditation system: contributing to guidelines and implications. Teach Learn Med. 2016;28:135–45.

    Article  Google Scholar 

  11. 11.

    Tekian A, Park YS, Tilton S, Prunty PF, Abasolo E, Zar F, et al. Competencies and feedback on internal medicine residents' end-of-rotation assessments over time: qualitative and quantitative analyses. Acad Med. 2019:Epub.

  12. 12.

    Tekian A, Park YS, Tilton S, Prunty PF, Abasolo E, Zar F, et al. Competencies and feedback on internal medicine Residents' end-of-rotation assessments over time: qualitative and quantitative analyses. Acad Med. 2019.

  13. 13.

    Simmons SW, Varley CK, Hunt J. Experiences with the child and adolescent psychiatry milestones: results of two Nationwide surveys. Acad Psychiatry. 2018;42:464–8.

    Article  Google Scholar 

  14. 14.

    Padilla A, Benjamin S, Lewis-Fernandez R. Assessing cultural psychiatry milestones through an objective structured clinical examination. Acad Psychiatry. 2016;40:600–3.

    Article  Google Scholar 

Download references

Funding

This study is supported by a research award from the American Board of Psychiatry and Neurology, Inc.

Author information

Affiliations

Authors

Corresponding author

Correspondence to R. Brett Lloyd.

Ethics declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Ethical Approval

This study was exempt by the Northwestern University Institutional Review Board (STU00207679) and exempt by the University of Illinois Institutional Review Board.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lloyd, R.B., Park, Y.S., Tekian, A. et al. Understanding Assessment Systems for Clinical Competency Committee Decisions: Evidence from a Multisite Study of Psychiatry Residency Training Programs. Acad Psychiatry 44, 734–740 (2020). https://doi.org/10.1007/s40596-019-01168-x

Download citation

Keywords

  • Clinical competency committee
  • Milestones
  • Assessment
  • Evaluation