Skip to main content
SpringerLink
Log in

Psychiatry Clinical Skills Evaluation: a Multisite Study of Validity

  • Empirical Report
  • Published:
Academic Psychiatry Aims and scope Submit manuscript

Abstract

Objective

Since 2007, the American Board of Psychiatry and Neurology (ABPN) has required that residency programs conduct a specific clinical skills evaluation (CSE) of physician-patient interaction, psychiatric interview and mental status examination, and case presentation on a directly observed patient interview as a prerequisite for certification. The authors examined a multisite database of CSE assessments to investigate the validity of the evaluation.

Methods

The authors collected 1156 CSE assessments from 4 residency programs conducted over a 6-year period, compared scoring patterns among the programs, score improvement over 4 years of residency, time and number of CSEs required to meet ABPN requirements, and patterns of scoring for individual faculty evaluators.

Results

The distribution of scores within each of the 4 programs showed similar, but nonidentical patterns. The number of CSEs required to meet the ABPN standards (3.5) and the point in training at which this was completed (late PGY-2) were the same in all programs. CSE scores were highly correlated with year of training but were not correlated with performance on an unrelated cognitive examination. Individual faculty members tended to stay within a moderate range of scores over multiple residents, partially attributable to year of training.

Conclusions

Taken together, these findings support the validity of the CSE as a measure of residents’ clinical skills in the specified areas and demonstrate a moderate-high degree of consistency in the scoring of the CSE across these 4 programs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. American Board of Psychiatry and Neurology, Inc. Requirements for clinical skills evaluation in psychiatry. 2017. https://www.abpn.com/wp-content/uploads/2015/01/CSE-Psychiatry-2017.pdf. Accessed 3 Nov 2020.

  2. ACGME Program Requirements for Graduate Medical Education in Psychiatry. 2020. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/400_Psychiatry_2020.pdf?ver=2020-06-19-123110-817. Accessed 3 Nov 2020.

  3. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78:826–30.

    Article  Google Scholar 

  4. Jibson MD, Broquet KE, Anzia JM, Beresin EV, Hunt JI, Kaye D, et al. ABPN Task Force on Clinical Skills Verification Rater Training. Clinical skills verification in general psychiatry: recommendations of the ABPN Task Force on Rater Training. Acad Psychiatry. 2012;36:363–8.

    Article  Google Scholar 

  5. AADPRT Virtual Training Office. ABPN approved CSV/CSA v.2013. 2013. https://portal.aadprt.org/public/vto/categories/Program%20Administration%20Tools/Clinical%20Skills%20Assessment/Forms/58000e48bf07a_AADPRT_CSV_3b_rev_8-5-2013.pdf. Accessed 3 Nov 2020.

  6. Dalack GW, Jibson MD. Clinical skills verification, formative feedback, and psychiatry residency trainees. Acad Psychiatry. 2012;36:122–5.

    Article  Google Scholar 

  7. Juul D, Brooks BA, Jozefowicz R, Jibson M, Faulkner L. Clinical skills assessment: the effects of moving certification requirements into neurology, child neurology, and psychiatry residency training. J Grad Med Educ. 2015;7:98–100.

    Article  Google Scholar 

  8. Rao NR, Kodali R, Mian A, Ramtekkar U, Kamarajan C, Jibson MD. Psychiatric residents’ attitudes towards and experiences with the clinical skills verification process - a pilot study of US and international medical graduates. Acad Psychiatry. 2012;36:316–22.

    Article  Google Scholar 

  9. Jibson MD, Keshavarzi N. Psychiatry clinical skills evaluation: interrater reliability of the American Board of Psychiatry and Neurology Required Assessment. Acad Psychiatry. 2020. https://doi.org/10.1007/s40596-020-01337-3 Accessed 3 Nov 2020.

  10. McDermott JF Jr, Tanguay PE, Scheiber SC, Juul D, Shore JH, Tucker GJ, et al. Reliability of the part II board certification examination in psychiatry: interexaminer consistency. Am J Psychiatry. 1991;148:1672–4.

    Article  Google Scholar 

  11. McDermott JF Jr, Tanguay PE, Scheiber SC, Juul D, Shore JH, Tucker GJ, et al. Reliability of the part II board certification examination in psychiatry: examination stability. Am J Psychiatry. 1993;150:1077–80.

    Article  Google Scholar 

  12. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830–7.

    Article  Google Scholar 

  13. Kane MT. Current concerns in validity theory. J Educ Measurement. 2001;38:319–42.

    Article  Google Scholar 

Download references

Acknowledgments

This study was reviewed by the University of Michigan Institutional Review Board and granted a Notice of Exemption on May 2, 2014 (OHRP IRB Registration Number: IRB00000246).

Funding

This study was supported by the American Board of Psychiatry and Neurology through a Fellowship Grant and Research Award.

Author information

Authors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Michael D. Jibson & Lisa S. Seyfried

  2. Northwestern University, Chicago, IL, USA

    Gaurava Agarwal & Joan M. Anzia

  3. University of Pennsylvania, Bryn Mawr, PA, USA

    Richard F. Summers

  4. Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Glen Oaks, NY, USA

    John Q. Young

Authors
  1. Michael D. Jibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gaurava Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joan M. Anzia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard F. Summers
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. John Q. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lisa S. Seyfried
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael D. Jibson.

Ethics declarations

Disclosure

Michael Jibson and Gaurava Agarwal were the recipients of grant funding from the American Board of Psychiatry and Neurology during the conduct of the study. Joan Anzia is a director of the American Board of Psychiatry and Neurology. Richard Summers is a member of the Accreditation Council on Graduate Medical Education Psychiatry Review Committee. On behalf of all other authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jibson, M.D., Agarwal, G., Anzia, J.M. et al. Psychiatry Clinical Skills Evaluation: a Multisite Study of Validity. Acad Psychiatry 45, 413–419 (2021). https://doi.org/10.1007/s40596-020-01388-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40596-020-01388-6

Keywords

Search

Navigation