Skip to main content
SpringerLink
Log in

Best practices for assessing competence and performance of the behavioral Health workforce

  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

The need for mechanisms to assess the competence and performance of the behavioral health workforce has received increasing attention. This article reviews strategies used in general medicine and other disciplines for assessing trainees and practitioners. The possibilities and limitations of various approaches are reviewed, and the implications for behavioral health are addressed. A conceptual model of competence is presented, and practical applications of this model are reviewed. Finally, guidelines are proposed for building competency assessment protocols for behavioral health.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Board of Medical Specialties. (2003). ABMS annual report and reference handbook. Available from: http://www.abms.org

  • D. Andrich (1988) Rasch models for measurement Sage Publications Newbury Park CA

    Google Scholar 

  • Art Therapy Credentials Board. (2004). Art therapy credentials board: Booklet of information and study guide for certification examination. Available from: http://www.atcb.org/

  • P.G. Bashook (1993) Clinical competence and continuing medical education: Lifelong learning to maintain competence C. Coles H.A. Holm (Eds) Learning in medicine Scandinavian University Press Oslo, Norway 21–41

    Google Scholar 

  • P.G. Bashook (1994) Beyond the traditional written and oral examinations: New certification methods J. Shore S. Scheiber (Eds) Certification, recertification, and lifetime learning in psychiatry American Psychiatric Press Washington, DC 117–138

    Google Scholar 

  • P.G. Bashook (2003) Structured case-based orals are a valid and reliable measure of physician’s clinical decision-making American Educational Research Association Chicago, IL

    Google Scholar 

  • P.G. Bashook J. Parboosingh (1998) ArticleTitleContinuing medical education: Recertification and the maintenance of competence British Medical Journal 316 545–548 Occurrence Handle9501722

    PubMed  Google Scholar 

  • P.G. Bashook S. Swing (2000) Toolbox of assessment methods©, version 1.1 Accreditation Council for Graduate Medical Education /American Board of Medical Specialties Evanston, IL

    Google Scholar 

  • R.L. Brennan (2001) ArticleTitleAn essay on the history and future of reliability from the perspective of replications Journal of Educational Measurement 38 IssueID4 295–317

    Google Scholar 

  • A.H. Browning A.C. Bugbee M.A. Mullins (Eds) (1996) Certification: A NOCA handbook National Organization for Competency Assurance Washington, DC

    Google Scholar 

  • S.M. Case D.B. Swanson (2003) Constructing written test questions for the basic and clinical sciences National Board of Medical Examiners Philadelphia, PA

    Google Scholar 

  • S.G. Clyman D.E. Melnick B.E. Clauser (1995) Computer-based case simulations E.L. Mancall P.G. Bashook (Eds) Assessing clinical reasoning: The oral examination and alternative methods American Board of Medical Specialties Evanston, IL 139–149

    Google Scholar 

  • J. Cunnington L. Southgate (2002) Relicensure, recertification, and practice-based assessment G.R. Norman D.I. Newble C.P.M. Vander Vleuten (Eds) International handbook of research in medical education Kluwer Academic Publishers Amsterdam 883–912

    Google Scholar 

  • J.E. Des Marchais P. Jean (1993) ArticleTitleEffects of examiner training on open-ended, high taxonomic level questioning in oral certification examinations Teaching & Learning in Medicine 5 IssueID1 24–28

    Google Scholar 

  • Educational Testing Service. (2004). Graduate record examination (GRE). Available from: http://www.gre.org/ttindex.html

  • J. Foulkes R. Bandaranayake R. Hayes G. Phillips A. Rothman L. Southgate et al. (1993) Combining components of assessment D. Newble B. Jolly R. Wakeford (Eds) The certification and recertification of doctors: Issues in the assessment of clinical competence Cambridge University Press Great Britain 134–150

    Google Scholar 

  • D.M. Gaba S.K. Howard B. Flanagan B.E. Smith K.J. Fish R. Botney (1998) ArticleTitleAssessment of clinical performance during simulated crises using both technical and behavioral ratings Anesthesiology 89 8–18 Occurrence Handle10.1097/00000542-199807000-00005 Occurrence Handle9667288

    Article  PubMed  Google Scholar 

  • Georgia Certified Peer Specialist Project. (2004). Georgia Certified Peer Specialist Project. Georgia Mental Health Consumer Network, Division of Mental Health, Developmental Disabilities and Addictive Diseases. Available from: www.gacps.org

  • M.T. Guagnano D. Merlitti M.R. Manigrasso V. Pace-Palitti S. Sensi (2002) ArticleTitleNew medical licensing examination using computer-based case simulations and standardized patients Academic Medicine 77 IssueID1 87–90 Occurrence Handle11788331

    PubMed  Google Scholar 

  • W. Hall C. Violata R. Lewkonia J. Lockyer H. Fidler J. Toews et al. (1999) ArticleTitleAssessment of physician performance in Alberta: The physician achievement review Canadian Medical Association Journal 161 IssueID1 52–57 Occurrence Handle10420867

    PubMed  Google Scholar 

  • Hoge, M.A. & Morris, J.A. (Eds.) (2002). Behavioral health workforce education and training. [Special issue]. Administration and Policy in Mental Health, 29(4/5), 297–303

  • Hoge, M.A. & Morris, J.A. (Eds.) (2004). Implementing best practices in behavioral health workforce education: Building a change agenda [Special issue]. Administration and Policy in Mental Health, 32(2), 83–205.

    Google Scholar 

  • Hoge, M.A., Tondora, J., & Marelli, A.F. The fundamentals of workforce competency: Implications for behavioral health. Administration and Policy in Mental Health(in press)

  • Institute of Medicine. (2000). To err is human: Building a safer health system. Washington, DC: National Academy Press

  • Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st Century. Washington, DC: National Academy Press

  • International Certification & Reciprocity Consortium/Alcohol & Other Drug Abuse. (2004). Standards for certified alcohol and other drug abuse counselors. Available from: http://www.icrcaoda.org/

  • InstitutionalAuthorNameJoint Committee on Testing Practices (1999) Standards for educational and psychological testing American Educational Research Association, American Psychological Association National Council on Measurement in Education Washington, DC

    Google Scholar 

  • D. Juul S.C. Scheiber (1994) The part II psychiatry examination: Facts about the oral examination J.H. Shore S.C. Scheiber (Eds) Certification, recertification, and lifetime learning in psychiatry American Psychiatric Press Washington, DC 71–90

    Google Scholar 

  • T. Kennedy G. Regehr J. Rosenfield W. Roberts L. Lingard (2003) Degrees of gap between knowledge and behavior: A qualitative study of clinician action following an educational intervention American Educational Research Association Chicago, IL

    Google Scholar 

  • A. LaDuca (1994) ArticleTitleValidation of professional licensure examinations: Professions theory, test design, and construct validity Evaluation & the Health Professions 17 IssueID2 178–197

    Google Scholar 

  • B.E. Landon S.L. Normand D. Blumenthal J. Daley (2003) ArticleTitlePhysician clinical performance assessment: Prospects and barriers Journal of the American Medical Association 290 IssueID9 1183–1189 Occurrence Handle10.1001/jama.290.9.1183 Occurrence Handle12953001

    Article  PubMed  Google Scholar 

  • S.R. Lew G.G. Page L.W. Schuwirth M. Baron-Maldonado J.M. Lescop N. Paget et al. (2002) ArticleTitleProcedures for establishing defensible programmes for assessing practice performance Medical Education 36 IssueID10 936–941 Occurrence Handle10.1046/j.1365-2923.2002.01319.x Occurrence Handle12390461

    Article  PubMed  Google Scholar 

  • R.L. Linn (1994) ArticleTitlePerformance assessment: Policy promises and technical measurement standards Educational Researcher 23 4–14

    Google Scholar 

  • M.E. Lunz (1995) Statistical methods to improve decision reproducability E.L. Mancall P.G. Bashook (Eds) Assessing clinical reasoning: The oral examination and alternative methods American Board of Medical Specialties Evanston, IL 97–106

    Google Scholar 

  • M.E. Lunz J.A. Stahl B.D. Wright (1994) ArticleTitleInterjudge reliability and decision reproducibility Educational Psychological Measurement 54 IssueID4 913–925

    Google Scholar 

  • E.L. Mancall P.G. Bashook (Eds) (1995) Assessing clinical reasoning: The oral examination and alternative methods American Board of Medical Specialties Evanston, IL

    Google Scholar 

  • E.L. Mancall P.G. Bashook J.L. Dockery (1996) Computer-based examinations for board certification: Today’s opportunities and tomorrow’s possibilities American Board of Medical Specialties Evanston, IL

    Google Scholar 

  • J. McDermott P.T. Scheiber D. Juul J. Shore G. Tucker L. McCurdy et al. (1991) ArticleTitleReliability of the part II board certification examination in psychiatry: Inter-examiner consistency American Journal of Psychiatry 148 IssueID12 1672–1674 Occurrence Handle1957929

    PubMed  Google Scholar 

  • G.E. Miller (1990) ArticleTitleThe assessment of clinical skills/competence/performance Academic Medicine 65 S63–S67 Occurrence Handle2400509

    PubMed  Google Scholar 

  • Morris, J.A., Goplerud, E.N., & Hoge, M.A. (2004). Workforce issues in behavioral health. Institute of Medicine, Unpublished manuscript

  • D. Newble D. Dauphinee M. Macdonald H. Mulholland B. Dawson G. Page et al. (1994) ArticleTitleGuidelines for assessing clinical competence Teaching and Learning in Medicine 6 IssueID3 213–220

    Google Scholar 

  • G.R. Norman D.A. Davis S. Lamb E. Hanna P. Caulford T. Kaigas (1993) ArticleTitleCompetency assessment of primary care physicians as part of a peer review program Journal of American Medical Association 270 IssueID9 1046–1051 Occurrence Handle10.1001/jama.270.9.1046

    Article  Google Scholar 

  • G. R. Norman D.B. Swanson S.M. Case (1996) ArticleTitleConceptual and methodological issues in studies comparing assessment formats Teaching & Learning in Medicine 8 IssueID4 208–216

    Google Scholar 

  • P. O’Sullivan K. Cogbill T. McClain M. Reckase J. Clardy (2002) ArticleTitlePortfolios as a novel approach for residency evaluation Academic Psychiatry 26 IssueID3 173–178 Occurrence Handle10.1176/appi.ap.26.3.173 Occurrence Handle12824135

    Article  PubMed  Google Scholar 

  • G. Page (1995) Assessing reasoning and judgment E.L. Mancall P.G. Bashook (Eds) Assessing clinical reasoning: The oral examination and alternative methods American Board of Medical Specialties Evanston, IL 19–27

    Google Scholar 

  • G. Page G. Bordage T. Allen (1995) ArticleTitleDeveloping key-feature problems and examinations to assess clinical decision-making skills Academic Medicine 70 IssueID3 194–201 Occurrence Handle7873006

    PubMed  Google Scholar 

  • P.G. Ramsey M.D. Wenrich J.D. Carline T.S. Inui E.B. Larson J.P. LoGerfo (1993) ArticleTitleUse of peer ratings to evaluate physician performance Journal of American Medical Association 269 IssueID13 1655–1660 Occurrence Handle10.1001/jama.269.13.1655

    Article  Google Scholar 

  • G. Regehr H.M. MacRae R.K. Reznick D. Szalay (1998) ArticleTitleComparing the psychometric properties of checklists and global rating scales for assessing performance in an OSCE format examination Academic Medicine 73 IssueID9 993–997 Occurrence Handle9759104

    PubMed  Google Scholar 

  • J.J. Rethans J.J. Norcini M. Baron-Maldonado D. Blackmore B.C. Jolly T. LaDuca et al. (2002) ArticleTitleThe relationship between competence and performance: Implications for assessing practice performance Medical Education 36 IssueID10 901–909 Occurrence Handle10.1046/j.1365-2923.2002.01316.x Occurrence Handle12390456

    Article  PubMed  Google Scholar 

  • C. Roberts D.I. Newble A.J. O’Rourke (2002) ArticleTitlePortfolio-based assessments in medical education: Are they valid and reliable for summative purposes? Medical Education 36 IssueID10 899–900 Occurrence Handle10.1046/j.1365-2923.2002.01288.x Occurrence Handle12390455

    Article  PubMed  Google Scholar 

  • R.A. Roe (2002) ArticleTitleWhat makes a competent psychologist? European Psychologist 7 IssueID3 192–202 Occurrence Handle10.1027//1016-9040.7.3.192

    Article  Google Scholar 

  • Royal College of Physicians and Surgeons of Canada. (2004). Maintenance of certification. Available from: http://rcpsc.medical.org/

  • J.E. Sabin N. Daniels (2003) ArticleTitleStrengthening the consumer voice in managed care: VII. The Georgia peer specialist program Psychiatric Services 54 IssueID4 497–498 Occurrence Handle10.1176/appi.ps.54.4.497 Occurrence Handle12663836

    Article  PubMed  Google Scholar 

  • S.C. Scheiber T.A.M. Kramer S.E. Adamowski (2003) Core competencies for psychiatric practice American Psychiatric Publishing Arlington, VA

    Google Scholar 

  • L.W.T. Schuwirth L. Southgate G.G. Page N.S. Paget J.M.J. Lescop S.R. Lew et al. (2002) ArticleTitleWhen enough is enough: A conceptual basis for fair and defensible practice performance assessment Medical Education 36 IssueID10 925–930 Occurrence Handle10.1046/j.1365-2923.2002.01313.x Occurrence Handle12390459

    Article  PubMed  Google Scholar 

  • R.J. Shavelson N.M. Webb (1991) Generalizability theory: A primer Sage Publications Newbury Park CA

    Google Scholar 

  • Solomon, D., Reinhart, M., Bridgham, R., Munger, B., & Starnaman, S. (1990). An assessment of an oral examination format for evaluating clinical competence in emergency medicine. Academic Medicine, 65(S43–S44)

  • L.J. Southgate J. Cox T. David D. Hatch A. Howes N. Johnson et al. (2001) ArticleTitleThe general medical council’s performance procedures: Peer review of performance in the workplace Medical Education 35 IssueIDSuppl 1 9–19 Occurrence Handle11895258

    PubMed  Google Scholar 

  • D.B. Swanson G.R. Norman R.L. Linn (1995) ArticleTitlePerformance-based assessment: Lessons from the health professions Educational Researcher 24 IssueID5 5–11

    Google Scholar 

  • N. Taffinder C. Sutton R.J. Fishwick I.C. McManus A. Darzi (1998) Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: Results from randomised controlled studies using the MIST VR laparoscopic simulator J.D. Westwood H.M. Hoffman D. Stredney S.J. Weghorst (Eds) Medicine meets virtual reality IOS Press Amsterdam 124–130

    Google Scholar 

  • M.J. Weaver C.L. Ow D.J. Walker E.F. Degenhardt (1993) ArticleTitleA questionnaire for patients’ evaluations of their physicians’ humanistic behaviors Journal of General Internal Medicine 8 135–139 Occurrence Handle8455109

    PubMed  Google Scholar 

  • C. P. Winckel R.K. Reznick R. Cohen B. Taylor (1994) ArticleTitleReliability and construct validity of a structured technical skills assessment form American Journal of Surgery 167 IssueID4 423–427 Occurrence Handle10.1016/0002-9610(94)90128-7 Occurrence Handle8179088

    Article  PubMed  Google Scholar 

  • InstitutionalAuthorNameWorld Federation for Medical Education. (1998) ArticleTitleInternational standards in medical education: Assessment and accreditation of medical schools’ educational programmes: A WFME position paper Medical Education 32 549–558

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Medical Education, University of Illinois, 808 South Wood Street (MC591), Chicago, IL, 60612, USA

    Philip G. Bashook

Authors
  1. Philip G. Bashook
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philip G. Bashook.

Additional information

This work was supported in part by Contract No. 03M00013801D from the Substance Abuse and Mental Health Services Administration.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bashook, P.G. Best practices for assessing competence and performance of the behavioral Health workforce. APMH 32, 563–592 (2005). https://doi.org/10.1007/s10488-005-3265-z

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-005-3265-z

Keywords

Search

Navigation