Abstract
Objective
The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) tool is designed to assess performance of a medication management visit and to enhance feedback. Prior research indicated that the P-SCO was feasible to implement in a resident clinic and generated behaviorally specific, high-quality feedback. This research also highlighted problems with some of the instrument’s items. This study seeks to improve the items.
Methods
The authors initially revised the P-SCO items based on the problems identified by a prior study. Next, these items were iteratively modified by experts in clinical pharmacotherapy and educational assessment. Forty-five items emerged. Finally, faculty attending an annual department education retreat rated each item on its relevance (4-point scale) and provided comments on how the item might be revised. For final inclusion, an item must have met a quantitative threshold (i.e., content validity index equal to or greater than 0.8 and the lower end of the asymmetric confidence interval equal to or greater than 3.0) and received comments that were supportive.
Results
Forty-one of the 45 items had strong quantitative support. However, the comments endorsed lumping a number of items in order to decrease overlap between items and to shorten the instrument. This process resulted in the further elimination of 15 items.
Conclusions
The revised 26-item P-SCO builds upon prior evidence of feasibility and utility and now possesses additional evidence of content validity. The use of the tool should enhance feedback and improve the capacity of educational programs to assess performance.
This is a preview of subscription content, log in via an institution to check access.
References
Olfson M, Blanco C, Wang S, Laje G, Correll CU. National trends in the mental health care of children, adolescents, and adults by office-based physicians. JAMA Psychiatry. 2014;71(1):81–90. https://doi.org/10.1001/jamapsychiatry.2013.3074.
Institute of Medicine (U.S.). Committee on crossing the quality chasm: adaptation to mental health and addictive disorders., Improving the quality of health care for mental and substance-use conditions. Quality chasm series. 2006, Washington, DC: National Academies Press. xxiii, 504 pages.
Young JQ, et al. Developing end-of-training EPAs for psychiatry: results and methodological lessons. Acad Med. 2017 Nov 21. https://doi.org/10.1097/ACM.0000000000002058.
Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45. https://doi.org/10.3109/0142159X.2010.501190.
Leach DC. A model for GME: shifting from process to outcomes. A progress report from the Accreditation Council for Graduate Medical Education. Med Educ. 2004;38(1):12–4. https://doi.org/10.1111/j.1365-2923.2004.01732.x.
Young JQ, Nelson JC. Reconceptualizing medication management: implications for training and clinical practice. J Clin Psychiatry. 2009;70(12):1722–3. https://doi.org/10.4088/JCP.09ac05828whi.
Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90(4):411–3. https://doi.org/10.1097/ACM.0000000000000515.
Zisook S, Benjamin S, Balon R, Glick I, Louie A, Moutier C, et al. Alternate methods of teaching psychopharmacology. Acad Psychiatry. 2005;29(2):141–54. https://doi.org/10.1176/appi.ap.29.2.141.
Juul D, Winstead DK, Sheiber SC. Assessment of psychopharmacology on the American Board of Psychiatry and Neurology examinations. Acad Psychiatry. 2005;29(2):211–4. https://doi.org/10.1176/appi.ap.29.2.211.
Young JQ, Lieu S, O'Sullivan P, Tong L. Development and initial testing of a structured clinical observation tool to assess pharmacotherapy competence. Acad Psychiatry. 2011;35(1):27–34. https://doi.org/10.1176/appi.ap.35.1.27.
Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7. https://doi.org/10.1046/j.1365-2923.2003.01594.x.
Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382–5.
Penfield RD. A score method of constructing asymmetric confidence intervals for the mean of a rating scale item. Psychol Methods. 2003;8(2):149–63. https://doi.org/10.1037/1082-989X.8.2.149.
Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30(4):459–67. https://doi.org/10.1002/nur.20199.
Sanchez R, Sloan SR, Josephson CD, Ambruso DR, Hillyer CD, O'Sullivan P. Consensus recommendations of pediatric transfusion medicine objectives for clinical pathology residency training programs. Transfusion. 2010;50(5):1071–8. https://doi.org/10.1111/j.1537-2995.2009.02549.x.
Thrush CR, Putten JV, Rapp CG, Pearson LC, Berry KS, O'Sullivan PS. Content validation of the Organizational Climate for Research Integrity (OCRI) survey. J Empir Res Hum Res Ethics. 2007;2(4):35–52. https://doi.org/10.1525/jer.2007.2.4.35.
Moonen-van Loon JM, et al. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 2013;18(5):1087–102. https://doi.org/10.1007/s10459-013-9450-z.
Ten Cate O, et al. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach. 2015;37(11):983–1002. https://doi.org/10.3109/0142159X.2015.1060308.
Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: rater performance and behavior when considering multiple competencies. Teach Learn Med. 2016;28(1):41–51. https://doi.org/10.1080/10401334.2015.1107489.
Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943–54. https://doi.org/10.1111/medu.13067.
Funding
This study is supported by the ABPN Research Award.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The IRB for the first author’s institution determined that the study was exempt.
Disclosure
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Rights and permissions
About this article
Cite this article
Young, J.Q., Irby, D.M., Kusz, M. et al. Performance Assessment of Pharmacotherapy: Results from a Content Validity Survey of the Psychopharmacotherapy-Structured Clinical Observation (P-SCO) Tool. Acad Psychiatry 42, 765–772 (2018). https://doi.org/10.1007/s40596-017-0876-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40596-017-0876-0