Skip to main content
SpringerLink
Log in

Evidence for the Validity of the Psychopharmacotherapy-Structured Clinical Observation Tool: Results of a Factor and Time Series Analysis

  • Empirical Report
  • Published:
Academic Psychiatry Aims and scope Submit manuscript

Abstract

Objective

The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance of a medication visit. This study examines two dimensions of validity for the P-SCO: internal structure and how scores correlate with another variable associated with competence (experience).

Methods

The faculty completed 601 P-SCOs over 4 years. Multilevel exploratory factor analysis was performed with minimum thresholds for eigenvalue (≥ 1.0) and proportion of variance explained (≥ 5.0%). Internal reliability was assessed with Cronbach alpha. To examine how scores changed with experience, mean ratings (1–4 scale) were calculated for each factor by quarter of the academic year. Separate linear mixed models were also performed.

Results

The analysis yielded three factors that explained 50% of the variance and demonstrated high internal reliability: affective tasks (alpha = 0.90), cognitive tasks (alpha = 0.84), and hard tasks (alpha = 0.74). Items within “hard tasks” were assessment of substance use, violence risk, and adherence, and inquiry about interactions with other providers. Monitoring adverse effects did not load on the hard task factor but also had overall low mean ratings. Compared to the first quarter, fourth quarter scores for affective tasks (b = 0.54, p < 0.01) and hard tasks (b = 0.46, p = 0.02) were significantly improved while cognitive tasks had a non-significant increase. For the hard tasks, the proportion of residents with a low mean rating improved but was still over 30% during the fourth quarter.

Conclusions

The results provide evidence for the validity of the P-SCO with respect to its internal structure and how scores correlate with experience. Curricular implications are explored, especially for the tasks that were hard to learn.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Halman S, Dudek N, Wood T, Pugh D, Touchie C, McAleer S, et al. Direct observation of clinical skills feedback scale: development and validity evidence. Teach Learn Med. 2016;28(4):385–94.

    Article  Google Scholar 

  2. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;341:c5064.

    Article  Google Scholar 

  3. Al Ansari A, Ali SK, Donnon T. The construct and criterion validity of the mini-CEX: a meta-analysis of the published research. Acad Med. 2013;88(3):413–20.

    Article  Google Scholar 

  4. Leep Hunderfund AN, Rubin DI, Laughlin RS, Sorenson EJ, Watson JC, Jones LK, et al. Validity and feasibility of the EMG direct observation tool (EMG-DOT). Neurology. 2016;86(17):1627–34.

    Article  Google Scholar 

  5. Olupeliyawa AM, O’Sullivan AJ, Hughes C, Balasooriya CD. The Teamwork Mini-Clinical Evaluation Exercise (T-MEX): a workplace-based assessment focusing on collaborative competencies in health care. Acad Med. 2014;89(2):359–65.

    Article  Google Scholar 

  6. Watanabe Y, Bilgic E, Lebedeva E, McKendy KM, Feldman LS, Fried GM, et al. A systematic review of performance assessment tools for laparoscopic cholecystectomy. Surg Endosc. 2016;30(3):832–44.

    Article  Google Scholar 

  7. Watson MJ, Wong DM, Kluger R, Chuan A, Herrick MD, Ng I, et al. Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia. Anaesthesia. 2014;69(6):604–12.

    Article  CAS  Google Scholar 

  8. Feraco AM, Starmer AJ, Sectish TC, Spector ND, West DC, Landrigan CP. Reliability of verbal handoff assessment and handoff quality before and after implementation of a resident handoff bundle. Acad Pediatr. 2016;16(6):524–31.

    Article  Google Scholar 

  9. Jibson MD, Broquet KE, Anzia JM, Beresin EV, Hunt JI, Kaye D, et al. Clinical skills verification in general psychiatry: recommendations of the ABPN Task Force on Rater Training. Acad Psychiatry. 2012;36(5):363–8.

    Article  Google Scholar 

  10. Hung EK, Binder RL, Fordwood SR, Hall SE, Cramer RJ, McNiel D. A method for evaluating competency in assessment and management of suicide risk. Acad Psychiatry. 2012;36(1):23–8.

    Article  Google Scholar 

  11. McNiel DE, Hung EK, Cramer RJ, Hall SE, Binder RL. An approach to evaluating competence in assessing and managing violence risk. Psychiatr Serv. 2011;62(1):90–2.

    Article  Google Scholar 

  12. Weerasekera P, Antony MM, Bellissimo A, Bieling P, Shurina-Egan J, Spencer A, et al. Competency assessment in the McMaster Psychotherapy Program. Acad Psychiatry. 2003;27(3):166–73.

    Article  Google Scholar 

  13. Young JQ, Lieu S, O'Sullivan P, Tong L. Development and initial testing of a structured clinical observation tool to assess pharmacotherapy competence. Acad Psychiatry. 2011;35(1):27–34.

    Article  Google Scholar 

  14. Mojtabai R, Olfson M. National trends in psychotherapy by office-based psychiatrists. Arch Gen Psychiatry. 2008;65(8):962–70.

    Article  Google Scholar 

  15. Young JQ, Hasser C, Hung EH, Kusz M, O’Sullivan PS, Stewart C, Weiss A, Williams N. Developing end-of-training entrustable professional activities for psychiatry: results and methodological lessons. Acad Med. 2017; 1.

  16. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7.

    Article  Google Scholar 

  17. Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv Health Sci Educ. 2010;15(5):633–45.

    Article  Google Scholar 

  18. Hojat M, Paskin DL, Callahan CA, Nasca TJ, Louis DZ, Veloski J, et al. Components of postgraduate competence: analyses of thirty years of longitudinal data. Med Educ. 2007;41(10):982–9.

    Article  Google Scholar 

  19. Gilley WF, Uhlig GE. Factor analysis and ordinal data. Education. 1993;114:258–65.

    Google Scholar 

  20. Joreskog KG, Moustaki I. Factor analysis of ordinal variables: a comparison of three approaches. Multivar Behav Res. 2001;36(3):347–87.

    Article  CAS  Google Scholar 

  21. Reise SP, Ventura J, Nuechterlein KH, Kim KH. An illustration of multilevel factor analysis. J Pers Assess. 2005;84(2):126–36.

    Article  Google Scholar 

  22. Revelle W. Procedures for personality and psychological research. 2017, Northwestern University: https://cran.r-project.org/package=psych.

  23. Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90(11):1471–86.

    Article  Google Scholar 

  24. Young JQ, Nelson JC. Reconceptualizing medication management: implications for training and clinical practice. J Clin Psychiatry. 2009;70(12):1722–3.

    Article  Google Scholar 

  25. Young JQ, van Merrienboer J, Durning S, ten Cate O. Cognitive load theory: implications for medical education: AMEE guide no. 86. Med Teach. 2014;36(5):371–84.

    Article  Google Scholar 

  26. Moonen-van Loon JM, et al. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 2013;18(5):1087–102.

    Article  CAS  Google Scholar 

Download references

Funding

ABPN Research Award

Author information

Authors and Affiliations

  1. Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA

    John Q. Young

  2. Northwell Health, Great Neck, NY, USA

    Rehana Rasul

  3. UCSF School of Medicine, San Francisco, CA, USA

    Patricia S. O’Sullivan

Authors
  1. John Q. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rehana Rasul
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patricia S. O’Sullivan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John Q. Young.

Ethics declarations

The IRB for the first author’s institution determined that the study was exempt.

Disclosures

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Young, J.Q., Rasul, R. & O’Sullivan, P.S. Evidence for the Validity of the Psychopharmacotherapy-Structured Clinical Observation Tool: Results of a Factor and Time Series Analysis. Acad Psychiatry 42, 759–764 (2018). https://doi.org/10.1007/s40596-018-0928-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40596-018-0928-0

Keywords

Search

Navigation