Academic Psychiatry

, Volume 35, Issue 1, pp 27–34 | Cite as

Development and Initial Testing of a Structured Clinical Observation Tool to Assess Pharmacotherapy Competence

  • John Q. Young
  • Sandra Lieu
  • Patricia O’Sullivan
  • Lowell Tong
Original Article



The authors developed and tested the feasibility and utility of a new direct-observation instrument to assess trainee performance of a medication management session.


The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) instrument was developed based on multiple sources of expertise and then implemented in 4 university-based outpatient medication management clinics with 7 faculty supervising 17 third-year residents. After each observation by a faculty member of a medication management session, residents received feedback in writing (the completed P-SCO) and verbally in person. Targets were 8 P-SCO observations per academic year per resident (or 0.67 per month) and 16 observations per year completed by each faculty (or 1.3 per month). Qualitative thematic analysis was employed to compare the frequency, specificity, type (reinforcing vs. corrective), and content of comments documented on the P-SCO forms to midpoint and end of rotation global assessments by the same faculty for the same residents in the same rotation.


Faculty completed 2.4 (SD=1,2) P-SCOs per month during the study period. Each resident received 1.1 (SD=0.53) P-SCO observations per month. Faculty and residents completed significantly more observations than targeted (p=0.03 and p=0.003, respectively). Two percent of the P-SCOs had no written comments. Less than 3% of the P-SCO comments were nonspecific compared with 43% for the global assessments. Residents received, on average, 3.3 times more total, 2.6 times more reinforcing, and 5.3 times more corrective patient care specific comments on the P-SCO than on the global assessment (p<0.001). For the numerical ratings, residents received an average of 4.2 “exceeds expectations” and 1.7 “below expectations” ratings on P-SCOs compared with 2.6 and 0, respectively, on global assessments (p<0.02).


Faculty can feasibly use the P-SCO instrument in a training clinic. Compared with traditional global assessment, the P-SCO provided much more specific feedback information, a better balance of corrective to re-enforcing comments, and a greater spread of ratings related to competency in pharmacodverbally


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mojtabai R, Olfson M: National trends in psychotherapy by office-based psychiatrists. Arch Gen Psychiatry 2008; 65: 962–970PubMedCrossRefGoogle Scholar
  2. 2.
    US Institute of Medicine: Committee on Crossing the Quality Chasm: adaptation to mental health and addictive disorders, improving the quality of health care for mental and substance-use conditions. Quality Chasm Series. Washington, DC, National Academies Press, 2006Google Scholar
  3. 3.
    Daelmans HE, Overmeer RM, van der Hem-Stokroos HH, et al: In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006; 40:51–58PubMedCrossRefGoogle Scholar
  4. 4.
    Daelmans HE, van der Hem-Stokroos HH, Hoogenboom RJ, et al: Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship. Med Educ 2004; 38:1270–1277PubMedCrossRefGoogle Scholar
  5. 5.
    Turnbull J, MacFadyen J, Van Barneveld C, et al: Clinical work sampling: a new approach to the problem of in-training evaluation. J Gen Intern Med 2000; 15:556–561PubMedCentralPubMedCrossRefGoogle Scholar
  6. 6.
    Hays R, Wellard R: In-training assessment in postgraduate training for general practice. Med Educ 1998; 32:507–513PubMedCrossRefGoogle Scholar
  7. 7.
    Dunnington G, Reisner L, Witzke D, et al: Structured single-observer methods of evaluation for the assessment of ward performance on the surgical clerkship. Am J Surg 1990; 159:423–426PubMedCrossRefGoogle Scholar
  8. 8.
    Wass V, Jolly B: Does observation add to the validity of the long case? Med Educ 2001; 35:729–734PubMedCrossRefGoogle Scholar
  9. 9.
    Downing SM: Validity: on meaningful interpretation of assessment data. Med Educ 2003; 37:830–837PubMedCrossRefGoogle Scholar
  10. 10.
    Irby DM: What clinical teachers in medicine need to know. Acad Med 1994; 69:333–342PubMedCrossRefGoogle Scholar
  11. 11.
    Schum TR, Krippendorf RL, Biernat KA: Simple feedback notes enhance specificity of feedback to learners. Ambul Pediatr 2003; 3:9–11PubMedCrossRefGoogle Scholar
  12. 12.
    Schatzberg AF, Nemeroff CB: The American Psychiatric Publishing Textbook of Psychopharmacology, 3rd ed. Washington, DC, American Psychiatric Publishing, 2004Google Scholar
  13. 13.
    Janicak PG, Davis JM, Preskorn SH, et al: Principles and Practice of Psychopharmacotherapy, 4th ed. Philadelphia, Lippincott Williams & Wilkins, 2006Google Scholar
  14. 14.
    Scheiber SC, Kramer TAM: Core Competencies for Psychiatric Practice: What Clinicians Need To Know: A Report of the American Board of Psychiatry and Neurology, 1st ed. Washington, DC, American Psychiatric Publishing, 2003Google Scholar
  15. 15.
    ACGME: ACGME Psychiatry Program Requirements, 2007. Available at
  16. 16.
    Fawcett J, Epstein P, Fiester SJ, et al: Clinical management: imipramine/placebo administration manual. NIMH Treatment of Depression Collaborative Res Program. Psychopharmacol Bull 1987; 23:309–324PubMedGoogle Scholar
  17. 17.
    Miller M: Medication Clinic Training Procedures and Treatment Manual. Pittsburgh, University of Pittsburgh Medical Center Late-Life Depression Prevention Clinic, 1998, pp 1–21Google Scholar
  18. 18.
    Miller MD, Frank E, Reynolds CF 3rd: The art of clinical management in pharmacologic trials with depressed elderly patients: lessons from the Pittsburgh Study of Maintenance Therapies in Late-Life Depression. Am J Geriatr Psychiatry 1999; 7:228–234PubMedCrossRefGoogle Scholar
  19. 19.
    Elstein AS, Shulman LS, Sprafka SA: Medical problem solving: an analysis of clinical reasoning. Cambridge, Mass, Harvard University Press, 1978, pp xvi, 330Google Scholar
  20. 20.
    Williams RG, Dunnington GL, Klamen DL: Forecasting residents’ performance-partly cloudy. Acad Med 2005; 80:415–422PubMedCrossRefGoogle Scholar
  21. 21.
    Norcini JJ, Blank LL, Arnold GK, et al: The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med 1995; 123:795–799PubMedCrossRefGoogle Scholar
  22. 22.
    Kroboth FJ, Hanusa BH, Parker S, et al: The inter-rater reliability and internal consistency of a clinical evaluation exercise. J Gen Intern Med 1992; 7:174–179PubMedCrossRefGoogle Scholar
  23. 23.
    Wilkinson TJ, Campbell PJ, Judd SJ: Reliability of the long case. Med Educ 2008; 42:887–893PubMedCrossRefGoogle Scholar
  24. 24.
    Salerno SM, Jackson JL, O’Malley PG: Interactive faculty development seminars improve the quality of written feedback in ambulatory teaching. J Gen Intern Med 2003; 18: 831–834PubMedCentralPubMedCrossRefGoogle Scholar
  25. 25.
    Ende J: Feedback in clinical medical education. JAMA 1983; 250:777–781PubMedCrossRefGoogle Scholar
  26. 26.
    Smith CS, Francovich C, Gieselman J, et al: A broader theoretical model for feedback in ambulatory care. Adv Health Sci Educ Theory Pract 1998; 3:133–140PubMedCrossRefGoogle Scholar
  27. 27.
    Hewson MG, Little ML: Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med 1998; 13:111–116PubMedCentralPubMedCrossRefGoogle Scholar
  28. 28.
    Heidenreich C, Lye P, Simpson D, et al: The search for effective and efficient ambulatory teaching methods through the literature. Pediatrics 2000; 105(part 3):231–237PubMedGoogle Scholar
  29. 29.
    Lye PS, Biernat KA, Bragg DS, et al: A pleasure to work with: an analysis of written comments on student evaluations. Ambul Pediatr 2001; 1:128–131PubMedCrossRefGoogle Scholar
  30. 30.
    Paukert JL, Richards ML, Olney C: An encounter card system for increasing feedback to students. Am J Surg 2002; 183:300–304PubMedCrossRefGoogle Scholar
  31. 31.
    Holmboe ES, Fiebach NH, Galaty LA, et al: Effectiveness of a focused educational intervention on resident evaluations from faculty a randomized controlled trial. J Gen Intern Med 2001; 16:427–434PubMedCentralPubMedCrossRefGoogle Scholar
  32. 32.
    Holmboe ES, Yepes M, Williams F, et al: Feedback and the mini clinical evaluation exercise. J Gen Intern Med 2004; 19(part 2):558–561PubMedCentralPubMedCrossRefGoogle Scholar

Copyright information

© Academic Psychiatry 2011

Authors and Affiliations

  • John Q. Young
    • 1
  • Sandra Lieu
    • 1
  • Patricia O’Sullivan
    • 2
  • Lowell Tong
    • 1
  1. 1.Department of PsychiatryUCSF School of MedicineSan Francisco
  2. 2.Office of Medical EducationUCSF School of MedicineSan Francisco

Personalised recommendations