Skip to main content

Validity of Clinician’s Self-Reported Practice Elements on the Monthly Treatment and Progress Summary

The Journal of Behavioral Health Services & Research volume 42, pages 367–382 (2015)Cite this article

Abstract

An increased demand for accountability in community mental health systems has resulted in a need for valid, reliable measures of therapeutic practice. The Monthly Treatment and Progress Summary (MTPS), developed through the Hawaii Child and Adolescent Mental Health Division, is a clinician-report measure that describes therapeutic practices, treatment targets, and progress ratings for each treatment case. The current study evaluated the validity of the therapeutic strategies reported on the MTPS by comparing coder- and clinician-reported use of practices. Using 47 audio recordings from 19 youths’ therapy sessions, trained observers reliably coded 12 discrete practices. Four of the 12 practices were found to be valid according to clinician–coder agreement (intraclass correlations ≥0.60). The coding system was revised, utilizing extensiveness and experiential scales, and 100 % of practices were valid according to clinician–coder agreement. Practical use of the MTPS, implications for service systems, and future directions for research on the MTPS are discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Figure 1

References

  1. Fixsen DL, Naoom SF, Blase KA, et al. Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, 2005.

  2. Kazdin AE. Evidence-based treatment and practice – New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 2008; 63(3):146–159.

    Google Scholar 

  3. National Advisory Mental Health Council. Workgroup on Child and Adolescent Mental Health Intervention Development and Deployment. Blueprint for change: research on child and adolescent mental health: Report of the National Advisory Mental Health Council’s Workgroup on Child and Adolescent Mental Health Intervention Development and Deployment. Washington (DC): The National Institute of Mental Health, Office of Communications and Public Liaison, 2001. Retrieved January 26, 2010 from www.nimh.nih.gov/child/blueprint.cfm.

  4. U.S. Department of Health and Human Services. Mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health, 1999.

  5. Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 2001; 52:685–716.

    Google Scholar 

  6. Hrisos S, Eccles M, Francis J, et al. Are there valid proxy measures of clinical behaviour? A systematic review. Implementation Science, 2009; 4(1):37.

    Google Scholar 

  7. McMillen JC, Lenze S, Hawley K, et al. Revisiting practice-based research networks as a platform for mental health services research. Administration and Policy in Mental Health and Mental Health Services Research, 2009; 36(5):308–321.

    Google Scholar 

  8. Bickman L. Our quality-assurance methods aren’t so sure. Behavioral Healthcare Tomorrow, 2000; 9(3):41–42.

    Google Scholar 

  9. Ridgely MS, Giard J, Shern D, et al. Managed behavioral health care: An instrument to characterize critical elements of public sector programs. Health Services Research, 2002; 37(4):1105–1123.

    Google Scholar 

  10. Callaghan G. Demonstrating clinical effectiveness for individual practitioners and clinics. Professional Psychology: Research and Practice, 2001; 32(3):289–297.

    Google Scholar 

  11. Ottenbacher K, Cusick A. Goal attainment scaling as a method of clinical service evaluation. American Journal of Occupational Therapy, 1990; 44(6):519–525.

    Google Scholar 

  12. McLeod BD. Therapy Process Observational Coding System for Child Psychotherapy. Los Angeles, CA: University of California, 2001.

  13. Weersing RV, Weisz JR, Donenberg GR. Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 2002; 31(2):168–180.

    Google Scholar 

  14. Hogue A, Dauber S, Chinchilla P, et al. Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 2008; 35(2):137–147.

    Google Scholar 

  15. Kazdin A, Siegel T, Bass D. Drawing on clinical practice to inform research on child and adolescent psychotherapy: Survey of practitioners. Professional Psychology: Research and Practice, 1990; 21(3):189–198.

    Google Scholar 

  16. Southam-Gerow M, Weisz J, Kendall P. Youth with anxiety disorders in research and service clinics: Examining client differences and similarities. Journal of Clinical Child and Adolescent Psychology, 2003; 32(3):375–385.

    Google Scholar 

  17. Cicchetti D, Sparrow S. Developing criteria for establishing inter-rater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 1981; 86(2):127–137.

    Google Scholar 

  18. Garland AF, Brookman-Frazee L, Hurlburt M, et al. Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 2010; 61:788–795.

    Google Scholar 

  19. Hurlburt MS, Garland AF, Nguyen K, et al. Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 2010; 37:230–244.

    Google Scholar 

  20. Anastasi A. The concept of validity in the interpretation of test scores. Educational and Psychological Measurement, 1950; 10(1):67–78.

    Google Scholar 

  21. Chorpita BF, Daleiden EL, Weisz JR. Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 2005; 7(1):5–20.

    Google Scholar 

  22. Chorpita BF, Daleiden EL. Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 2009; 77(3):566–579.

    Google Scholar 

  23. Garland AF, Hawley KM, Brookman-Frazee L, et al. Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. Journal of the American Academy of Child and Adolescent Psychiatry, 2008; 47(5):505–514.

    Google Scholar 

  24. Garland AF, Hurlburt MS, Hawley KM. Examining psychotherapy processes in a services research context. Clinical Psychology: Science and Practice, 2006; 13(1):30–46.

    Google Scholar 

  25. McLeod BD, Weisz JR. The therapy process observational coding system-alliance scale: measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting and Clinical Psychology, 2005; 73(2):323–333.

    Google Scholar 

  26. Borntrager C, Chorpita BF, Higa-McMillan CK, et al. Provider attitudes towards evidence-based practices: Are the concerns with the evidence or with the manuals? Psychiatric Services, 2008; 60(5):1–5.

    Google Scholar 

  27. Walker P, Weersing VR, Warnick EM, et al. Predictors of youth therapist technique use in community care. Poster presented at the annual meeting of the Association for Behavioral and Cognitive Therapies, Orlando, FL, November, 2008

  28. Child and Adolescent Mental Health Division. Service Provider Monthly Treatment and Progress Summary. Honolulu: HI: Hawaii Department of Health Child and Adolescent Mental Health Division, 2003.

    Google Scholar 

  29. Nakamura BJ, Daleiden EL, Mueller CW. Validity of treatment target progress ratings as indicators of youth improvement. Journal of Child and Family Studies, 2007; 16(5):729–734.

    Google Scholar 

  30. Daleiden E, Lee J, Tolman R. Child and Adolescent Mental Health Division: 2004 Annual Report. Honolulu, HI: Child and Adolescent Mental Health Division, 2004. Retrieved October 30, 2008, from http://www.hawaii.gov/health/mental-health/camhd/library/pdf/paf/paf-002.pdf

  31. Chorpita BF, Daleiden EL. 2007 Biennial report: Effective psychosocial interventions for youth with behavioral and emotional needs. Honolulu, HI: Child and Adolescent Mental Health Division, Hawaii Department of Health, 2007.

  32. Daleiden E, Chorpita BF. From data to wisdom: Quality improvement strategies supporting large-scale implementation of evidence based services. Child and Adolescent Psychiatric Clinics of North America, 2005; 14:329–349.

    Google Scholar 

  33. Daleiden E, Tolman R. Annual evaluation report: Fiscal year 2005. Honolulu, HI: Hawaii Department of Health Child and Adolescent Mental Health Division, 2005.

  34. Higa-McMillan CK, Daleiden EL, Pestle SL, et al. Evidence-based practice and practice-based evidence: Using local and national data to encourage youth provider change in a public mental health system. Presented as part of a symposium at the Association for Behavioral and Cognitive Therapies, Orlando, FL, November, 2008.

  35. Jensen J, Bergin A, Greaves D. The meaning of eclecticism: New survey and analysis of components. Professional Psychology: Research and Practice. 1990; 21(2):124–130.

    Google Scholar 

  36. Norcross J. The psychotherapist’s own psychotherapy: Educating and developing psychologists. American Psychologist, 2005; 60(8):840–850.

    Google Scholar 

  37. Weisz J, Chorpita B.F, Palinkas L, et al. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth. Archives of General Psychiatry, 2011: E1-E9.

  38. Child and Adolescent Mental Health Division. Service Provider Monthly Treatment and Progress Summary. Honolulu: HI: Hawaii Department of Health Child and Adolescent Mental Health Division, 2008. Retrieved October 30, 2008, from http://hawaii.gov/health/mental-health/camhd/library/pdf/paf/paf-002.pdf

  39. Child and Adolescent Mental Health Division. Instructions and Codebook for Service Provider Monthly Treatment and Progress Summary. Honolulu: HI: Hawaii Department of Health Child and Adolescent Mental Health Division, 2008. Retrieved October 30, 2008, from http://hawaii.gov/health/mental-health/camhd/library/pdf/paf/paf-001.pdf

  40. Marder AM, Borntrager CF, Southam-Gerow MA, et al. Relationships between therapist attitudes toward evidence-based treatments and initial adherence to prescribed interventions with youth. Presented as part of a symposium at the Association for Behavioral and Cognitive Therapies, Orlando, FL, November, 2008.

  41. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 1979; 86(2):420–428.

    Google Scholar 

Download references

Acknowledgments

The preparation of this manuscript was supported, in kind, through the MacArthur Foundation granted to PIs: Bruce Chorpita and John Weisz. In addition, this work was supported in part by a research and evaluation contract from the State of Hawaii Department of Health, Child and Adolescent Mental Health Division to Charles Mueller.

Conflict of Interest

The authors report no conflicts of or competing interests.

Author information

Affiliations

  1. Department of Psychology, University of Montana, 32 Campus Dr., Skaggs Bldg. 143, Missoula, MT, 59812, USA

    Cameo F. Borntrager PhD

  2. Department of Psychology, University of California Los Angeles, 3243 Franz Hall, Box 951563, Los Angeles, CA, 90095, USA

    Bruce F. Chorpita PhD

  3. Department of Psychology, University of Hawaii, 2430 Campus Rd., Gartley 110, Honolulu, HI, 96822, USA

    Trina Orimoto MA, Allison Love MA & Charles W. Mueller PhD

Authors
  1. Cameo F. Borntrager PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruce F. Chorpita PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Trina Orimoto MA
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Allison Love MA
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Charles W. Mueller PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cameo F. Borntrager PhD.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Borntrager, C.F., Chorpita, B.F., Orimoto, T. et al. Validity of Clinician’s Self-Reported Practice Elements on the Monthly Treatment and Progress Summary. J Behav Health Serv Res 42, 367–382 (2015). https://doi.org/10.1007/s11414-013-9363-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-013-9363-x

Keywords

  • Usual Care
  • Youth Mental Health
  • Therapeutic Practice
  • Practice Element
  • Community Mental Health Clinic