Abstract
Therapist-report measures of evidence-based interventions have enormous potential utility as quality indicators in routine care; yet, few such tools have shown strong psychometric properties. This study describes reliability and validity characteristics of a therapist-report measure of family therapy techniques for treating adolescent conduct and substance use problems: Inventory of Therapy Techniques for Core Elements of Family Therapy (ITT-CEFT). Study participants included 31 staff therapists treating 68 adolescent clients in eight community-based mental health and substance use clinics. Therapists submitted ITT-CEFT checklists and companion audio recordings for 189 sessions. The ITT-CEFT contains 13 techniques identified as core elements of three manualized family therapy models that are empirically supported for the target group. Therapists also reported on their use of three motivational interventions, and independent observers coded the submitted recordings. ITT-CEFT factor validity was shown via confirmatory factor analyses of the tool’s theoretical structure. Derived modules were: Family Engagement (four items; Cronbach’s α = .72); Relational Orientation (five items; α = .74); and Interactional Change (four items; α = .66). Concurrent validity analyses showed fair-to-excellent therapist reliability compared to observer ratings (ICCs range .64–.75); they showed moderate therapist accuracy compared to observer mean scores, reflecting a tendency to overestimate delivery of the techniques. Discriminant validity analyses showed tool differentiation from motivational interventions. Results offer provisional evidence for the feasibility of using the therapist-report ITT-CEFT to anchor quality procedures for family therapy interventions in real-world settings.
Trial Registration: The parent clinical trial is registered at www.ClinicalTrials.gov, ID: NCT03342872 (registration date: 11.10.17).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Asparouhov, T. (2005). Sampling weights in latent variable modeling. Structural Equation Modeling, 12, 411–434. https://doi.org/10.1207/s15328007sem1203_4
Browne, M., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. Bollen & J. Long (Eds.), Testing Structural Models. Newbury Park, CA: Sage Publications.
Baldwin, S. A., Christian, S., Berkeljon, A., & Shadish, W. R. (2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy, 38, 281–304. https://doi.org/10.1111/j.1752-0606.2011.00248.x
Barth, R. P., Kolivoski, K. M., Lindsey, M. A., Lee, B. R., & Collins, K. S. (2014). Translating the common elements approach: Social work’s experiences in education, practice, and research. Journal of Clinical Child & Adolescent Psychology, 43, 301–311. https://doi.org/10.1080/15374416.2013.848771
Becker-Haimes, E. M., Klein, M. R., McLeod, B. D., Schoenwald, S. K., Dorsey, S., Hogue, A., … Beidas, R. S. (2021). The TPOCS-self-reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY): A case study of pragmatic measure development. Implementation Research and Practice. https://doi.org/10.1177/2633489521992553
Borntrager, C. F., Chorpita, B. F., Orimoto, T., Love, A., & Mueller, C. W. (2015). Validity of clinician’s self-reported practice elements on the monthly treatment and progress summary. The Journal of Behavioral Health Services & Research, 42(3), 367–382. https://doi.org/10.1007/s11414-013-9363-x
Brookman-Frazee, L., Stadnick, N. A., Lind, T., Roesch, S., Terrones, L., Barnett, M. L., … Lau, A. S. (2020). Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children’s Mental Health Services. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01054-x
Chapman, J. E., McCart, M. R., Letourneau, E. J., Sheidow, A., & J. . (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting and Clinical Psychology, 81, 674–680. https://doi.org/10.1037/a0033021
Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566–579. https://doi.org/10.1037/a0014565
Chorpita, B., Daleiden, E., Ebesutani, C., Young, J., Becker, K., Nakamura, B., et al. (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18, 154–172. https://doi.org/10.1111/j.1468-2850.2011.01247.x
Chorpita, B. F., Daleiden, E., & Weisz, J. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7, 5–20. https://doi.org/10.1007/s11020-005-1962-6
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. https://doi.org/10.1037/1040-3590.6.4.284
Cook, J. M., Biyanova, T., Elhai, J., Schnurr, P. P., & Coyne, J. C. (2010). What do psychotherapists really do in practice? An internet study of over 2000 practitioners. Psychotherapy: Theory, Research, Practice, Training, 47, 260–267. https://doi.org/10.1037/a0019788
Davey, M., Duncan, T., Kissil, K., Davey, A., & Fish, L. S. (2011). Second-order change in marriage and family therapy: A web-based modified Delphi study. American Journal of Family Therapy, 39, 100–111. https://doi.org/10.1080/01926187.2010.530929
Dopp, A. R., Borduin, C. M., White, I., Mark, H., & Kuppens, S. (2017). Family-based treatments for serious juvenile offenders: A multilevel meta-analysis. Journal of Consulting and Clinical Psychology, 85, 335. https://doi.org/10.1037/ccp0000183
Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105–112. https://doi.org/10.1177/014662168100500115
Frank, H. E., Becker-Haimes, E. M., & Kendall, P. C. (2020). Therapist training in evidence-based interventions for mental health: A systematic review of training approaches and outcomes. Clinical Psychology: Science and Practice, 27(3), e12330. https://doi.org/10.1111/cpsp.12330
Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research, 37, 208–220. https://doi.org/10.1007/s10488-009-0237-8
Gifford, E. V., Tavakoli, S., Weingardt, K. R., Finney, J. W., Pierson, H. M., et al. (2012). How do components of evidence-based psychological treatment cluster in practice? A survey and cluster analysis. Journal of Substance Abuse Treatment, 42, 45–55. https://doi.org/10.1016/j.jsat.2011.07.008
Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures: What they are and why we need them. American Journal of Preventive Medicine, 45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23. https://doi.org/10.20982/tqmp.08.1.p023
Henderson, C. E., Hogue, A., & Dauber, S. (2019). Family therapy techniques and one-year clinical outcomes among adolescents in usual care for behavior problems. Journal of Consulting and Clinical Psychology, 87, 308–312. https://doi.org/10.1037/ccp0000376
Henggeler, S. W., & Schaeffer, C. M. (2016). Multisystemic therapy: Clinical overview, outcomes, and implementation research. Family Process, 55, 514–528. https://doi.org/10.1111/famp.12232
Herschell, A. D., Quetsch, L. B., & Kolko, D. J. (2020). Measuring adherence to key teaching techniques in an evidence-based treatment: A comparison of caregiver, therapist, and behavior observation ratings. Journal of Emotional and Behavioral Disorders, 28(2), 92–103. https://doi.org/10.1177/1063426618821901
Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation of policy, research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 40, 451–455. https://doi.org/10.1007/s10488-013-0512-6
Hogue, A., Bobek, M., Dauber, S., Henderson, C. E., McLeod, B. D., & Southam-Gerow, M. A. (2017a). Distilling the core elements of family therapy for adolescent substance use: Conceptual and empirical solutions. Journal of Child & Adolescent Substance Abuse, 26, 437–453. https://doi.org/10.1080/1067828x.2017.1322020
Hogue, A., Bobek, M., Dauber, S., Henderson, C. E., McLeod, B. D., & Southam-Gerow, M. A. (2019). Core elements of family therapy for adolescent behavior problems: Empirical distillation of three manualized treatments. Journal of Clinical Child and Adolescent Psychology, 48, 29–41. https://doi.org/10.1080/15374416.2018.1555762
Hogue, A., Dauber, S., & Henderson, C. E. (2014a). Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research, 41, 126–139. https://doi.org/10.1007/s10488-012-0442-8
Hogue, A., Dauber, S., & Henderson, C. E. (2017b). Benchmarking family therapy for adolescent behavior problems in usual care: Fidelity, outcomes, and therapist performance differences. Administration and Policy in Mental Health and Mental Health Services Research, 44, 626–641. https://doi.org/10.1007/s10488-016-0769-7
Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014b). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy in Mental Health and Mental Health Services Research, 41, 697–705. https://doi.org/10.1007/s10488-013-0520-6
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42, 229–243. https://doi.org/10.1007/s10488-014-0548-2
Hogue, A., Henderson, C. E., Becker, S. J., & Knight, D. K. (2018). Evidence base on outpatient behavioral treatments for adolescent substance use, 2014–2017: Outcomes, treatment delivery, and promising horizons. Journal of Clinical Child and Adolescent Psychology, 47, 499–526. https://doi.org/10.1080/15374416.2018.1466307
Hogue, A., Liddle, H. A., Rowe, C., Turner, R. M., Dakof, G. A., & LaPann, K. (1998). Treatment adherence and differentiation in individual versus family therapy for adolescent substance abuse. Journal of Counseling Psychology, 45, 104–114. https://doi.org/10.1037/0022-0167.45.1.104
Hogue, A., Ozechowski, T., Robbins, M. S., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology: Science and Practice, 20, 60–77. https://doi.org/10.1111/cpsp.12023
Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244. https://doi.org/10.1007/s10488-009-0251-x
Institute of Medicine. (2015). Psychosocial interventons for mental and substance use disorders: A framework for establishing evidence-based standards. National Academies Press.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Kreft, I., & DeLeeuw, J. (1998). Introducing multilevel modeling. Sage Publications.
Liddle, H. A. (2016). Multidimensional family therapy: Evidence base for transdiagnostic treatment outcomes, change mechanisms, and implementation in community settings. Family Process, 55, 558–576. https://doi.org/10.1111/famp.12243
Lock, J. (2015). An update on evidence-based psychosocial treatments for eating disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 44, 707–721. https://doi.org/10.1080/15374416.2014.971458
Lyon, A., & Koerner, K. (2016). User-centered design for psychosocial intervention development and implementation. Clinical Psychology: Science and Practice, 23, 180–200. https://doi.org/10.1111/cpsp.12154
McCart, M. R., & Sheidow, A. J. (2016). Evidence-based psychosocial treatments for adolescents with disruptive behavior. Journal of Clinical Child and Adolescent Psychology, 45, 529–563. https://doi.org/10.1080/15374416.2016.1146990
McHugh, R., Murray, H., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically supported treatments: The promise of transdiagnostic interventions. Behavior Research and Therapy, 47, 946–953. https://doi.org/10.1016/j.brat.2009.07.005
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20, 14–32. https://doi.org/10.1111/cpsp.12020
Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus User's Guide (Seventh ed.). Los Angeles, CA: Muthen & Muthen.
Nichols, M. P., & Schwartz, R. C. (1991). Family therapy: Cocnepts and methods (2nd ed.). Allyn and Bacon.
Olin, S. S., Kutash, K., Pollock, M., Burns, B. J., Kuppinger, A., Craig, N., … Hoagwood, K. E. (2014). Developing quality indicators for family support services in community team-based mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 7–20. https://doi.org/10.1007/s10488-013-0501-9
Regan, J., Daleiden, E. L., & Chorpita, B. F. (2013). Integrity in mental health systems: An expanded framework for managing uncertainty in clinical care. Clinical Psychology: Science and Practice, 20, 78–98. https://doi.org/10.1111/cpsp.12024
Robbins, M. S., Alexander, J. F., Turner, C. W., & Hollimon, A. (2016). Evolution of functional family therapy as an evidence-based practice for adolescents with disruptive behavior problems. Family Process, 55, 543–557. https://doi.org/10.1111/famp.12230
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0
Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. https://doi.org/10.1037/0033-2909.86.2.420
Slesnick, N., Erdem, G., Bartle-Haring, S., & Brigham, G. S. (2013). Intervention with substance-abusing runaway adolescents and their families: Results of a randomized clinical trial. Journal of Consulting & Clinical Psychology, 81, 600–614. https://doi.org/10.1037/a0033463
Southam-Gerow, M. A., McLeod, B. D., Arnold, C. C., Rodríguez, A., Cox, J. R., Reise, S. P., … Kendall, P. C. (2016). Initial development of a treatment adherence measure for cognitive–behavioral therapy for child anxiety. Psychological Assessment, 28, 70–80. https://doi.org/10.1037/pas0000141
Stirman, S. W. (2020). Commentary: Challenges and opportunites in the assessment of fidelity and related constructs. Administration and Policy in Mental Health and Mental Health Services Research, 47(6), 932–934. https://doi.org/10.1007/s10488-020-01069-4
Szapocznik, J., & Hervis, O. E. (2020). Brief strategic family therapy. American Psychological Association.
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013). The comparative effectiveness of outpatient treatment for adolescent substance abuse: A meta-analysis. Journal of Substance Abuse Treatment, 44, 145–158. https://doi.org/10.1016/j.jsat.2012.05.006
Wampold, B. E., & Serlin, R. C. (2000). The consequence of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425. https://doi.org/10.1037/1082-989x.5.4.425
Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., Daleiden, E. L., Bearman, S., & Weisz, J. R. (2013). Tracking evidence-based practice with youth: Validity of MATCH and standard manual consultation records. Journal of Clinical Child and Adolescent Psychology, 42, 44–55. https://doi.org/10.1080/15374416.2012.700505
Watzlawick, P., & Weakland, J. H. (1977). The interactional view. Norton.
Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. https://doi.org/10.1037/a0018912
Weersing, R. V., Jeffreys, M., Do, M., Schwartz, K., & Bolano, C. (2017). Evidence base update of psychosocial treatments for child and adolescent depression. Journal of Clinical Child and Adolescent Psychology, 46, 11–43. https://doi.org/10.1080/15374416.2016.1220310
Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180. https://doi.org/10.1207/s15374424jccp3102_03
Acknowledgements
Preparation of this article was supported by the National Institute on Drug Abuse (R34DA044740; PI: Hogue). The authors gratefully acknowledge the dedicated team of observational coders who made this study possible (in addition to authors MB, AM, LB): Ashli Cleveland, Jenna Cutler, Diana Gonzalez, Cori Hammond, Zachary Kletter, Sosi Korian, Madeline Little, Michelle Martinez, Kelly Roberts, Kennie Swanson, Erica Zakotyansky.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to report.
Ethical Approval
The study was conducted under approval by the governing Institutional Review Board. Data were collected from March 2019 through March 2020.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hogue, A., Bobek, M., Porter, N. et al. Therapist Self-Report of Fidelity to Core Elements of Family Therapy for Adolescent Behavior Problems: Psychometrics of a Pragmatic Quality Indicator Tool. Adm Policy Ment Health 49, 298–311 (2022). https://doi.org/10.1007/s10488-021-01164-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-021-01164-0