Reliability of Therapist Self-Report on Treatment Targets and Focus in Family-Based Intervention

  • Aaron HogueEmail author
  • Sarah Dauber
  • Craig E. Henderson
  • Howard A. Liddle
Original Article


Reliable therapist-report methods appear to be an essential component of quality assurance procedures to support adoption of evidence-based practices in usual care, but studies have found weak correspondence between therapist and observer ratings of treatment techniques. This study examined therapist reliability and accuracy in rating intervention target (i.e., session participants) and focus (i.e., session content) in a manual-guided, family-based preventive intervention implemented with 50 inner-city adolescents at risk for substance use. A total of 106 sessions selected from three phases of treatment were rated via post-session self-report by the participating therapist and also via videotape by nonparticipant coders. Both groups estimated the amount of session time devoted to model-prescribed treatment targets (adolescent, parent, conjoint) and foci (family, school, peer, prosocial, drugs). Therapists demonstrated excellent reliability with coders for treatment targets and moderate to high reliability for treatment foci across the sample and within each phase. Also, therapists did not consistently overestimate their degree of activity with targets or foci. Implications of study findings for fidelity assessment in routine settings are discussed.


Therapist self-report Quality assurance Treatment fidelity Treatment targets Therapy focus Usual care 



Preparation of this article was supported by Grants R01DA023945 from the National Institute on Drug Abuse and HD1SP07054 from the Center for Substance Abuse Prevention. The authors are grateful for the dedicated work of the prevention counselors at Temple Teen Care and the observational coders for the TBRS-2 project.


  1. Barber, J. P., Sharpless, B., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology, 38, 493–500.CrossRefGoogle Scholar
  2. Barnoski, R. (2003). Outcome evaluation of Washington State’s research-based programs for juvenile offenders. Washington State Institute for Public Policy ( Scholar
  3. Bearsley-Smith, C., Sellick, K., Chesters, J., & Francis, K. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy in Mental Health and mental Health Services Research, 35, 423–435.PubMedCrossRefGoogle Scholar
  4. Becker, S. J., & Curry, J. F. (2008). Outpatient interventions for adolescent substance abuse: A quality of evidence review. Journal of Consulting and Clinical Psychology, 76, 531–543.PubMedCrossRefGoogle Scholar
  5. Carroll, K. M., Nich, C., & Rounsaville, B. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320.CrossRefGoogle Scholar
  6. Cicchetti, D. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.CrossRefGoogle Scholar
  7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  8. Diamond, G. S., & Liddle, H. A. (1996). Resolving a therapeutic impasse between parents and adolescents in multidimensional family therapy. Journal of Consulting and Clinical Psychology, 64, 481–488.PubMedCrossRefGoogle Scholar
  9. Diamond, G. S., & Liddle, H. A. (1999). Transforming negative parent-adolescent interactions: From impasse to dialogue. Family Process, 38, 5–26.PubMedCrossRefGoogle Scholar
  10. Dobson, K. S., & Shaw, B. F. (1993). The training of cognitive therapists: What have we learned from treatment manuals? Psychotherapy, 30, 573–577.CrossRefGoogle Scholar
  11. Fixsen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la parte Florida Mental Health Institute, The National Implementation Research Network. (Report No.: FMHI Publication #231).Google Scholar
  12. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105–112.CrossRefGoogle Scholar
  13. Garland, A. F., Bickman, L., & Chorpita, B. F. (2010a). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 15–26.PubMedCentralPubMedCrossRefGoogle Scholar
  14. Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010b). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research, 37, 208–220.PubMedCentralPubMedCrossRefGoogle Scholar
  15. Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., et al. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology, 78, 537–550.PubMedCentralPubMedCrossRefGoogle Scholar
  16. Henggeler, S. W., & Sheidow, A. J. (2012). Empirically supported family-based treatments for conduct disorder and delinquency in adolescents. Journal of Marital and Family Therapy, 38, 30–58.PubMedCentralPubMedCrossRefGoogle Scholar
  17. Hoagwood, K. E. (2005). Family-based services in children’s mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690–713.PubMedCrossRefGoogle Scholar
  18. Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., Inclan, J., et al. (2008a). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.PubMedCentralPubMedCrossRefGoogle Scholar
  19. Hogue, A., Dauber, S., Samuolis, J., & Liddle, H. A. (2006). Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. Journal of Family Psychology, 20, 535–543.PubMedCentralPubMedCrossRefGoogle Scholar
  20. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008b). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.PubMedCentralPubMedCrossRefGoogle Scholar
  21. Hogue, A., Johnson-Leckrone, J., & Liddle, H. A. (1999). Recruiting high-risk families into family-based prevention and prevention research. Journal of Mental Health Counseling, 21, 337–351.Google Scholar
  22. Hogue, A., & Liddle, H. A. (1999). Family-based preventive intervention: An approach to preventing substance use and antisocial behavior. American Journal of Orthopsychiatry, 69, 278–293.PubMedCrossRefGoogle Scholar
  23. Hogue, A., Liddle, H. A., & Becker, D. (2002a). Multidimensional family prevention for at-risk adolescents. In T. Patterson (Ed.), Comprehensive handbook of psychotherapy, Vol. II Cognitive-behavioral approaches (pp. 141–166). New York: Wiley.Google Scholar
  24. Hogue, A., Liddle, H. A., Becker, D., & Johnson-Leckrone, J. (2002b). Family-based prevention counseling for high-risk young adolescents: Immediate outcomes. Journal of Community Psychology, 30, 1–22.CrossRefGoogle Scholar
  25. Hogue, A., Liddle, H. A., Dauber, S., & Samuolis, J. (2004). Linking session focus to treatment outcome in evidence-based treatments for adolescent substance abuse. Psychotherapy, 41, 83–96.PubMedCentralPubMedCrossRefGoogle Scholar
  26. Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33, 332–345.CrossRefGoogle Scholar
  27. Hogue, A., Liddle, H. A., Singer, A., & Leckrone, J. (2005). Intervention fidelity in family-based prevention counseling for adolescent problem behaviors. Journal of Community Psychology, 33, 191–211.CrossRefGoogle Scholar
  28. Hogue, A., Ozechowski, T. J., Robbins, M. S., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology, 20, 60–77.Google Scholar
  29. Huey, S., Henggeler, S. W., Brondino, M. J., & Pickrel, S. G. (2000). Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting & Clinical Psychology, 68, 451–467.CrossRefGoogle Scholar
  30. Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244.PubMedCentralPubMedCrossRefGoogle Scholar
  31. Kelley, S. D., Vides de Andrade, A. R., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 287–300.PubMedCrossRefGoogle Scholar
  32. Liddle, H. A., Rowe, C. L., Gonzalez, A., Henderson, C. E., Dakof, G. A., & Greenbaum, P. E. (2006). Changing provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. The American Journal on Addictions, 15, 102–112.PubMedCrossRefGoogle Scholar
  33. Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.PubMedCentralPubMedCrossRefGoogle Scholar
  34. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. American Psychologist, 65, 73–84.PubMedCrossRefGoogle Scholar
  35. McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology, 20, 14–32.PubMedGoogle Scholar
  36. Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology, 12, 365–383.Google Scholar
  37. Robbins, M. S., Feaster, D. J., Horigian, V. E., Puccinelli, M. J., Henderson, C. E., & Szapocznik, J. (2011). Therapist adherence in brief strategic family therapy for adolescent drug abusers. Journal of Consulting & Clinical Psychology, 79, 43–53.CrossRefGoogle Scholar
  38. Schmidt, S. E., Liddle, H. A., & Dakof, G. A. (1996). Changes in parenting practices and adolescent drug abuse during multidimensional family therapy. Journal of Family Psychology, 10, 12–27.CrossRefGoogle Scholar
  39. Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s…fidelity measurement in the real world. Clinical Psychology, 18, 142–147.PubMedCentralPubMedGoogle Scholar
  40. Schoenwald, S. K., Chapman, J. E., Kelleher, K. E., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35, 84–97.PubMedCrossRefGoogle Scholar
  41. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43.PubMedCentralPubMedCrossRefGoogle Scholar
  42. Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology, 34, 658–670.PubMedCrossRefGoogle Scholar
  43. Schoenwald, S. K., Sheidow, A., & Letourneau, E. J. (2004). Toward effective quality assurance in evidence-based practice: Links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology, 33, 94–104.PubMedCrossRefGoogle Scholar
  44. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.PubMedCrossRefGoogle Scholar
  45. Waldron, H. B., & Turner, C. W. (2008). Evidence-based psychosocial treatments for adolescent substance abuse. Journal of Clinical Child and Adolescent Psychology, 37, 238–261.PubMedCrossRefGoogle Scholar
  46. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211.PubMedCrossRefGoogle Scholar
  47. Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.CrossRefGoogle Scholar
  48. Zazzali, J. L., Sherbourne, C., Hoagwood, K. E., Greene, D., Bigley, M. F., & Sexton, T. (2008). The adoption and implementation of an evidence based practice in child and family mental health services organizations: A pilot study of functional family therapy in New York State. Administration and Policy in Mental Health and Mental Health Services Research, 35, 38–49.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Aaron Hogue
    • 1
    Email author
  • Sarah Dauber
    • 1
  • Craig E. Henderson
    • 2
  • Howard A. Liddle
    • 3
  1. 1.Treatment Research DivisionNational Center on Addiction and Substance Abuse (CASA) at Columbia UniversityNew YorkUSA
  2. 2.Sam Houston State UniversityHuntsvilleUSA
  3. 3.Center for Treatment Research on Adolescent Drug AbuseUniversity of Miami Miller School of MedicineMiamiUSA

Personalised recommendations