Skip to main content
SpringerLink
Log in

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. T scores ≥70 on the Externalizing Scale of the parent-report version of the Child Behavior Checklist (Achenbach 1991).

References

  • Achenbach, T. M. (1991). Manual for the child behavior checklist/4-18 and 1991 profile. Burlington: University of Vermont, Department of Psychiatry.

    Google Scholar 

  • Berkel, C., Mauricio, A. M., Schoenfelder, E. N., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.

    Article  PubMed  Google Scholar 

  • Brody, G. H., Murry, V. M., Gerrard, M., Gibbons, F. X., Molgaard, V., McNair, L., & Neubaum-Carlan, E. (2004). The strong African american families program: translating research into prevention programming. Child Development, 75, 900–917. doi:10.2307/3696600.

    Article  PubMed  Google Scholar 

  • Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. doi:10.1186/1748-5908-2-40.

    Article  PubMed  PubMed Central  Google Scholar 

  • Chiapa, A., Smith, J. D., Kim, H., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2015). The trajectory of fidelity in a multiyear trial of the family check-up predicts change in child problem behavior. Journal of Consulting and Clinical Psychology. doi:10.1037/ccp0000034.

    PubMed  PubMed Central  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  • Cross, W. F., & West, J. C. (2011). Examining implementer fidelity: Conceptualizing and measuring adherence and competence. Journal of Children’s Services, 6, 18–33. doi:10.5042/jcs.2011.0123.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dishion, T. J., & Stormshak, E. A. (2007). Intervening in children's lives: An ecological, family-centered approach to mental health care. Washington, DC: American Psychological Association.

    Book  Google Scholar 

  • Dishion, T. J., Shaw, D. S., Connell, A., Gardner, F. E. M., Weaver, C., & Wilson, M. (2008). The family check-up with high-risk indigent families: Preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child Development, 79, 1395–1414. doi:10.1111/j.1467-8624.2008.01195.x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dishion, T.J., Knutson, N., Brauer, L., Gill, A., & Risso, J. (2010). Family Check-Up: COACH ratings manual. Unpublished coding manual. Available from the Child and Family Center, 6217. Eugene: University of Oregon OR 97403.

  • Forgatch, M. S., & DeGarmo, D. S. (2011). Sustaining fidelity following the nationwide PMTO™ implementation in Norway. Prevention Science, 12, 235–246. doi:10.1007/s11121-011-0225-6.

    Article  PubMed  PubMed Central  Google Scholar 

  • Forgatch, M. S., DeGarmo, D. S., & Beldavs, Z. G. (2005a). An efficacious theory-based intervention for stepfamilies. Behavior Therapy, 36, 357–365. doi:10.1016/S0005-7894(05)80117-0.

    Article  PubMed  PubMed Central  Google Scholar 

  • Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005b). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13. doi:10.1016/S0005-7894(05)80049-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child & Adolescent Psychology, 31, 155–167. doi:10.1207/S15374424JCCP3102_02.

    Article  Google Scholar 

  • Hogue, A., Liddle, H. A., Becker, D., & Johnson-Leckrone, J. (2002). Family-based prevention counseling for high-risk young adolescents: Immediate outcomes. Journal of Community Psychology, 30, 1–22. doi:10.1002/jcop.1047.

    Article  Google Scholar 

  • Hogue, A., Liddle, H. A., Singer, A., & Leckrone, J. (2005). Intervention fidelity in family‐based prevention counseling for adolescent problem behaviors. Journal of Community Psychology, 33, 191–211. doi:10.1002/jcop.20031.

    Article  Google Scholar 

  • Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555. doi:10.1037/0022-006x.76.4.544.

    Article  PubMed  PubMed Central  Google Scholar 

  • Huey, S. J., Jr., Henggeler, S. W., Brondino, M. J., & Pickrel, S. G. (2000). Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting and Clinical Psychology, 68, 451–467. doi:10.1037/0022-006x.68.3.451.

    Article  PubMed  Google Scholar 

  • Hukkelberg, S. S., & Ogden, T. (2013). Working alliance and treatment fidelity as predictors of externalizing problem behaviors in parent management training. Journal of Consulting and Clinical Psychology, 81, 1010–1020. doi:10.1037/a0033825.

    Article  PubMed  Google Scholar 

  • Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press.

    Google Scholar 

  • Kazdin, A. E. (2003). Research design in clinical psychology (4th ed.). Boston: Allyn & Bacon.

    Google Scholar 

  • Knutson, N. M., Forgatch, M. S., & Rains, L. A. (2003). Fidelity of implementation rating system (FIMP): The training manual for PMTO. Eugene: Oregon Social Learning Center.

    Google Scholar 

  • Landsverk, J. A., Brown, C. H., Chamberlain, P., Palinkas, L. A., Ogihara, M., Czaja, S., & Horwitz, S. M. (2012). Design and analysis in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating research to practice (pp. 225–260). New York: Oxford University Press.

    Chapter  Google Scholar 

  • Lee, C.-Y., August, G., Realmuto, G., Horowitz, J., Bloomquist, M., & Klimes-Dougan, B. (2008). Fidelity at a distance: Assessing implementation fidelity of the early risers prevention program in a going-to-scale intervention trial. Prevention Science, 9, 215–229. doi:10.1007/s11121-008-0097-6.

    Article  PubMed  Google Scholar 

  • Meier, S. T. (1999). Training the practitioner-scientist: Bridging case conceptualization, assessment, and intervention. The Counseling Psychologist, 27, 846–869. doi:10.1177/0011000099276008.

    Article  Google Scholar 

  • Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparing people for change (2nd ed.). New York: Guilford Press.

    Google Scholar 

  • Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499. doi:10.1016/j.cpr.2013.01.010.

    Article  PubMed  Google Scholar 

  • Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12, 365–383. doi:10.1093/Clipsy/Bpi045.

    Google Scholar 

  • Prado, G., Pantin, H., Schwartz, S. J., Lupei, N. S., & Szapocznik, J. (2006). Predictors of engagement and retention into a parent-centered, ecodevelopmental HIV preventive intervention for Hispanic adolescents and their families. Journal of Pediatric Psychology, 31, 874–890. doi:10.1093/jpepsy/jsj046.

    Article  PubMed  Google Scholar 

  • R Core Team (2012). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/.

  • Rao, C.R. (2009). Linear statistical inference and its applications (Vol. 22): Wiley.

  • Safran, J. D., Muran, J. C., Samstag, L. W., & Stevens, C. (2001). Repairing alliance ruptures. Psychotherapy: Theory, Research, Practice, Training, 38, 406–412. doi:10.1037/0033-3204.38.4.406.

    Article  Google Scholar 

  • Schoenwald, S. K., & Henggeler, S. W. (2004). A public health perspective on the transport of evidence-based practices. Clinical Psychology: Science and Practice, 11, 360–363. doi:10.1093/clipsy.bph092.

    Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J., Frazier, S., Sheidow, A., & Southam-Gerow, M. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi:10.1007/s10488-010-0321-0.

    Article  PubMed  PubMed Central  Google Scholar 

  • Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420.

    Article  PubMed  CAS  Google Scholar 

  • Silberschatz, G., Fretter, P. B., & Curtis, J. T. (1986). How do interpretations influence the process of psychotherapy? Journal of Consulting and Clinical Psychology, 54, 646–652. doi:10.1037/0022-006X.54.5.646.

    Article  PubMed  CAS  Google Scholar 

  • Smith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013). Indirect effects of fidelity to the family check-up on changes in parenting and early childhood problem behaviors. Journal of Consulting and Clinical Psychology, 81, 962–974. doi:10.1037/a0033950.

    Article  PubMed  Google Scholar 

  • Smith, J. D., Stormshak, E. A., & Kavanagh, K. (2014). Results of a pragmatic effectiveness-implementation hybrid trial of the family check-up in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research, 42, 265–278. doi:10.1007/s10488-014-0566-0.

    Article  Google Scholar 

  • SPSS Statistics Version 22.0.0 (2014). Chicago: SPSS Inc.

  • Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. doi:10.1037/a0018912.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Institute on Drug Abuse (NIDA) through a pilot study grant awarded to Justin Smith by the Center for Prevention Implementation Methodology for Drug Abuse and Sex Risk Behavior (DA027828) and DA016110 to Thomas Dishion, Daniel Shaw, and Melvin Wilson. Justin Smith and Naomi Knoble received support from National Institute of Mental Health research training grant T32 MH20012, awarded to Elizabeth Stormshak. The authors wish to thank Cady Berkel, Amanda Chiapa, Guillermo Prado, and Juan Villamar for their helpful comments on earlier drafts of this manuscript; Hendricks Brown and Carlos Gallo for their assistance with the statistical analyses; the Early Steps team in Eugene, Pittsburgh, and Charlottesville; and the families who participated in the study.

Conflict of Interest

Thomas Dishion is the developer of the Family Check-Up program. Justin Smith, Kimbree Brown, Karina Ramos, Naomi Knoble, Daniel Shaw, and Melvin Wilson have no conflicts of interest or financial relationships relevant to this article to disclose.

Author information

Author notes

  1. Kimbree Brown

    Present address: Oregon Social Learning Center, Eugene, OR, USA

  2. Karina Ramos

    Present address: University of California at Berkeley, Berkeley, CA, USA

  3. Naomi B. Knoble

    Present address: Nemours/duPont Children’s Hospital, Wilmington, DE, USA

Authors and Affiliations

  1. Center for Prevention Implementation Methodology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA

    Justin D. Smith

  2. REACH Institute, Arizona State University, Tempe, AZ, USA

    Thomas J. Dishion

  3. Oregon Research Institute, Eugene, OR, USA

    Thomas J. Dishion

  4. Prevention Science Institute, University of Oregon, Eugene, OR, USA

    Kimbree Brown, Karina Ramos & Naomi B. Knoble

  5. University of Pittsburgh, Pittsburgh, PA, USA

    Daniel S. Shaw

  6. University of Virginia, Charlottesville, VA, USA

    Melvin N. Wilson

  7. Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of Medicine, 750 N Lake Shore Drive, Chicago, IL, 60657, USA

    Justin D. Smith

Authors
  1. Justin D. Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas J. Dishion
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kimbree Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Karina Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Naomi B. Knoble
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel S. Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Melvin N. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justin D. Smith.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Smith, J.D., Dishion, T.J., Brown, K. et al. An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention. Prev Sci 17, 62–70 (2016). https://doi.org/10.1007/s11121-015-0589-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-015-0589-0

Keywords

Search

Navigation