Toward the Effective and Efficient Measurement of Implementation Fidelity

  • Sonja K. Schoenwald
  • Ann F. Garland
  • Jason E. Chapman
  • Stacy L. Frazier
  • Ashli J. Sheidow
  • Michael A. Southam-Gerow
Original Paper

Abstract

Implementation science in mental health is informed by other academic disciplines and industries. Conceptual and methodological territory charted in psychotherapy research is pertinent to two elements of the conceptual model of implementation posited by Aarons and colleagues (2010)—implementation fidelity and innovation feedback systems. Key characteristics of scientifically validated fidelity instruments, and of the feasibility of their use in routine care, are presented. The challenges of ensuring fidelity measurement methods are both effective (scientifically validated) and efficient (feasible and useful in routine care) are identified as are examples of implementation research attempting to balance these attributes of fidelity measurement.

Keywords

Implementation fidelity Fidelity measurement methods Adherence 

References

  1. Aarons, G. A., Sommerfeld, D., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77, 270–280.CrossRefPubMedGoogle Scholar
  2. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2010). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research.Google Scholar
  3. AERA, APA, & NCME. (1999). Standards for educational and psychological testing. Washington, DC: AERA.Google Scholar
  4. Baumann, B. L., Kolko, D. J., Collins, K., & Herschell, A. D. (2006). Understanding practitioners’ characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse and Neglect, 30, 771–787.CrossRefPubMedGoogle Scholar
  5. Bearsley-Smith, C., Sellick, K., Chesters, J., Francis, K., & Gippsland Adolescent Depression Research Group. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy Mental Health, 35, 423–435.CrossRefGoogle Scholar
  6. Cappella, E., Frazier, S. L., Atkins, M. S., Schoenwald, S. K., & Glisson, C. (2008). Enhancing schools’ capacity to support children in poverty: An ecological model of school-based mental health services. Administration and Policy In Mental Health, 35, 395–409.CrossRefPubMedGoogle Scholar
  7. Carroll, K. M., & Rounsaville, B. J. (2007). A vision of the next generation of behavioral therapies research in the addictions. Addiction, 102, 850–862.CrossRefPubMedGoogle Scholar
  8. Chamberlain, P. (2003). Treating chronic juvenile offenders: Advances made through the Oregon multidimensional treatment foster care model. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  9. Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. A., Reid, J. B., et al. (2008). Prevention of behavior problems for children in foster care: Outcomes and mediation effects. Prevention Science, 9, 17–27.CrossRefPubMedGoogle Scholar
  10. Chamberlain, P., Sprengelmeyer, P., Saldana, L., & Padget, C. (2010) Web-based observations to monitor treatment fidelity. Manuscript in preparation.Google Scholar
  11. Child and Adolescent Mental Health Division (2003). Instructions and codebook for provider monthly summaries. Honolulu, HI: Hawaii Department of Health Child and Adolescent Mental Health Division. www.hawaii.gov/health/mental-health/camhd/resources/prov-agency/library/pdf/paf/paf-001.pdf. Accessed 23 March 2010.
  12. Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566–579.CrossRefPubMedGoogle Scholar
  13. Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7, 5–20.CrossRefPubMedGoogle Scholar
  14. Fixsen, D. L, Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).Google Scholar
  15. Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13.CrossRefPubMedGoogle Scholar
  16. Garland, A. F., Hawley, K. M., Brookman-Frazee, L., & Hurlburt, M. S. (2008a). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 505–514.CrossRefPubMedGoogle Scholar
  17. Garland, A. F., McCabe, K. M., & Yeh, M. (2008b). Ethical challenges in practice-based mental health services research: Examples from research with children and families. Clinical Psychology: Science and Practice, 15, 118–124.CrossRefGoogle Scholar
  18. Hayes, S. C. (1998). Market-driven treatment development. The Behavior Therapist, 21, 32–33.Google Scholar
  19. Heaton, K. J., Hill, C. E., & Edwards, L. A. (1995). Comparing molecular and molar methods of judging therapist techniques. Psychotherapy Research, 5, 141–153.CrossRefGoogle Scholar
  20. Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (2009). Multisystemic therapy for antisocial behavior in children and adolescents (2nd ed.). New York: The Guilford Press.Google Scholar
  21. Hepner, K. A., Greenwood, G. L., Azocar, F., Miranda, J., & Burnam, M. A. (2009). Usual care psychotherapy for depression in a large managed behavioral health organization. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-009-0247-6.
  22. Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33, 332–345.Google Scholar
  23. Hurlburt, M. S., Garland, A. G., Nguyen, K., & Brookman-Frazee, L. (2009). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–243.CrossRefGoogle Scholar
  24. Kelley, S. D., Vides de Andrade, A. R., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-010-0298-8.
  25. Malik, M. L., Beutler, L. E., Alimohamed, S., Gallagher-Thompson, D., & Thompson, L. (2003). Are all cognitive therapies alike? A comparison of cognitive and noncognitive therapy process and implications for the application of empirically supported treatments. Journal of Consulting and Clinical Psychology, 71, 150–158.CrossRefPubMedGoogle Scholar
  26. Manderscheid, R. W. (1998). From many to one: Addressing the crisis of quality in managed behavioral health care at the millennium. The Journal of Behavioral Health Services & Research, 25, 233–238.CrossRefGoogle Scholar
  27. McHugh, R. K., Murray, H. W., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47(11), 946–953.CrossRefPubMedGoogle Scholar
  28. Mihalic, S. (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4(83–86), 99–105.Google Scholar
  29. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.Google Scholar
  30. National Institute of Mental Health. (2006). The road ahead: Research partnerships to transform services. A report by the National Advisory Mental Health Council’s Workgroup on Services and Clinical Epidemiology Research. Rockville, MD: National Institutes of Health, Department of Health and Human Services.Google Scholar
  31. Orimoto, T. E., Higa-McMillan, C. K., Mueller, C., & Tolman, R. T. (2009, February). Organization of therapeutic practices in treatment as usual. Paper presented at the 22nd annual research conference of the Research and Training Center for Children’s Mental Health, Tampa, FL.Google Scholar
  32. Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841.CrossRefPubMedGoogle Scholar
  33. Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. Research in Organizational Change and Development, 15, 63–134.CrossRefGoogle Scholar
  34. Rossi, P. H. (1978). Issues in the evaluation of human services delivery. Evaluation Quarterly, 2(4), 573–599.CrossRefGoogle Scholar
  35. Schoenwald, S. K. (2008). Toward evidence-based transport of evidence-based treatments: MST as an example. Journal of Child and Adolescent Substance Abuse, 17(3), 69–91.CrossRefGoogle Scholar
  36. Schoenwald, S. K., & Henggeler, S. W. (2004). A public health perspective on the transport of evidence based practices. Clinical Science and Practice, 11, 360–363.CrossRefGoogle Scholar
  37. Schoenwald, S. K., Henggeler, S. W., Brondino, M. J., & Rowland, M. D. (2000). Multisystemic therapy: Monitoring treatment fidelity. Family Process, 39, 83–103.CrossRefPubMedGoogle Scholar
  38. Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35, 84–97.CrossRefPubMedGoogle Scholar
  39. Sheidow, A. J., Donohue, B. C., Hill, H. H., Henggeler, S. W., & Ford, J. D. (2008). Development of an audio-tape review system for supporting adherence to an evidence-based treatment. Professional Psychology: Research and Practice, 39, 553–560.CrossRefGoogle Scholar
  40. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. E. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630.CrossRefPubMedGoogle Scholar
  41. Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.Google Scholar
  42. Young, J., Daleiden, E. L., Chorpita, B. F., Schiffman, S., & Mueller, C. W. (2007). Assessing stability between treatment planning documents in a system of care. Administration and Policy in Mental Health and Mental Health Services Research, 34, 530–539.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Sonja K. Schoenwald
    • 1
  • Ann F. Garland
    • 2
  • Jason E. Chapman
    • 1
  • Stacy L. Frazier
    • 3
  • Ashli J. Sheidow
    • 1
  • Michael A. Southam-Gerow
    • 4
  1. 1.Family Services Research Center, Department of Psychiatry and Behavioral SciencesMedical University of South CarolinaCharlestonUSA
  2. 2.Child and Adolescent Services Research CenterUniversity of CaliforniaSan DiegoUSA
  3. 3.University of IllinoisChicagoUSA
  4. 4.Virginia Commonwealth UniversityRichmondUSA

Personalised recommendations