Skip to main content

Toward the Effective and Efficient Measurement of Implementation Fidelity

Administration and Policy in Mental Health and Mental Health Services Research volume 38, pages 32–43 (2011)Cite this article

Abstract

Implementation science in mental health is informed by other academic disciplines and industries. Conceptual and methodological territory charted in psychotherapy research is pertinent to two elements of the conceptual model of implementation posited by Aarons and colleagues (2010)—implementation fidelity and innovation feedback systems. Key characteristics of scientifically validated fidelity instruments, and of the feasibility of their use in routine care, are presented. The challenges of ensuring fidelity measurement methods are both effective (scientifically validated) and efficient (feasible and useful in routine care) are identified as are examples of implementation research attempting to balance these attributes of fidelity measurement.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

References

  • Aarons, G. A., Sommerfeld, D., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77, 270–280.

    Article  PubMed  Google Scholar 

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2010). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research.

  • AERA, APA, & NCME. (1999). Standards for educational and psychological testing. Washington, DC: AERA.

    Google Scholar 

  • Baumann, B. L., Kolko, D. J., Collins, K., & Herschell, A. D. (2006). Understanding practitioners’ characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse and Neglect, 30, 771–787.

    Article  PubMed  Google Scholar 

  • Bearsley-Smith, C., Sellick, K., Chesters, J., Francis, K., & Gippsland Adolescent Depression Research Group. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy Mental Health, 35, 423–435.

    Article  Google Scholar 

  • Cappella, E., Frazier, S. L., Atkins, M. S., Schoenwald, S. K., & Glisson, C. (2008). Enhancing schools’ capacity to support children in poverty: An ecological model of school-based mental health services. Administration and Policy In Mental Health, 35, 395–409.

    Article  PubMed  Google Scholar 

  • Carroll, K. M., & Rounsaville, B. J. (2007). A vision of the next generation of behavioral therapies research in the addictions. Addiction, 102, 850–862.

    Article  PubMed  Google Scholar 

  • Chamberlain, P. (2003). Treating chronic juvenile offenders: Advances made through the Oregon multidimensional treatment foster care model. Washington, DC: American Psychological Association.

    Book  Google Scholar 

  • Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. A., Reid, J. B., et al. (2008). Prevention of behavior problems for children in foster care: Outcomes and mediation effects. Prevention Science, 9, 17–27.

    Article  PubMed  Google Scholar 

  • Chamberlain, P., Sprengelmeyer, P., Saldana, L., & Padget, C. (2010) Web-based observations to monitor treatment fidelity. Manuscript in preparation.

  • Child and Adolescent Mental Health Division (2003). Instructions and codebook for provider monthly summaries. Honolulu, HI: Hawaii Department of Health Child and Adolescent Mental Health Division. www.hawaii.gov/health/mental-health/camhd/resources/prov-agency/library/pdf/paf/paf-001.pdf. Accessed 23 March 2010.

  • Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566–579.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7, 5–20.

    Article  PubMed  Google Scholar 

  • Fixsen, D. L, Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

  • Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13.

    Article  PubMed  Google Scholar 

  • Garland, A. F., Hawley, K. M., Brookman-Frazee, L., & Hurlburt, M. S. (2008a). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 505–514.

    Article  PubMed  Google Scholar 

  • Garland, A. F., McCabe, K. M., & Yeh, M. (2008b). Ethical challenges in practice-based mental health services research: Examples from research with children and families. Clinical Psychology: Science and Practice, 15, 118–124.

    Article  Google Scholar 

  • Hayes, S. C. (1998). Market-driven treatment development. The Behavior Therapist, 21, 32–33.

    Google Scholar 

  • Heaton, K. J., Hill, C. E., & Edwards, L. A. (1995). Comparing molecular and molar methods of judging therapist techniques. Psychotherapy Research, 5, 141–153.

    Article  Google Scholar 

  • Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (2009). Multisystemic therapy for antisocial behavior in children and adolescents (2nd ed.). New York: The Guilford Press.

    Google Scholar 

  • Hepner, K. A., Greenwood, G. L., Azocar, F., Miranda, J., & Burnam, M. A. (2009). Usual care psychotherapy for depression in a large managed behavioral health organization. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-009-0247-6.

  • Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33, 332–345.

    Google Scholar 

  • Hurlburt, M. S., Garland, A. G., Nguyen, K., & Brookman-Frazee, L. (2009). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–243.

    Article  Google Scholar 

  • Kelley, S. D., Vides de Andrade, A. R., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-010-0298-8.

  • Malik, M. L., Beutler, L. E., Alimohamed, S., Gallagher-Thompson, D., & Thompson, L. (2003). Are all cognitive therapies alike? A comparison of cognitive and noncognitive therapy process and implications for the application of empirically supported treatments. Journal of Consulting and Clinical Psychology, 71, 150–158.

    Article  PubMed  Google Scholar 

  • Manderscheid, R. W. (1998). From many to one: Addressing the crisis of quality in managed behavioral health care at the millennium. The Journal of Behavioral Health Services & Research, 25, 233–238.

    CAS  Article  Google Scholar 

  • McHugh, R. K., Murray, H. W., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47(11), 946–953.

    Article  PubMed  Google Scholar 

  • Mihalic, S. (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4(83–86), 99–105.

    Google Scholar 

  • Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.

    Google Scholar 

  • National Institute of Mental Health. (2006). The road ahead: Research partnerships to transform services. A report by the National Advisory Mental Health Council’s Workgroup on Services and Clinical Epidemiology Research. Rockville, MD: National Institutes of Health, Department of Health and Human Services.

  • Orimoto, T. E., Higa-McMillan, C. K., Mueller, C., & Tolman, R. T. (2009, February). Organization of therapeutic practices in treatment as usual. Paper presented at the 22nd annual research conference of the Research and Training Center for Children’s Mental Health, Tampa, FL.

  • Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841.

    Article  PubMed  Google Scholar 

  • Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. Research in Organizational Change and Development, 15, 63–134.

    Article  Google Scholar 

  • Rossi, P. H. (1978). Issues in the evaluation of human services delivery. Evaluation Quarterly, 2(4), 573–599.

    Article  Google Scholar 

  • Schoenwald, S. K. (2008). Toward evidence-based transport of evidence-based treatments: MST as an example. Journal of Child and Adolescent Substance Abuse, 17(3), 69–91.

    Article  Google Scholar 

  • Schoenwald, S. K., & Henggeler, S. W. (2004). A public health perspective on the transport of evidence based practices. Clinical Science and Practice, 11, 360–363.

    Article  Google Scholar 

  • Schoenwald, S. K., Henggeler, S. W., Brondino, M. J., & Rowland, M. D. (2000). Multisystemic therapy: Monitoring treatment fidelity. Family Process, 39, 83–103.

    CAS  Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35, 84–97.

    Article  PubMed  Google Scholar 

  • Sheidow, A. J., Donohue, B. C., Hill, H. H., Henggeler, S. W., & Ford, J. D. (2008). Development of an audio-tape review system for supporting adherence to an evidence-based treatment. Professional Psychology: Research and Practice, 39, 553–560.

    Article  Google Scholar 

  • Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. E. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630.

    CAS  Article  PubMed  Google Scholar 

  • Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.

    Google Scholar 

  • Young, J., Daleiden, E. L., Chorpita, B. F., Schiffman, S., & Mueller, C. W. (2007). Assessing stability between treatment planning documents in a system of care. Administration and Policy in Mental Health and Mental Health Services Research, 34, 530–539.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The primary support for this manuscript was provided by NIMH research grants 1P30MH074678 (J. Landsverk, PI) and 1P20MH0784458 (M. Atkins, PI). The authors thank the Implementation Methods Research Group (NIMH grant 1P30MH074678) Executive Committee for helpful reviews of manuscript drafts that lead to the inclusion of Table 2 and Figure 1; Marc Atkins, Bruce Chorpita, and David Henry for their participation in Think Tank discussions pertinent to the manuscript; and, V. Robin Weersing.

Author information

Authors and Affiliations

  1. Family Services Research Center, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 67 President Street, Ste. MC406, MSC 861, Charleston, SC, 29425, USA

    Sonja K. Schoenwald, Jason E. Chapman & Ashli J. Sheidow

  2. Child and Adolescent Services Research Center, University of California, San Diego, CA, USA

    Ann F. Garland

  3. University of Illinois, Chicago, IL, USA

    Stacy L. Frazier

  4. Virginia Commonwealth University, Richmond, VA, USA

    Michael A. Southam-Gerow

Authors
  1. Sonja K. Schoenwald
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ann F. Garland
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jason E. Chapman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stacy L. Frazier
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ashli J. Sheidow
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael A. Southam-Gerow
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sonja K. Schoenwald.

Additional information

Author Note

Sonja K. Schoenwald is a Board Member and stockholder in MST Services, LLC, which has the exclusive licensing agreement through the Medical University of South Carolina for the dissemination of MST technology.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Schoenwald, S.K., Garland, A.F., Chapman, J.E. et al. Toward the Effective and Efficient Measurement of Implementation Fidelity. Adm Policy Ment Health 38, 32–43 (2011). https://doi.org/10.1007/s10488-010-0321-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-010-0321-0

Keywords

  • Implementation fidelity
  • Fidelity measurement methods
  • Adherence