Skip to main content
SpringerLink
Log in

Concordance between Community Supervisor and Provider Ratings of Fidelity: Examination of Multi-Level Predictors and Outcomes

  • Original Paper
  • Published:
Journal of Child and Family Studies Aims and scope Submit manuscript

Abstract

Fidelity is conceptualized as a key mechanism impacting treatment outcomes, yet community examination of fidelity is limited, partly due to limited feasible training methods and tools. Provider-report tools represent a feasible method for improving routine fidelity monitoring. Additional work examining the effectiveness of these methods and concordance with other fidelity sources is needed. Further, provider and client factors impact both intervention implementation and concordance, but their impact on fidelity measurement is poorly understood. Using a train-the-trainer methodology, we examine concordance between three methods of assessing fidelity (trained independent coders, supervisor evaluation and provider self-report) using a fidelity assessment tool adapted for community use. Multilevel models examined predictors (provider- and child-level factors) and outcomes (provider average fidelity) of concordance. Results suggest supervisors and providers are able to use the fidelity tool, but demonstrated variable concordance, with higher concordance with trained coders for supervisors than providers. Provider credentials and child language were marginally predictive of rater concordance. Higher supervisor accuracy was associated with higher provider fidelity. Findings inform broader efforts to improve the development and implementation of feasible fidelity tools for routine use in community settings and highlight multi-level factors impacting the adoption and implementation of such tools.

Highlights

  • This study examined a fidelity assessment tool adapted for community use, including concordance with other fidelity measures and predictors and outcomes of this tool.

  • Community supervisors and providers learned to use our fidelity tool, supporting initial feasibility and reliability.

  • Findings highlight multilevel factors impacting the adoption and implementation of such tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors.Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7.

    Article  PubMed  Google Scholar 

  • Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213.

    Article  PubMed  Google Scholar 

  • AFIRM Team. (2015). AFIRM online learning modules. Chapel Hill, NC: National Professional Development Center on Autism Spectrum Disorder.

  • Azad, G. F., Reisinger, E., Xie, M., & Mandell, D. S. (2015). Parent and teacher concordance on the social responsiveness scale for children with autism. School Mental Health, 8(3), 368–376. https://doi.org/10.1007/s12310-015-9168-6.

    Article  PubMed  PubMed Central  Google Scholar 

  • Beidas, R. S., Maclean, J., Fishman, J., Dorsey, S., Schoenwald, S., Mandell, D. S., & Marcus, S. C. (2016). A randomized trial to identify accurate and cost-effective fidelity measurement methods for cognitive-behavioral therapy: project FACTS study protocol. BMC Psychiatry, 16, 1–10. https://doi.org/10.1186/s12888-016-1034-z.

    Article  Google Scholar 

  • Boyd, R. D., & Corley, M. J. (2001). Outcome survey of early intensive behavioral intervention for young children with autism in a community setting. Autism: The International Journal of Research and Practice, 5(4), 430–441. https://doi.org/10.1177/1362361301005004007.

    Article  Google Scholar 

  • Brookman-Frazee, L., Haine, R. A., Baker-Ericzén, M. J., Zoffness, R., & Garland, A. F. (2010). Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy.Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 254–269. https://doi.org/10.1007/s10488-009-0244-9.

    Article  PubMed  Google Scholar 

  • Campbell, B. K., Buti, A., Fussell, H. E., Srikanth, P., McCarty, D., & Guydish, J. R. (2013). Therapist predictors of treatment delivery fidelity in a community based trial of 12-step facilitation. American Journal of Drug and Alcohol Abuse, 39(5), 304–3011. https://doi.org/10.3109/00952990.2013.799175.

    Article  Google Scholar 

  • Chapman, J. E., Mccart, M. R., Letourneau, E. J., & Sheidow, A. J. (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting and Clinical Psychology, 81(4), 674–680. https://doi.org/10.1037/a0033021.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  • De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A. G., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-Informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498.

    Article  PubMed  PubMed Central  Google Scholar 

  • De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: a critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), 483–509. https://doi.org/10.1037/0033-2909.131.4.483.

    Article  PubMed  Google Scholar 

  • Dickson, K. S., Suhrheinrich, J., Rieth, S. R., & Stahmer, A. C. (2018). Parent and teacher concordance of child outcomes for youth with autism spectrum disorder. Journal of Autism and Developmental Disorders, 48(5), 1423–1435. https://doi.org/10.1007/s10803-017-3382-z.

    Article  PubMed  PubMed Central  Google Scholar 

  • Elliott, C. D. (2007). Differential ability scales (second). San Antonio, TX: Harcourt Assessment.

    Google Scholar 

  • Eyberg, S. (1999). Parent-child interaction therapy: integrity checklists and session materials. Retrieved from the University of Florida Parent Child Interaction Therapy Web site: http://www.pcit.org.

  • Garland, A. F., Brookman-Frazee, L., & McLeod, B. D. (2008). Scoring manual for the PRAC study therapy process observational coding system for child Psychotherapy: Strategies Scale. Department of Psychiatry, University of California, San Diego.

  • Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research. 37(3), 208–220. https://doi.org/10.1007/s10488-009-0237-8.

    Article  PubMed  Google Scholar 

  • Hennggeler, S. W. (2004). Decreasing effect sizes for effectiveness studies—implications for the transport of evidence-based treatments: comment on Curtis, Ronan, and Borduin (2004). Journal of Family Psychology, 18(3), 420–423.

    Article  Google Scholar 

  • Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clinical Psychology Review, 30(4), 448–466. https://doi.org/10.1016/j.cpr.2010.02.005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention.Administration and Policy in Mental Health and Mental Health Services Research, 41(5), 697–705. https://doi.org/10.1007/s10488-013-0520-6.

    Article  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: correspondence between therapists and observers.Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2.

    Article  PubMed  Google Scholar 

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: concordance of therapist and observational perspectives.Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244. https://doi.org/10.1007/s10488-009-0251-x.

    Article  PubMed  Google Scholar 

  • Institute of Medicine (2015). Psychosocial interventions for mental and substance use disorders. Washington, D.C.: National Academies Press. https://doi.org/10.17226/19013.

  • Kimber, M., Barac, R., & Barwick, M. (2019). Monitoring fidelity to an evidence-based treatment: practitioner perspectives. Clinical Social Work Journal, 47(2), 207–221.

    Article  Google Scholar 

  • Lau, A. S., Stadnick, N., Regan, J., Roesch, S., Barnett, M. L., Saifan, D., & Brookman-Frazee, L. (2017). Therapist report of adaptations to delivery of evidence-based practices within a system-driven reform of publicly funded children’s mental health services. Journal of Consulting & Clinical Psychology, 85(7), 664–675.

    Article  Google Scholar 

  • Lord, C., DiLavore, M., Risi, P. C., Gotham, K., & Bishop, S. (2012). Autism diagnostic observation schedule. Second edition. Torrance, CA: Western Psychological Services.

  • Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19(2), 181–193. https://doi.org/10.1080/10503300802688460.

    Article  PubMed  PubMed Central  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. The American Psychologist, 65(2), 73–84. https://doi.org/10.1037/a0018121.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., Islam, N., & Wheat, E. (2013a). Designing, conducting, and evaluating therapy process research. The Oxford handbook of research strategies for clinical psychology. https://doi.org/10.1093/oxfordhb/9780199793549.013.0009.

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., & Smith, M. M. (2013b). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020.Making.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436–443. https://doi.org/10.1080/15374411003691750.

    Article  Google Scholar 

  • Moullin, J. C., Dickson, K. S., Stadnick, N., Rabin, B. A., & Aarons, G. A. (2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework. Implementation Science, 14(1).

  • Moullin, J. C., Ehrhart, M. G., & Aarons, G. A. (2018). The role of leadership in organizational implementation and sustainment in service agencies. Research on Social Work Practice, 28(5), 558–567. https://doi.org/10.1177/1049731517718361.

    Article  Google Scholar 

  • Mullen, E. (1995). Mullen Scales of Early Learning. Circle Pines, MN: AGS Publishing.

    Google Scholar 

  • National Research Council. (2001). Educating children with autism. Committee on educational interventions for children with autism. In: C. Lord & J. P. McGee (Eds), Division of behavioral and social sciences and education. Washington, DC: National Academy Press.

  • National Autism Center (2015). Findings and Conclusions: National Standards Project, Phase 2. Randolph, M.A.

  • Nelson, M. C., Cordray, D. S., Hulleman, C. S., Darrow, C. L., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services and Research, 39(4), 374–396. https://doi.org/10.1007/s11414-012-9295-x.

    Article  PubMed  Google Scholar 

  • Pellecchia, M., Connell, J. E., Beidas, R. S., Xie, M., Marcus, S. C., & Mandell, D. S. (2015). Dismantling the active ingredients of an intervention for children with autism. Journal of Autism & Developmental Disorders, 45(9), 2917–2927. https://doi.org/10.1007/s10803-015-2455-0.

    Article  Google Scholar 

  • Proctor, E. K., Landsverk, J., Aarons, G. A., Chambers, D. A., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 1–17. https://doi.org/10.1007/s10488-008-0197-4.

    Article  Google Scholar 

  • Proctor, E. K., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G. A., Bunger, A., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  PubMed  Google Scholar 

  • Rabin, B. A., & Brownson, R. C. (2017). Terminology for dissemination and implementation research. In: Dissemination and implementation research in health: Translating science to practice. Oxford University Press.

  • Rodriguez, A. (2016). The influence of client-, family-, and therapist-level pretreatment characteristics on therapist delivery of youth psychotherapy treatments. http://search.proquest.com/openview/6aff4136f823c8ec78c37bd15f9c13fd/1?pq-origsite=gscholar&cbl=18750&diss=y.

  • Rogers, S. J., & Dawson, G. (2010). Early Start Denver Model for young children with autism: Promoting language, learning, and engagement. Guilford Press.

  • Scherill, J. (2015). Pragmatic strategies for assessing psychotherapy quality in practice. Washington, D.C.

  • Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25(1), 146–156. https://doi.org/10.1037/a0029715.A.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., & Sheidow, A. J. (2011). Toward the effective and efficient measurement of implementation fidelity.Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology, 34(4), 658–670. https://doi.org/10.1207/s15374424jccp3404_8.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Mehta, T. G., Frazier, S. L., & Shernoff, E. S. (2013). Clinical supervision in effectiveness and implementation research. Clinical Psychology: Science and Practice, 20(1), 44–49. https://doi.org/10.1111/cpsp.12022.

    Article  Google Scholar 

  • Schoenwald, S. K., Sheidow, A. J., Letourneau, E. J., & Liao, J. G. (2003). Transportability of multisystemic therapy: evidence for multilevel influences. Mental Health Services Research, 5(4), 223–239.

    Article  Google Scholar 

  • Shire, S. Y., & Kasari, C. (2014). Train the trainer effectiveness trials of behavioral intervention for individuals with autism: a systematic review. American Journal on Intellectual and Developmental Disabilities, 119(5), 436–451. https://doi.org/10.1352/1944-7558-119.5.436.

    Article  PubMed  Google Scholar 

  • Stahmer, A. C., Suhrheinrich, J., Roesch, S., Zeedyk, S. M., Wang, T., Chan, N., & Lee, H. S. (2019). Evaluating key intervention components: The relationship between fidelity and child response. Research in Developmental Disabilities. https://doi.org/10.1016/j.ridd.2019.04.003.

  • Stirman, S., Marques, L., Creed, T. A., Gutner, C. A., Derubeis, R., Barnett, P. G., Kuhn, E., Suvak, M., Owen, J., Vogt, D., Jo, B., Schoenwald, S. K., Johnson, C., Mallard, Beristianos, M. & La Bash, H. (2018). Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study, 1–11. https://doi.org/10.1186/s13012-018-0756-3.

  • Stirman, S. W., Baumann, A. A., & Miller, C. J. (2019). The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 1–10. https://doi.org/10.1186/s13012-019-0898-y.

    Article  Google Scholar 

  • Stirman, S. W., Calloway, A., Katherine Toder, B., Christopher Miller, B. J., DeVito, A. K., Samuel Meisel, B. N., & Crits-Christoph, P. (2013a). Community mental health provider modifications to cognitive therapy: implications for sustainability. Psychiat-Ric Services, 64(10), 1056–1059. https://doi.org/10.1176/appi.ps.201200456.

    Article  Google Scholar 

  • Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013b). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8(1), 65. https://doi.org/10.1186/1748-5908-8-65.

    Article  PubMed  Google Scholar 

  • Stratis, E. A., & Lecavalier, L. (2015). Informant agreement for youth with autism spectrum disorder or intellectual disability: a meta-analysis. Journal of Autism and Developmental Disorders, 45(4), 1026–1041. https://doi.org/10.1007/s10803-014-2258-8.

    Article  PubMed  Google Scholar 

  • Suhrheinrich, J. (2015). A sustainable model for training teachers to use pivotal response training. Autism, 19(6), 713–723. https://doi.org/10.1177/1362361314552200.

    Article  PubMed  Google Scholar 

  • Suhrheinrich, J., Stahmer, A. C., Reed, S., Schreibman, L., Reisinger, E., & Mandell, D. S. (2013). Implementation challenges in translating pivotal response training into community settings. Journal of Autism & Developmental Disorders, 43, 2970–2976. https://doi.org/10.1007/s10803-013-1826-7.

    Article  Google Scholar 

  • Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: a therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child and Adolescent Psychology, 31(2), 168–180. https://doi.org/10.1207/153744202753604458.

    Article  PubMed  Google Scholar 

  • Wilczynski, S., & Christian, L. (2008). Project: promoting evidence-based practice in autism spectrum disorders. Effective practices for children with Autism: educational and behavior support interventions that work.

  • Williams, N. J., Frank, H. E., Frederick, L., Beidas, R. S., Mandell, D. S., Aarons, G. A., & Locke, J. (2019). Organizational culture and climate profiles: relationships with fidelity to three evidence-based practices for autism in elementary schools.Implementation Science, 14(1), 1–10. https://doi.org/10.1186/s13012-019-0863-9.

    Article  Google Scholar 

  • Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S., Brock, M.E., Plavnick, J.B., Fleury, V.P., & Schultz, T. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: a comprehensive review. Journal of Autism and Developmental Disorders, 1951–1966. https://doi.org/10.1007/s10803-014-2351-z.

  • Yarber, L., Brownson, C. A., Jacob, R. R., Baker, E. A., Jones, E., Baumann, C., Deshpande, A.D., Gillespie, K.N., Scharff, D.P., & Brownson, R. C. (2015). Evaluating a train-the-trainer approach for improving capacity for evidence-based decision making in public health. BMC Health Services Research. https://doi.org/10.1186/s12913-015-1224-2.

  • Zimmerman, I. L., Steiner, V. G., & Pond, R. E. (2011). Preschool language scales–fifth edition (PLS-5). Bloomington, MN: Pearson.

Download references

Funding

This study was funded by NIMH Grant R33 MH097033, NIMH Grant K01 MH109574 and NIMH Grant K23 MH115100.

Author information

Authors and Affiliations

  1. San Diego State University, 5500 Campanile Drive, San Diego, CA, 92182, USA

    Kelsey S. Dickson & Jessica Suhrheinrich

  2. Child and Adolescent Services Research Center, 3020 Children’s Way MC5033, San Diego, CA, 92123, USA

    Kelsey S. Dickson & Jessica Suhrheinrich

Authors
  1. Kelsey S. Dickson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jessica Suhrheinrich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kelsey S. Dickson.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dickson, K.S., Suhrheinrich, J. Concordance between Community Supervisor and Provider Ratings of Fidelity: Examination of Multi-Level Predictors and Outcomes. J Child Fam Stud 30, 542–555 (2021). https://doi.org/10.1007/s10826-020-01877-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10826-020-01877-0

Keywords

Search

Navigation