Redesigning Implementation Measurement for Monitoring and Quality Improvement in Community Delivery Settings
The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the “New Beginnings Program” serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.
KeywordsImplementation Measurement Evidence-based programs Parenting Pragmatic measures Technology
Support for the development of this manuscript was provided by the National Institute of Drug Abuse: R01DA026874 (Sandler), R01DA033991 (Berkel and Mauricio), and competitive funding from the Center for Prevention Implementation Methodology (Ce-PIM), P30-DA027828 (Brown/Berkel), and Diversity Supplement R01DA033991-03S1 (Berkel/Gallo).
Compliance With Ethical Standards
Conflict of Interest
Sandler is a developer of the NBP and has an LLC that trains facilitators to deliver the program. Remaining authors declare that they have no conflict of interest.
Research Involving Human Participants
All procedures performed in studies involving human participants were in accordance with the ethical standards of Arizona State University’s IRB and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed Consent was obtained from all individual participants included in the study.
- Aarons, G. A., & Sawitzky, A. C. (2006). Organizational climate partially mediates the effect of culture on work attitudes and staff turnover in mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 33(3), 289–301. https://doi.org/10.1007/s10488-006-0039-1.CrossRefGoogle Scholar
- Berkel, C., Mauricio, A. M., Sandler, I. N., Wolchik, S. A., Gallo, C. G., & Brown, C. H. (2018). The cascading effects of multiple dimensions of implementation on program outcomes: A test of a theoretical model. Prevention Science, 19(6), 782–794. https://doi.org/10.1007/s11121-017-0855-4.CrossRefGoogle Scholar
- Berkel, C., Sandler, I. N., Wolchik, S. A., Brown, C. H., Gallo, C. G., Chiapa, A., et al. (2018). “Home practice is the program:” Parents’ practice of program skills as predictors of outcomes in the New Beginnings Program effectiveness trial. Prevention Science, 19(5), 663–673. https://doi.org/10.1007/s11121-016-0738-0.CrossRefGoogle Scholar
- Brown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L., Wingood, G., et al. (2013). A computational future for preventing HIV in minority communities: How advanced technology can improve implementation of effective programs. Journal of Acquired Immune Deficiency Syndromes, 63(Supp 1), S72–S84. https://doi.org/10.1097/QAI.0b013e31829372bd.CrossRefGoogle Scholar
- Can, D., Marín, R. A., Georgiou, P. G., Imel, Z. E., Atkins, D. C., & Narayanan, S. S. (2016). “It sounds like…:” A natural language processing approach to detecting counselor reflections in motivational interviewing. Journal of Counseling Psychology, 63(3), 343–350. https://doi.org/10.1037/cou0000111.CrossRefGoogle Scholar
- Chacko, A., Isham, A., Cleek, A. F., & McKay, M. M. (2016). Using mobile health technology to improve behavioral skill implementation through homework in evidence-based parenting intervention for disruptive behavior disorders in youth: Study protocol for intervention development and evaluation. Pilot and Feasibility Studies, 2, 57. https://doi.org/10.1186/s40814-016-0097-4.CrossRefGoogle Scholar
- Gallo, C. G., Berkel, C., Sandler, I. N., & Brown, C. H. (2015). Improving implementation of behavioral interventions by monitoring quality of delivery in speech. Paper presented at the annual conference on the Science of Dissemination and Implementation, Washington, DC.Google Scholar
- Gallo, C. G., Berkel, C., Sandler, I. N., & Brown, C. H. (2016). Developing computer-based methods for assessing quality of implementation in parent-training behavioral interventions. Paper presented at the annual meeting of the Society for Prevention Research, San Francisco, CA.Google Scholar
- Gallo, C. G., Li, Y., Berkel, C., Mehrotra, S., Liu, L., Benbow, N., et al. (under review). Recognizing emotion in speech for assessing the implementing behavioral interventions.Google Scholar
- Gallo, C. G., Pantin, H., Villamar, J., Prado, G., Tapia, M. I., Ogihara, M., et al. (2015). Blending qualitative and computational linguistics methods for fidelity assessment: Experience with the Familias Unidas preventive intervention. Administration and Policy in Mental Health and Mental Health Services Research, 42, 574–585. https://doi.org/10.1007/s10488-014-0538-4.CrossRefGoogle Scholar
- Herman, P. M., Mahrer, N. E., Wolchik, S. A., Porter, M. M., Jones, S., & Sandler, I. N. (2015). Cost-benefit analysis of a preventive intervention for divorced families: Reduction in mental health and justice system service use costs 15 years later. Prevention Science, 16, 586–596. https://doi.org/10.1007/s11121-014-0527-6.CrossRefGoogle Scholar
- Moos, R. (1981). Group environment scale manual. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
- NRC/IOM. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: NRC/IOM.Google Scholar
- Sandler, I. N., Schoenfelder, E. N., Wolchik, S. A., & MacKinnon, D. P. (2011). Long-term impact of prevention programs to promote effective parenting: Lasting effects but uncertain processes. Annual Review of Psychology, 62, 299–329. https://doi.org/10.1146/annurev.psych.121208.131619.CrossRefGoogle Scholar
- Sandler, I. N., Wolchik, S. A., Berkel, C., Jones, S., Mauricio, A. M., Tein, J.-Y., et al. (2016). Effectiveness trial of the New Beginnings Program (NBP) for divorcing and separating parents: Translation from an experimental prototype to an evidence-based community service. In M. Israelashvili & J. L. Romano (Eds.), Cambridge handbook of international prevention science (pp. 81–106). Cambridge: Cambridge University Press.Google Scholar
- Schoenfelder, E., Sandler, I. N., Millsap, R. E., Wolchik, S. A., Berkel, C., & Ayers, T. S. (2012). Responsiveness to the Family Bereavement Program: What predicts responsiveness? What does responsiveness predict? Prevention Science, 14, 545–556. https://doi.org/10.1007/s11121-012-0337-7.CrossRefGoogle Scholar
- Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J. A., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35, 84–97. https://doi.org/10.1007/s10488-007-0147-6.CrossRefGoogle Scholar
- Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration & Policy in Mental Health & Mental Health Services Research, 38, 32–43. https://doi.org/10.1007/s10488-010-0321-0.CrossRefGoogle Scholar
- Shermis, M., Burstein, J., Elliot, N., Miel, S., & Foltz, P. (2015). Automated writing evaluation: A growing body of knowledge. In C. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research. New York: Guilford Press.Google Scholar
- Wolchik, S. A., Sandler, I. N., Tein, J.-Y., Gunn, H., Mazza, G. L., Kim, H.-J., et al. (2016). Main and moderated effects of the New Beginnings Program versus a low dose comparison. Paper presented at the annual meeting of the society for prevention research, San Francisco, CA.Google Scholar
- Wolchik, S. A., Sandler, I. N., Tein, J.-Y., Mahrer, N. E., Millsap, R., Winslow, E., et al. (2013). Fifteen-year follow-up of a randomized trial of a preventive intervention for divorced families: Effects on mental health and substance use outcomes in young adulthood. Journal of Consulting and Clinical Psychology, 81, 660–673. https://doi.org/10.1037/a0033235.CrossRefGoogle Scholar
- Wolchik, S., Sandler, I., Weiss, L., & Winslow, E. (2007). New Beginnings: An empirically-based program to help divorced mothers promote resilience in their children. In J. M. Briesmeister & C. E. Schaefer (Eds.), Handbook of parent training: Helping parents prevent and solve problem behaviors (3rd ed., pp. 25–62). Hoboken, NJ: Wiley.Google Scholar
- Wolchik, S. A., West, S. G., Sandler, I. N., Tein, J.-Y., Coatsworth, D., Lengua, L., et al. (2000). An experimental evaluation of theory-based mother and mother–child programs for children of divorce. Journal of Consulting and Clinical Psychology, 68, 843–856. https://doi.org/10.1037/0022-006x.68.5.843.CrossRefGoogle Scholar