Skip to main content
SpringerLink
Log in

Evaluating the Impact of Implementation Factors on Family-Based Prevention Programming: Methods for Strengthening Causal Inference

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

Despite growing recognition of the important role implementation plays in successful prevention efforts, relatively little work has sought to demonstrate a causal relationship between implementation factors and participant outcomes. In turn, failure to explore the implementation-to-outcome link limits our understanding of the mechanisms essential to successful programming. This gap is partially due to the inability of current methodological procedures within prevention science to account for the multitude of confounders responsible for variation in implementation factors (i.e., selection bias). The current paper illustrates how propensity and marginal structural models can be used to improve causal inferences involving implementation factors not easily randomized (e.g., participant attendance). We first present analytic steps for simultaneously evaluating the impact of multiple implementation factors on prevention program outcome. Then, we demonstrate this approach for evaluating the impact of enrollment and attendance in a family program, over and above the impact of a school-based program, within PROSPER, a large-scale real-world prevention trial. Findings illustrate the capacity of this approach to successfully account for confounders that influence enrollment and attendance, thereby more accurately representing true causal relations. For instance, after accounting for selection bias, we observed a 5 % reduction in the prevalence of 11th grade underage drinking for those who chose to receive a family program and school program compared to those who received only the school program. Further, we detected a 7 % reduction in underage drinking for those with high attendance in the family program.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Here we consider implementation in the context of a larger prevention effort, as opposed to a single program in isolation. In particular, we focus on the role of participant choice to enroll and engage programs as key factors that influence the success of prevention efforts (as opposed to more narrow definitions of implementation that focus on the role of a facilitator to adhere to program curricula, i.e., implementation quality).

References

  • Albin, J.B., Lee, B., Dumas, J., Slater, J., & Witmer, J. (1985). Parent training with Canadian families. Canada's Mental Health, December, 20–24.

  • August, G. J., Bloomquist, M. L., Lee, S. S., Realmuto, G. M., & Hektner, J. M. (2006). Can evidence-based prevention programs be sustained in community practice settings? The Early Risers' Advanced-Stage effectiveness trial. Prevention Science, 7, 151–165. doi:10.1007/s11121-005-0024-z.

    Article  PubMed  Google Scholar 

  • Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2010). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.

    Article  Google Scholar 

  • Brody, G. H., Murry, V. M., Chen, Y.-f., Kogan, S. M., & Brown, A. C. (2006). Effects of family risk factors on dosage and efficacy of a family-centered preventive intervention for rural African Americans. Prevention Science, 7, 281–291. doi:10.1007/s11121-006-0032-7.

    Article  PubMed  Google Scholar 

  • Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. doi:10.1186/1748-5908-2-40.

    Article  PubMed Central  PubMed  Google Scholar 

  • Chilenski, S. M., & Greenberg, M. T. (2009). The importance of the community context in the epidemiology of early adolescent substance use and delinquency in a rural sample. American Journal of Community Psychology, 44, 287–301. doi:10.1007/s10464-009-9258-4.

    Article  PubMed Central  PubMed  Google Scholar 

  • Coffman, D. L. (2011). Estimating causal effects in mediation analysis using propensity scores. Structural Equation Modeling, 18, 357–369.

    Article  PubMed Central  PubMed  Google Scholar 

  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155.

    Article  CAS  PubMed  Google Scholar 

  • Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005). A strategy for optimizing and evaluating behavioral interventions. Annals of Behavioral Medicine, 30, 65–73. doi:10.1207/s15324796abm3001_8.

    Article  PubMed  Google Scholar 

  • Connell, A. M., Dishion, T. J., Yasui, M., & Kavanagh, K. (2007). An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent problem behavior. Journal of Consulting and Clinical Psychology, 75, 568–579. doi:10.1037/0022-006X.75.4.568.

    Article  PubMed  Google Scholar 

  • D’Agostino, R. B. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17, 2265–2281.

    Article  PubMed  Google Scholar 

  • D’Agostino, R. B., & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95, 749–759.

    Article  Google Scholar 

  • Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & DeRousie, R. M. S. (2010). Implementation quality: Lessons learned in the context of the Head Start REDI trial. Early Childhood Research Quarterly, 25, 284–298. doi:10.1016/j.ecresq.2010.04.001.

    Article  PubMed Central  PubMed  Google Scholar 

  • Dumas, J. E., Moreland, A. D., Gitter, A. H., Pearl, A. M., & Nordstrom, A. H. (2006). Engaging parents in preventive parenting groups: Do ethnic, socioeconomic, and belief match between parents and group leaders matter? Health Education & Behavior, 35, 619–633. doi:10.1177/1090198106291374.

    Article  Google Scholar 

  • Dumas, J. E., Nissley-Tsiopinis, J., & Moreland, A. D. (2006). From intent to enrollment, attendance, and participation in preventive parenting groups. Journal of Child and Family Studies, 16, 1–26. doi:10.1007/s10826-006-9042-0.

    Article  Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.

    Article  PubMed  Google Scholar 

  • Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53. doi:10.1023/B:PREV.0000013981.28071.52.

    Article  PubMed  Google Scholar 

  • Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175. doi:10.1007/s11121-005-5553-y.

    Article  PubMed  Google Scholar 

  • Foster, M. (2003). Propensity score matching: An illustrative analysis of dose response. Medical Care, 41, 1183–1192.

    Article  PubMed  Google Scholar 

  • Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327. doi:10.2105/AJPH.89.9.1322.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Glasgow, R. E., McKay, H. G., Piette, J. D., & Reynolds, K. D. (2001). The RE-AIM framework for evaluating interventions: What can it tell us about approaches to chronic illness management? Patient Education and Counseling, 44, 119–127. doi:10.1016/S0738-3991(00)00186-5.

    Article  CAS  PubMed  Google Scholar 

  • Graczyk, P., Domitrovich, C., & Zins, J. (2007). Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. In M. Weist (Ed.), Handbook of school mental health: Advancing practice and research. New York: Springer.

    Google Scholar 

  • Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206–213. doi:10.1007/s11121-007-0070-9.

    Article  PubMed  Google Scholar 

  • Harder, V. S., Stuart, E. A., & Anthony, J. C. (2008). Adolescent cannabis problems and young adult depression: Male–female stratified propensity score analyses. American Journal of Epidemiology, 168, 592–601.

    Article  PubMed Central  PubMed  Google Scholar 

  • Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234–249.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hill, J. L., Brooks-Gunn, J., & Waldfogel, J. (2003). Sustained effects of high participation in an early intervention for low-birth-weight premature infants. Developmental Psychology, 39, 730–744.

    Article  PubMed  Google Scholar 

  • Hirano, K., & Imbens, G. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catherization. Health Services and Outcome Research Methodology, 2, 259–278.

    Article  Google Scholar 

  • Holden, G., Rosenberg, G., Barker, K., Tuhrim, S., & Brenner, B. (1993). The recruitment of research participants: A review. Social Work in Health Care, 19, 1–44.

    Article  CAS  PubMed  Google Scholar 

  • Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–970.

    Article  Google Scholar 

  • Imbens, G. W. (2000). The role of the propensity score in estimating dose–response functions. Biometrika, 87, 706–710.

    Article  Google Scholar 

  • Kam, C., Greenberg, M., & Walls, C. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.

    Article  PubMed  Google Scholar 

  • Littell, R., Milliken, G., Stroup, W., Wolfinger, R., & Schabenberger, O. (2006). SAS for mixed models (2nd ed.). Cary: SAS Institute.

    Google Scholar 

  • Little, R., & Rubin, D. (2002). Statistical analysis with missing data (2nd ed.). New York: John Wiley.

    Google Scholar 

  • Luellen, J. K., Shadish, W. R., & Clark, M. H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29, 530–558.

    Article  PubMed  Google Scholar 

  • McGowan, M., Nix, R., Murphy, S., Bierman, K., & Conduct Problems Prevention Research Group. (2010). Investigating the impact of selection bias in dose–response analyses of preventive interventions. Prevention Science, 11, 239–251.

    Article  PubMed Central  PubMed  Google Scholar 

  • Prinz, R. J., & Miller, G. E. (1994). Family-based treatment for childhood antisocial behavior: Experimental influences on dropout and engagement. Journal of Clinical Child Psychology, 62, 645–650.

    CAS  Google Scholar 

  • Redmond, C., Spoth, R. L., Shin, C., Schainker, L. M., Greenberg, M. T., & Feinberg, M. (2009). Long-term protective factor outcomes of evidence-based interventions implemented by community teams through a community–university partnership. The Journal of Primary Prevention, 30, 513–530. doi:10.1007/s10935-009-0189-5.

    Article  PubMed Central  PubMed  Google Scholar 

  • Robins, J., Hernan, M. A., & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 11, 550–560.

    Article  CAS  PubMed  Google Scholar 

  • Rohrbach, L. A., Hodgson, C. S., Broder, B. I., Montgomery, S. B., Flay, B. R., Hansen, W. B., et al. (1994). Parental participation in drug abuse prevention: Results from the midwestern prevention project. Journal of Research on Adolescence, 4, 295–317. doi:10.1207/s15327795jra0402_7.

    Article  Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

    Article  Google Scholar 

  • Rubin, D. B. (2005). Causal inference using potential outcomes. Journal of the American Statistical Association, 100, 322–331. doi:10.1198/016214504000001880.

    Article  CAS  Google Scholar 

  • Sampson, R. J., Sharkey, P., & Raudenbush, S. W. (2008). From the cover: Inaugural article: Durable effects of concentrated disadvantage on verbal ability among African-American children. Proceedings of the National Academy of Sciences, 105, 845–852. doi:10.1073/pnas.0710189104.

    Article  CAS  Google Scholar 

  • SAS Institute Inc. (2004). SAS/STAT 9.1 user's guide. Cary: SAS Institute Inc.

    Google Scholar 

  • Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177. doi:10.1037/1082-989X.7.2.147.

    Article  PubMed  Google Scholar 

  • Schafer, J. L., & Kang, J. D. Y. (2008). Average causal effects from non-randomized studies: A practical guide and simulated example. Psychological Methods, 13, 279–313.

    Article  PubMed  Google Scholar 

  • Spoth, R., & Greenberg, M. (2011). Impact challenges in community science-with-practice: Lessons from PROSPER on transformative practitioner-scientist partnerships and prevention infrastructure development. American Journal of Community Psychology, 48, 106–119. doi:10.1007/s10464-010-9417-7.

    Article  PubMed Central  PubMed  Google Scholar 

  • Spoth, R., & Redmond, C. (1995). Parent motivation to enroll in parenting skills programs: A model of family context and health belief predictors. Journal of Family Psychology, 9, 294–310.

    Article  Google Scholar 

  • Spoth, R., Redmond, C., Kahn, J., & Shin, C. (1997). A prospective validation study of inclination, belief, and context predictors of family-focused prevention involvement. Family Process, 36, 403–429.

    Article  CAS  PubMed  Google Scholar 

  • Spoth, R., Redmond, C., & Shin, C. (2000a). Modeling factors influencing enrollment in family-focused preventive intervention research. Prevention Science, 1, 213–225.

    Article  CAS  PubMed  Google Scholar 

  • Spoth, R., Redmond, C., & Shin, C. (2000b). Reducing adolescents' aggressive and hostile behaviors: Randomized trial effects of a brief family intervention 4 years past baseline. Archives of Pediatrics & Adolescent Medicine, 154, 1248–1257.

    Article  CAS  Google Scholar 

  • Spoth, R., Greenberg, M., Bierman, K., & Redmond, C. (2004). Prosper community–university partnership model for public education systems: Capacity-building for evidence-based, competence-building prevention. Prevention Science, 5, 31–39. doi:10.1023/B:PREV.0000013979.52796.8b.

    Article  PubMed  Google Scholar 

  • Spoth, R., Guyll, M., Lillehoj, C. J., Redmond, C., & Greenberg, M. (2007). Prosper study of evidence-based intervention implementation quality by community–university partnerships. Journal of Community Psychology, 35, 981–999. doi:10.1002/jcop.20207.

    Article  PubMed Central  PubMed  Google Scholar 

  • Spoth, R., Redmond, C., Shin, C., Greenberg, M., Clair, S., & Feinberg, M. (2007). Substance-use outcomes at 18 months past baseline. American Journal of Preventive Medicine, 32, 395–402. doi:10.1016/j.amepre.2007.01.014.

    Article  PubMed Central  PubMed  Google Scholar 

  • Spoth, R., Guyll, M., Redmond, C., Greenberg, M., & Feinberg, M. (2011). Six-year sustainability of evidence-based intervention implementation quality by community-university partnerships: The PROSPER study. American Journal of Community Psychology, 48, 412–425. doi:10.1007/s10464-011-9430-5.

    Article  PubMed Central  PubMed  Google Scholar 

  • Spreewenberg, M., Bartak, A., Croon, M., Hagenaars, J., Buschblach, J., Andrea, H., et al. (2010). The multiple propensity score as control for bias in the comparison of more than two treatment arms. Medical Care, 48, 166–174.

    Article  Google Scholar 

  • Stuart, E. A., Perry, D. F., Le, H.-N., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science, 9, 288–298. doi:10.1007/s11121-008-0104-y.

    Article  PubMed Central  PubMed  Google Scholar 

  • Taylor, B., Graham, J., Cumsille, P., & Hansen, W. (2000). Modeling prevention program effects on growth in substance use analysis of data from the adolescent alcohol prevention trial. Prevention Science, 4, 183–197. doi:10.1023/A:1026547128209.

    Article  Google Scholar 

  • Toomey, T. L., Williams, C. L., Perry, C. L., Murray, D. M., Dudoritz, B., & Veblen-Mortenson, S. (1996). An alcohol primary prevention program for parents of 7th graders: The amazing alternatives! Home Program. Journal of Child and Adolescent Substance Abuse, 5, 35–53.

    Article  Google Scholar 

  • Vanderweele, T. J. (2008). Ignorability and stability assumptions in neighborhood effects research. Statistics in Medicine, 27, 1934–1943. doi:10.1002/sim.3139.

    Article  PubMed  Google Scholar 

  • Williams, C. L., Perry, C. L., Dudovitz, B., Veblen-Mortenson, S., Anstine, P. S., Komro, K. A., et al. (1995). A home-based prevention program for sixth-grade alcohol use: Results from Project Northland. Journal of Primary Prevention, 16, 125–147.

    Article  CAS  PubMed  Google Scholar 

  • Winslow, E. B., Bonds, D., Wolchik, S., Sandler, I., & Braver, S. (2009). Predictors of enrollment and retention in a preventive parenting intervention for divorced families. The Journal of Primary Prevention, 30, 151–172. doi:10.1007/s10935-009-0170-3.

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Duke University, Durham, NC, USA

    D. Max Crowley

  2. The Pennsylvania State University, University Park, PA, USA

    Donna L. Coffman, Mark E. Feinberg & Mark T. Greenberg

  3. Iowa State University, Ames, IA, USA

    Richard L. Spoth

Authors
  1. D. Max Crowley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donna L. Coffman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark E. Feinberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark T. Greenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Richard L. Spoth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Max Crowley.

Additional information

This project was supported by the National Institute on Drug Abuse (NIDA) grants P50-DA010075-15, R03-DA026543, and T32-DA 0176. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIDA or the National Institutes of Health.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Crowley, D.M., Coffman, D.L., Feinberg, M.E. et al. Evaluating the Impact of Implementation Factors on Family-Based Prevention Programming: Methods for Strengthening Causal Inference. Prev Sci 15, 246–255 (2014). https://doi.org/10.1007/s11121-012-0352-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-012-0352-8

Keywords

Search

Navigation