Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation

Abstract

The first purpose of this review was to assess the impact of implementation on program outcomes, and the second purpose was to identify factors affecting the implementation process. Results from over quantitative 500 studies offered strong empirical support to the conclusion that the level of implementation affects the outcomes obtained in promotion and prevention programs. Findings from 81 additional reports indicate there are at least 23 contextual factors that influence implementation. The implementation process is affected by variables related to communities, providers and innovations, and aspects of the prevention delivery system (i.e., organizational functioning) and the prevention support system (i.e., training and technical assistance). The collection of implementation data is an essential feature of program evaluations, and more information is needed on which and how various factors influence implementation in different community settings.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. 1.

    We use the term dissemination to refer to the first of Roger’s stages of diffusion although some apply the term to multiple stages of the diffusion process.

  2. 2.

    These journals included American Journal of Community Psychology, Health Education and Behavior, Health Education Research, Journal of Community Psychology, Journal of Primary Prevention, and Prevention Science.

  3. 3.

    We refer here to traditional distinctions regarding what type of data were collected and how the data were analyzed. Several studies combined qualitative and quantitative methods.

  4. 4.

    The risk status of students was the most important factor; i.e., students selected for intervention because of their early aggressive behavior improved the most.

  5. 5.

    A fourth factor, gender of the participants, was also related to program outcomes but was not included in their subsequent analyses.

  6. 6.

    Based on their pioneering survey of school-based implementation, Berman and McLaughlin (1976) were the first to stress the importance of mutual adaptation, that is, the organization should adapt to the innovation at the same time as the innovation is adapted to fit the organization. To our knowledge, the extent to which this has occurred during the diffusion of prevention or promotion programs has not been assessed.

References

  1. *Abbott, R. D., O’Donnell, J., Hawkins, J. D., Hill, K. G., Kosterman, R., & Catalano, R. F. (1998). Changing teaching practices to promote achievement and bonding to school. American Journal of Orthopsychiatry, 68, 542–552.

    PubMed  Google Scholar 

  2. *Aber, J. L., Jones, S. M., Brown, J. L., Chaudry, N., & Samples, F. (1998). Resolving conflict creatively: Evaluating the developmental effects of a school based violence prevention program in neighborhood and classroom context. Development and Psychopathology, 10, 187–213.

    PubMed  Google Scholar 

  3. *Allison, K. R., Silverman, G., & Dignam, C. (1990). Effects on students of teacher training in use of a drug education curriculum. Journal of Drug Education, 20, 31–46.

    PubMed  Google Scholar 

  4. Altschuld, J. W., Kumar, D. D., Smith, D. W., & Goodway, J. D. (1999). School-based educational innovations: Case illustrations of context sensitive evaluations. Family and Community Health, 22, 66–79.

    Google Scholar 

  5. American Psychological Association. (2003). Guidelines on multicultural education, training, research, practice and organizational change for psychologists. American Psychologist, 58, 377–402.

    Google Scholar 

  6. Ary, D. V., Biglan, A., Glasgow, R., Zoref, L., Black, C., Ochs, L., et al. (1990). The efficacy of social-influence prevention programs versus “standard care”: Are new initiatives needed? Journal of Behavioral Medicine, 13, 281–296.

    PubMed  Google Scholar 

  7. *August, G. J., Bloomquist, M. L., Lee, S. S., Realmuto, G. M., & Hektner, J. M. (2006). Can evidence-based prevention programs be sustained in community practice settings? The Early Risers; advanced-stage effectiveness trial. Prevention Science, 7, 151–165.

    PubMed  Google Scholar 

  8. *August, G. J., Egan, E. A., Realmuto, G. M., & Hektner, J. M. (2003a). Parceling component effects of a multifaceted prevention program for disruptive elementary school children. Journal of Abnormal Child Psychology, 31, 515–527.

    PubMed  Google Scholar 

  9. *August, G. J., Lee, S. S., Bloomquist, M. L., Realmuto, G. M., & Hektner, J. M. (2003b). Dissemination of an evidence-based preventive intervention for aggressive children living in diverse, urban neighborhoods. Prevention Science, 4, 271–286.

    PubMed  Google Scholar 

  10. *Backer, T. E. (2002). Finding the balance: Program fidelity and adaptation in substance abuse prevention: A state-of-the-art review. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration.

    Google Scholar 

  11. *Barr, J. E., Tubman, J. G., Montgomery, M. J., & Soza-Vento, R. M. (2002). Amenability and implementation in secondary school antitobacco programs. American Journal of Health Behavior, 26, 3–15.

    PubMed  Google Scholar 

  12. *Basch, C. E., Sliepcevich, E. M., Gold, R. S., Duncan, D. F., & Kolbe, L. J. (1985). Avoiding type III errors in health education program evaluations: A case study. Health Education Quarterly, 12, 315–331.

    PubMed  Google Scholar 

  13. *Basen-Engquist, K., O’Hara-Tompkins, N., Lovato, C. Y., Lewis, M. J., Parcel, G. S., & Gingiss, P. (1994). The effect of two types of teacher training on implementation of smart choices: A tobacco prevention curriculum. Journal of School Health, 64, 334–339.

    PubMed  Google Scholar 

  14. *Battistich, V., Schaps, E., Watson, M., Solomon, D., & Lewis, C. (2000). Effects of the child development project on students’ drug use and other problem behaviors. The Journal of Primary Prevention, 21, 75–99.

    Google Scholar 

  15. *Battistich, V., Schaps, E., & Wilson, N. (2004). Effects of an elementary school intervention on students’ “connectedness” to school and social adjustment during middle school. Journal of Primary Prevention, 24, 243–262.

    Google Scholar 

  16. *Bauman, L., Stein, R., & Ireys, H. (1991). Reinventing fidelity: The transfer of social technology among settings. American Journal of Community Psychology, 19, 619–639.

    PubMed  Google Scholar 

  17. *Bell, M. L., Kelley-Baker, T., Rider, R., & Ringwalt, C. (2005). Protecting You/Protecting me: Effects of an alcohol prevention and vehicle safety program on elementary students. Journal of School Health, 75, 171–177.

    PubMed  Google Scholar 

  18. *Bellg, A., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23, 443–451.

    PubMed  Google Scholar 

  19. *Berman, P., & McLaughlin, M. W. (1976). Implementation of educational innovation. The Educational Forum, 40, 345–370.

    Google Scholar 

  20. *Binford, V. M., & Newwell, J. M. (1991). Richmond, Virginia’s two decades of experience with Ira Gordon’s approach to parent education. The Elementary School Journal, 91, 233–237.

    Google Scholar 

  21. *Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., et al. (1987). The fidelity-adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253–268.

    Google Scholar 

  22. *Bloom, H. S., Hill, C. J., & Riccio, J. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22, 551–575.

    Google Scholar 

  23. Botvin, G. J. (2000). Preventing drug abuse in schools: Social and competence enhancement approaches targeting individual-level etiologic factors. Addictive Behaviors, 25, 887–897.

    PubMed  Google Scholar 

  24. *Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106–1112.

    PubMed  Google Scholar 

  25. *Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E. (1990). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3 year study. Journal of Consulting and Clinical Psychology, 58, 437–446.

    PubMed  Google Scholar 

  26. *Botvin, G. J., Baker, E., Filazzola, A. D., & Botvin, E. M. (1990). A cognitive-behavioral approach to substance abuse prevention: 1 Year follow-up. Addictive Behaviors, 15, 47–63.

    PubMed  Google Scholar 

  27. *Botvin, G. J., Baker, E., Renick, N. L., Filazzola, A. D., & Botvin, E. M. (1994). A cognitive-behavioral approach to substance abuse prevention. Addictive Behaviors, 9, 137–147.

    Google Scholar 

  28. *Botvin, G. J., Dusenbury, L., Baker, E., James-Ortiz, S., Botvin, E. M., & Kerner, J. (1992). Smoking prevention among urban minority youth: Assessing effects on outcome and mediating variables. Health Psychology, 11, 290–299.

    PubMed  Google Scholar 

  29. *Botvin, G. J., Dusenbury, L., Baker, E., James-Ortiz, S., & Kerner, J. (1989). A skills training approach to smoking prevention among Hispanic youth. Journal of Behavioral Medicine, 12, 279–296.

    PubMed  Google Scholar 

  30. *Bush, P. J., Zuckerman, A. E., Taggart, V. S., Theiss, P. K., Peleg, E. O., & Smith, S. A. (1989). Cardiovascular risk factor prevention in black school children: The “Know Your Body” evaluation project. Health Education Quarterly, 16, 215–227.

    PubMed  Google Scholar 

  31. Chinman, M., Hannah, G., Wandersman, A., Ebener, P. P., Hunter, S. B., Imm, P., et al. (2005). Developing a community science research agenda for building community capacity for effective preventive interventions. American Journal of Community Psychology, 35, 143–157.

    PubMed  Google Scholar 

  32. *Chinman, M., Imm, P., & Wandersman, A. (2004). Getting to outcomes 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation (Technical Report). Santa Monica, CA: RAND Corporation.

    Google Scholar 

  33. *Cho, H., Hallfors, D. D., & Sanchez, V. (2005). Evaluation of a high school peer group intervention for at-risk youth. Journal of Abnormal Child Psychology, 33, 363–374.

    PubMed  Google Scholar 

  34. *Conduct Problems Prevention Research Group. (1999). Initial impact of the Fast Track prevention trial for conduct problems: II. Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.

    Google Scholar 

  35. Connell, D. B., Turner, R. R., & Mason, E. F. (1985). Summary of findings on the school health evaluation: Health promotion, effectiveness, implementation, and costs. Journal of School Health, 55, 316–321.

    PubMed  Google Scholar 

  36. *Cook, T. D., Habib, F. N., Phillips, M., Settersten, R. A., Shagle, S. C., & Degirmencioglu, S. M. (1999). Comer’s school development program in prince George’s county, Maryland: A theory-based evaluation. American Educational Research Journal, 36, 543–597.

    Google Scholar 

  37. *Cook, T. D., Murphy, R. F., & Hunt, H.D. (2000). Comer’s school development program in Chicago: A theory-based evaluation. American Educational Research Journal, 37, 535–597.

    Google Scholar 

  38. *Cooke, M. (2000). The dissemination of a smoking cessation program: Predictors of program awareness, adoption and maintenance. Health Promotion International, 15, 113–124.

    Google Scholar 

  39. *Cullen, K. W., Baranowski, T., Baranowski, J., Herbert, D., deMoor, C., Hearn, M. D., et al. (1999). Influence of school organizational characteristics on the outcomes of a school health promotion program. Journal of School Health, 69, 376–380.

    PubMed  Google Scholar 

  40. *Cunningham, P. B., & Henggeler, S. W. (2001). Implementation of an empirically based drug and violence prevention and intervention program in public school settings. Journal of Consulting and Clinical Psychology, 30, 221–232.

    Google Scholar 

  41. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control. Clinical Psychology Review, 18, 23–45.

    PubMed  Google Scholar 

  42. *Derzon, J. H., Sale, E., Springer, J. F., & Brounstein, P. (2005). Estimating intervention effectiveness: Synthetic projection of field evaluation results. The Journal of Primary Prevention, 26, 321–343.

    PubMed  Google Scholar 

  43. Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221.

    Google Scholar 

  44. *Dubas, J. S., Lynch, K. B., Galano, J., Geller, S., & Hunt, D. (1998). Preliminary evaluation of a resiliency-based preschool substance abuse and violence prevention project. Journal of Drug Education, 28, 235–255.

    PubMed  Google Scholar 

  45. *DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157–198.

    PubMed  Google Scholar 

  46. *Dufrene, B. A., Noell, G. H., Gilbertson, D. N., & Duhon, G. J. (2005). Monitoring implementation of reciprocal peer tutoring: Identifying and intervening with students who do not maintain accurate implementation. School Psychology Review, 34, 74–86.

    Google Scholar 

  47. Durlak, J. A. (1997). Successful prevention programs for children and adolescents. New York: Plenum.

    Google Scholar 

  48. Durlak, J. A. (1985). School-based prevention programs for children and adolescents. Thousand Oaks, CA: Sage.

  49. Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention & Intervention in the Community, 17, 5–18.

    Google Scholar 

  50. Dusenbury, L., Brannigan, R., Falco, F., & Hansen, W. B. (2003a). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.

    PubMed  Google Scholar 

  51. *Dusenbury, L., Brannigan, R., Falco, M., & Lake, A. (2003b). An exploration of fidelity of implementation in drug abuse prevention among five professional groups. Journal of Alcohol and Drug Education, 47, 4–19.

    Google Scholar 

  52. *Elder, J. P., Perry, C. L., Stone, E. J., Johnson, C. C., Yang, M., Edmundson, E. W., et al. (1996). Tobacco use measurement, prediction, and intervention in elementary schools in four states: The CATCH study. Preventive Medicine, 25, 489–494.

    Google Scholar 

  53. *Elias, M. J., Gara, M., Ubriaco, M., Rothbaum, P. A., Clabby, J. F., & Schuyler, T. (1986). Impact of a preventive social problem solving intervention on children’s coping with middle-school stressors. American Journal of Community Psychology, 14, 259–275.

    PubMed  Google Scholar 

  54. *Elias, M. J., Zins, J. E., Graczyk, P. A., & Weissberg, R. P. (2003). Implementation, sustainability, and scaling up of social-emotional and academic innovations in public schools. School Psychology Review, 32, 303–319.

    Google Scholar 

  55. *Engstrom, M., Jason, L. A., Townsend, S. M., Pokorny, S. B., & Curie, C. J. (2002). Community readiness for prevention: Applying stage theory to multi-community interventions. Journal of Prevention & Intervention in the Community, 24, 29–46.

    Google Scholar 

  56. *Epstein, A. S. (1993). Training for quality: Improving early childhood programs systematic training. Monographs of the High/Scope Educational Research Foundation, No. 9. Ypsilanti, MI: The High/Scope Press.

    Google Scholar 

  57. *Fagan, A. A., & Mihalic, S. (2003). Strategies for enhancing the adoption of school-based prevention programs: Lessons learned from the blueprints for violence prevention replications of the life skills training program. Journal of Community Psychology, 31, 235–253.

    Google Scholar 

  58. Felner, R. D., Favazza, A., Shim, M., Brand, S., Gu, K., & Noonan, N. (2001). Whole school improvement and restructuring as prevention and promotion. Journal of School Psychology, 39, 177–202.

    Google Scholar 

  59. *Felner, R. D., Jackson, A., Kasak, D., Mulhall, P., Brand, S., & Flowers, N. (1997). The impact of middle school reform for the middle years: Longitudinal study of a network engaged in Turning Points-based comprehensive school transformation. Phi Delta Kappan, 78, 528–532.

    Google Scholar 

  60. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Retrieved November 1, 2006, from http://nirn.fmhi.usf.edu/resources/publications/Monograph/pdf/monograph_full.pdf.

  61. *Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of component adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13.

    PubMed  Google Scholar 

  62. *Fors, S. W., & Doster, M. E. (1985). Implications of results: Factors for success. Journal of School Health, 55, 332–334.

    PubMed  Google Scholar 

  63. *Fullan, M., & Pomfret, A. (1977). Research on curriculum and instruction implementation. Review of Educational Research, 47, 335–397.

    Google Scholar 

  64. *Gerstenblith, S. A., Soule, D. A., Gottfredson, D. C., Lu, S. Kellstrom, M. A., Womer, S. C., et al. (2005). After-school programs, antisocial behavior, and positive youth development: An exploration of the relationship between program implementation and changes in youth behavior. In J. L. Mahoney, J. S. Eccles & R. W. Larson (Eds.), Organized activities as contexts of development: Extracurricular activities, after-school and community programs. Mahwah, NJ: Erlbaum.

  65. *Gingiss, P. L. (1992). Enhancing program implementation and maintenance through a multi-phase approach to peer-based staff development. Journal of School Health, 62, 161–166.

    PubMed  Google Scholar 

  66. *Gingiss, P. L., Gottlieb, N. H., & Brink, S. G. (1994). Increasing teacher receptivity toward use of tobacco prevention education programs. Journal of Drug Education, 24, 163–176.

    PubMed  Google Scholar 

  67. *Gingiss, P. M., Roberts-Gray, C., & Boerm, M. (2006). Bridge-It: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207.

    PubMed  Google Scholar 

  68. *Goldman, K. D. (1994). Perceptions of innovations as predictors of implementation levels: The diffusion of a nationwide health education campaign. Health Education Quarterly, 21, 433–443.

    PubMed  Google Scholar 

  69. *Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39, 3–35.

    Google Scholar 

  70. *Gottfredson, D. C., Gottfredson, G. D., & Hybl, L. G. (1993). Managing adolescent behavior: A multiyear, multischool study. American Educational Research Journal, 30, 179–215.

    Google Scholar 

  71. Granner, M. L., & Sharpe, P. A. (2004). Evaluating community coalition characteristics and functioning: A summary of measurement tools. Health Education Research, 19, 514–532.

    PubMed  Google Scholar 

  72. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Diffusion of innovations in health service organizations: A systematic literature review. Oxford: Blackwell.

    Google Scholar 

  73. *Greenwood, C. R., Dinwiddie, G., Bailey, V., Carta, J. J., Dorsey, D., Kohler, F. W., et al. (1987). Field replication of classwide peer tutoring. Journal of Applied Behavior Analysis, 20, 151–160.

    PubMed  Google Scholar 

  74. *Greenwood, C. R., Tapia, Y., Abbott, M., & Walton, C. (2003). A building-based case study of evidence-based literacy practices: Implementation, reading behavior, and growth in reading fluency, K-4. The Journal of Special Education, 37, 95–110.

    Google Scholar 

  75. *Greenwood, C. R., Terry, B., Arreaga-Mayer, C., & Finney, R. (1992). The classwide peer tutoring program: Implementation factors moderating students’ achievement. Journal of Applied Behavior Analysis, 25, 101–116.

    PubMed  Google Scholar 

  76. *Hahn, E. J., Noland, M. P., Rayens, M. K., & Christie, D. M. (2002). Efficacy of training and fidelity of implementation of the life skills training program. Journal of School Health, 72, 282–287.

    PubMed  Google Scholar 

  77. *Hansen, W. B., Graham, J. W., Wolkenstein, B. H., & Rohrbach, L. A. (1991). Program integrity as a moderator of prevention program effectiveness: Results for fifth grade students in the adolescent alcohol prevention trial. Journal of Studies on Alcohol, 52, 568–579.

    PubMed  Google Scholar 

  78. *Hansen, W. B., & McNeal, R. B. (1999). Drug education practice: Results of an observational study. Health Education Research, 14, 85–97.

    PubMed  Google Scholar 

  79. *Harachi, T. W., Abbott, R. D., Catalano, R. F., Haggerty, K. P., & Fleming, C. B. (1999). Opening the black box: Using process evaluation measures to assess implementation and theory building. American Journal of Community Psychology, 27, 711–731.

    PubMed  Google Scholar 

  80. *Henderson, J. L., MacKay, S., & Peterson-Badali, M. (2006). Closing the research-practice gap: Factors affecting adoption and implementation of a children’s mental health program. Journal of Clinical Child and Adolescent Psychology, 35, 2–12.

    PubMed  Google Scholar 

  81. *Hogan, J. A., Baca, I., Daley, C., Garcia, T., Jaker, J., Lowther, M., et al. (2003). Disseminating science-based prevention: Lessons learned from CSAP’s CAPTs. Journal of Drug Education, 33, 233–244.

    PubMed  Google Scholar 

  82. Hogue, A., Liddle, H. A., Singer, A., & Leckrone, J. (2005). Intervention fidelity in family-based prevention counseling for adolescent problem behaviors. Journal of Community Psychology, 33, 191–211.

    Google Scholar 

  83. *Hopkins, R. H., Mauss, A. L., Kearney, K. A., & Weisheit, R. A. (1988). Comprehensive evaluation of a model alcohol education curriculum. Journal of Studies on Alcohol, 49, 38–50.

    PubMed  Google Scholar 

  84. *Hopper, C. A., Munoz, K. D., Gruber, M. B., MacConnie, S., Schonfeldt, B., & Shunk, T. (1996). A school-based cardiovascular exercise and nutrition program with parent participation: An evaluation study. Children’s Health Care, 25, 221–235.

    Google Scholar 

  85. *Ialongo, N., Werthamer, L., Kellam, S. G., Brown, C. H., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599–641.

    PubMed  Google Scholar 

  86. *James, S., Reddy, P., Ruiter, A. M., & van den Borne, B. (2006). The impact of an HIV and AIDS life skills program on secondary school students in Kwazulu-Natal, South Africa. AIDS Education and Prevention, 18, 281–294.

    Google Scholar 

  87. *Jason, L. A., Pokorny, S. B., Kunz, C., & Adams, M. (2004). Maintenance of community change: Enforcing youth access to tobacco laws. Journal of Drug Education, 34, 105–119.

    PubMed  Google Scholar 

  88. *Kalafat, J., & Ryerson, D. M. (1999). The implementation and institutionalization of a school-based youth suicide prevention program. Journal of Primary Prevention, 19, 157–175.

    Google Scholar 

  89. *Kallestad, J. H., & Olweus, D. (2003). Predicting teachers’ and schools’ implementation of the Olweus bullying prevention program: A multilevel study. Prevention & Treatment, 6. Retrieved June 20, 2005.

  90. *Kam, C.-M., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.

    PubMed  Google Scholar 

  91. Kazdin, A. E., Bass, D., Ayers, W. A., & Rodgers, A. (1990). Empirical and clinical focus of child and adolescent psychotherapy research. Journal of Consulting and Clinical Psychology, 58, 729–740.

    PubMed  Google Scholar 

  92. *Kealey, K. A., Peterson, A. V., Gaul, M. A., & Dinh, K. T. (2000). Teacher training as a behavior change process: Principles and results from a longitudinal study. Health Education & Behavior, 27, 64–81.

    Google Scholar 

  93. *Kegler, M. C., Steckler, A., Malek, S. H., & McLeroy, K. (1998a). A multiple case study of implementation in 10 local project ASSIST coalitions in North Carolina. Health Education Research, 13, 225–238.

    PubMed  Google Scholar 

  94. *Kegler, M. C., Steckler, A., McLeroy, K., & Malek, S. H. (1998b). Factors that contribute to effective community health promotion coalitions: A study of 10 project ASSIST coalitions in North Carolina. Health Education & Behavior, 25, 338–353.

    Google Scholar 

  95. *Kegler, M. C., & Wyatt, V. H. (2003). A multiple case study of neighborhood partnerships for positive youth development. American Journal of Health Behavior, 27, 156–169.

    PubMed  Google Scholar 

  96. *Kelly, J. A., Somlai, A. M., DiFranceisco, W. J., Otto-Salaj, L. L., McAuliffe, T. L., Hackl, K. L., et al. (2000). Bridging the gap between the science and service of HIV prevention: Transferring effective research-based HIV prevention interventions to community AIDS service providers. American Journal of Public Health, 90, 1082–1088.

    PubMed  Google Scholar 

  97. Kendrick, J. S., Zahniser, C., Miller, N., Salas, N., Stine, J. Gargiuillo, P. M., et al. (1995). Integrating smoking cessation into routine public prenatal care: The smoking cessation pregnancy project. American Journal of Public Health, 85, 217–222.

    PubMed  Google Scholar 

  98. *Kerr, D. M., Kent, L., & Lam, T. C. M. (1985). Measuring program implementation with a classroom observation instrument: The interactive teaching map. Evaluation Review, 9, 461–482.

    Google Scholar 

  99. *Komro, K. A., Perry, C. L., Veblen-Mortenson, S., Bosma, L. M., Dudovitz, B. S., Williams, C. L., et al. (2004). Brief report: The adaptation of Project Northland for urban youth. Journal of Pediatric Psychology, 29, 457–466.

    PubMed  Google Scholar 

  100. *Komro, K. A., Perry, C. L., Veblen-Mortenson, S., Farbakhsh, K., Kugler, K. C., Alfano, K. A., et al. (2006). Cross-cultural adaptation and evaluation of a home-based program for alcohol use prevention among urban youth: The “Slick Tracy Home Team Program”. The Journal of Primary Prevention, 27, 135–154.

    PubMed  Google Scholar 

  101. *Kramer, L., Laumann, G., & Brunson, L. (2000). Implementation and diffusion of the rainbows program in rural communities: Implications for school-based prevention programming. Journal of Educational and Psychological Consultation, 11, 37–64.

    Google Scholar 

  102. *Lapan, R. T., Gysbers, N. C., & Petroski, G. F. (2001). Helping seventh graders be safe and successful: A statewide study of the impact of comprehensive guidance and counseling programs. Journal of Consulting & Development, 79, 320–330.

    Google Scholar 

  103. *Levenson-Gingiss, P., Gottlieb, N. H., & Brink, S. G. (1994). Increasing teacher receptivity toward use of tobacco prevention education programs. Journal of Drug Education, 24, 163–176.

    Google Scholar 

  104. *Lillehoj, C. J. G., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257.

    Google Scholar 

  105. *Lynch, K. B., Geller, S. R., Hunt, D. R., Galano, J., & Dubas, J. S. (1998). Successful program develoment using implementation evaluation. Journal of Prevention & Intervention in the Community, 17, 51–64.

    Google Scholar 

  106. *MacDonald, M. A., & Green, L. W. (2001). Reconciling concept and context: The dilemma of implementation in school-based health promotion. Health Education & Behavior, 28, 749–768.

    Google Scholar 

  107. *McCormick, L. K., Steckler, A., & McLeroy, K. R. (1994). Diffusion of innovation in schools: A study of adoption and implementation of school-based tobacco prevention curricula. American Journal of Health Promotion, 9, 210–219.

    Google Scholar 

  108. *McGraw, S., Sellers, D., Stone, E., Bebchuk, J., Edmundson, E., Johnson, C., et al. (1996). Using process data to explain outcomes: An illustration from the child and adolescent trial for cardiovascular health (CATCH). Evaluation Review, 20, 291–312.

    PubMed  Google Scholar 

  109. *McKenzie, T. L., Nader, P. R., Strikmiller, P. K., Yang, M., Stone, E. J., erry, C. L, et al. (1996). School physical education: Effect of the child and adolescent trial for cardiovascular health. Preventive Medicine, 25, 423–431.

    PubMed  Google Scholar 

  110. *Mihalic, S., Irwin, K., Fagan, A., Ballard, D., & Elliott, D. (2004). Successful program implementation: Lessons from blueprints. Electronic report. U.S. Department of Justice, Office of Justice Programs). Retrieved August 10, 2006, from http://www.ojp.usdoj.gov/ojjdp.

  111. Miranda, J., Bernal, G., Lau, A., Kohn, L., Hwangh, W., & LaFromboise, T. (2005). State of the science on psychosocial interventions for ethnic minorities. Annual Review of Clinical Psychology, 1, 113–142.

    PubMed  Google Scholar 

  112. Mitchell, C. M. (1983). The dissemination of a social intervention: Process and effectiveness of two types of paraprofessional change agents. American Journal of Community Psychology, 11, 723–738.

    PubMed  Google Scholar 

  113. *Moskowitz, J. M., Schaps, E., & Malvin, J. H. (1982). Process and outcome evaluations in primary prevention: The magic circle program. Evaluation Review, 6, 775–788.

    Google Scholar 

  114. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Developmental, measurement, and validation. American Journal of Evaluation, 24, 315–340.

    Google Scholar 

  115. Nastasi, B. K., & Schensul, S. L. (2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43, 177–195.

    Google Scholar 

  116. National Research Council, Institute of Medicine. (2002). Community programs to promote youth development. Washington, DC: National Academy Press.

    Google Scholar 

  117. *Noell, G. H., Duhon, G. J., Gatti, S. L., & Connell, J. E. (2002). Consultation, follow-up, and implementation of behavior management interventions in general education. School Psychology Review, 31, 217–234.

    Google Scholar 

  118. *Noell, G. H., & Witt, J. C. (1999). When does consultation lead to intervention implementation? The Journal of Special Education, 33, 29–35.

    Google Scholar 

  119. *Noell, G. H., Witt, J. C., Gilbertson, D. N., Ranier, D. D., & Freeland, J. T. (1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77–88.

    Google Scholar 

  120. *Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L. et al. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87–106.

    Google Scholar 

  121. *Parcel, G. S., Ross, J. G., Lavin, A. T., Portnoy, B., Nelson, G. D., & Winters, F. (1991). Enhancing implementation of the teenage health teaching modules. Journal of School Health, 61, 35–38.

    PubMed  Google Scholar 

  122. *Payne, A. A., Gottfredson, D. C., & Gottfredson, G. D. (2006). School predictors of the intensity of implementation of school-based prevention programs: Results from a national study. Prevention Science, 7, 225–237.

    PubMed  Google Scholar 

  123. *Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. B., Dwyer, J. H., Johnson, C. A. et al. (1990). Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264–289.

    Google Scholar 

  124. *Perry, C. L., Murray, D. M., & Griffin, G. (1990). Evaluating the statewide dissemination of smoking prevention curricula: Factors in teacher compliance. Journal of School Health, 60, 501–504.

    PubMed  Google Scholar 

  125. *Perry, C. L., Sellers, D. E., Johnson, C., Pedersen, S., Bachman, K. J., Parcel, G. S., et al. (1997). The Child and Adolescent Trial for Cardiovascular Health (CATCH): Intervention, implementation, and feasibility for elementary schools in the United States. Health Education and Behavior, 24, 717–735.

    Google Scholar 

  126. *Piper, D. L., King, M. J., & Moberg, D. P. (1993). Implementing a middle school health promotion research project: Lessons our textbook didn’t teach us. Evolution & Program Planning, 16, 171–180.

    Google Scholar 

  127. *Pluye, P., Potvin, L., Denis, J. L., Pelletier, J., & Mannoni, C. (2005). Program sustainability begins with the first events. Evaluation and Program Planning, 28, 123–137.

    Google Scholar 

  128. *Rankin, W., Tarnai, J., Fagan, N. J., Mauss, A. L., & Hopkins, R. H. (1978). An evaluation of workshops designed to prepare teachers in alcohol education. Journal of Alcohol & Drug Education, 23, 1–13.

    Google Scholar 

  129. Resnicow, K., Baranowski, T., Ahluwalia, J. S., & Braithwaite, R. L. (1999). Cultural sensitivity in public health: Defined and demystified. Ethnicity and Disease, 9, 10–21.

    PubMed  Google Scholar 

  130. *Resnicow, K., Cohn, L., Reinhardt, J., Cross, D., Futterman, R., Kirschner, E., et al. (1992). A 3-year evaluation of the Know Your Body program in inner-city schoolchildren. Health Education Quarterly, 19, 463–480.

    PubMed  Google Scholar 

  131. *Resnicow, K., Davis, M., Smith, M., Baranowski, T., Lin, L. S., Baranowski, J., et al. (1998a). Results of the TeachWell worksite wellness program. American Journal of Public Health, 88, 250–257.

    PubMed  Google Scholar 

  132. *Resnicow, K., Davis, M., Smith, M., Lazarus-Yaroch, A., Baranowski, T., Baranowski, J., et al. (1998b). How best to measure implementation of school health curricula: A comparison of three measures. Health Education Research, 13, 239–250.

    PubMed  Google Scholar 

  133. *Reynolds, K. D., Franklin, F. A., Leviton, L. C., Maloy, J., Harrington, K. F., Yaroch, A. L., et al. (2000). Methods, results, and lessons learned from process evaluation of the High 5 school-based nutrition intervention. Health Education & Behavior, 27, 177–186.

    Google Scholar 

  134. *Richard, L., Lehoux, P., Breton, E., Denis, J., Labrie, L., & Leonard, C. (2004). Implementing the ecological approach in tobacco control programs: Results of a case study. Evaluation and Program Planning, 27, 409–421.

    Google Scholar 

  135. *Riley, B. L. (2003). Dissemination of heart health promotion in the Ontario public health system: 1989–1999. Health Education Research, 18, 15–31.

    PubMed  Google Scholar 

  136. *Riley, B. L., Taylor, S. M., & Elliott, S. J. (2001). Determinants of implementing heart healthy promotion activities in Ontario public health units: A social ecological perspective. Health Education Research, 16, 425–441.

    PubMed  Google Scholar 

  137. *Riley, B. L., Taylor, S. M., & Elliott, S. J. (2003). Organizational capacity and implementation change: A comparative case study of heart health promotion in Ontario public health agencies. Health Education Research, 18, 754–769.

    PubMed  Google Scholar 

  138. *Ringwalt, C. L., Ennett, S., Johnson, R., Rohrbach, L. A., Simons-Rudolph, A., Vincus, A., et al. (2003). Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Education & Behavior, 30, 375–391.

    Google Scholar 

  139. *Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.

  140. *Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.

    PubMed  Google Scholar 

  141. *Ross, J. G., Kuepker, R. V., Nelson, G. D., Saavedra, P., & Hubbard, B. M. (1991). Teenage health teaching modules: Impact of teacher training on implementation and student outcomes. Journal of School Health, 61, 31–34.

    PubMed  Google Scholar 

  142. Saunders, R. P., Ward, D., Felton, G. M., Dowda, M., & Pate, R. R. (2006). Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP). Evaluation and Program Planning, 29, 352–364.

    Google Scholar 

  143. *Scheirer, M. A. (1990). The life cycle of an innovation: Adoption versus discontinuation of the fluoride mouth rinse program in schools. Journal of Health and Social Behavior, 31, 203–215.

    PubMed  Google Scholar 

  144. *Scheirer, M. A. (2005). Is sustainability possible: A review and commentary on empirical studies of program sustainability. American Journal of Evaluation, 26, 320–347.

    Google Scholar 

  145. *Scheirer, M. A., Shediac, M. C., & Cassady, C. E. (1995). Measuring the implementation of health promotion programs: The case of the Breast and Cervical Cancer Program in Maryland. Health Education Research, 10, 11–25.

    PubMed  Google Scholar 

  146. Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning for the sustainability of community-based health programs: Conceptual frameworks and future directions for research, practice and policy. Health Education Research, 13, 87–108.

    PubMed  Google Scholar 

  147. *Smith, D. W., McCormick, L. K., Steckler, A. B., & McLeroy, K. R. (1993). Teachers’ use of health curricula: Implementation of growing healthy, project SMART, and the teenage health teaching modules. Journal of School Health, 63, 349–354.

    PubMed  Google Scholar 

  148. *Smith, D. W., Redican, K. J., & Olsen, L. K. (1992). The longevity of growing healthy: An analysis of the eight original sites implementing the school health curriculum project. Journal of School Health, 62, 83–87.

    PubMed  Google Scholar 

  149. *Smith, J. D., Schneider, B. H., Smith, P. K., & Ananiadou, K. (2004). The effectiveness of whole-school antibullying programs: A synthesis of evaluation research. School Psychology Review, 33, 547–560.

    Google Scholar 

  150. *Solomon, D., Battistich, V., Watson, M., Schaps, E., & Lewis, C. (2000). A six-district study of educational change: Direct and mediated effects of the Child Development Project. Social Psychology Education, 4, 3–51.

    Google Scholar 

  151. *Spoth, R., Guyll, M., Trudeau, L., & Goldberg-Lillehoj, C. (2002). Two studies of proximal outcomes and implementation quality of universal preventive interventions in a community-university collaboration context. Journal of Community Psychology, 30, 499–518.

    Google Scholar 

  152. *Steckler, A., McLeroy, K. R., Goodman, R. M., Smith, D., Dawson, L., & Howell, K. (1989). The importance of school district policies in the dissemination of tobacco use curricula in North Carolina schools. Family & Community Health, 12, 14–25.

    Google Scholar 

  153. *Sterling-Turner, H. E., Watson, T. S., & Moore, J. W. (2002). The effects of direct training and treatment integrity on treatment outcomes in school consultation. School Psychology Quarterly, 17, 47–77.

    Google Scholar 

  154. *Stevens, V., Van Oost, P., & De Bourdeaudhuij, I. (2001). Implementation process of the Flemish antibullying intervention and relation with program effectiveness. Journal of School Psychology, 39, 303–317.

    Google Scholar 

  155. Stith, S., Pruitt, I., Dees, J., Fronce, M., Green, N., Som, A. et al. (2006). Implementing community-based prevention programming: A review of the literature. Journal of Primary Prevention, 27, 599–617.

    PubMed  Google Scholar 

  156. *Story, M., Mays, R. W., Bishop, D. B., Perry, C. L., Taylor, G., Smyth, M., et al. (2000). 5-a-day power plus: Process evaluation of a multicomponent elementary school program to increase fruit and vegetable consumption. Health Education & Behavior, 27, 187–200.

    Google Scholar 

  157. *Stotts, A. L., diClemente, C. C., & Dolan-Mullen, P. (2002). One-to-one: A motivational intervention for resistant pregnant smokers. Addictive Behaviors, 27, 275–292.

    PubMed  Google Scholar 

  158. *St. Pierre, T. L., & Kaltreider, D. L. (2001). Reflections on implementing a community agency-school prevention program. Journal of Community Psychology, 29, 107–116.

    Google Scholar 

  159. *St. Pierre, Y. L., & Kaltreider, L. (2004). Tales of refusal, adoption, and maintenance: Evidence-based substance abuse prevention via school extension collaborations. American Journal of Evaluation, 25, 479–491.

    Google Scholar 

  160. *Taggart, V. S., Bush, P. J., Zuckerman, A. E., & Theiss, P. K. (1990). A process evaluation of the District of Columbia “Know Your Body” project. Journal of School Health, 60, 60–66.

    PubMed  Google Scholar 

  161. *Tappe, M. K., Galer-Unti, R. A., & Bailey, K. C. (1995). Long-term implementation of the teenage health teaching modules by trained teachers: A case study. Journal of School Health, 65, 411–415.

    PubMed  Article  Google Scholar 

  162. *Telzrow, C. F., McNamara, K., & Hollinger, C. L. (2000). Fidelity of problem-solving implementation and relationship to student performance. School Psychology Review, 29, 443–461.

    Google Scholar 

  163. Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537–567.

    Google Scholar 

  164. *Tortu, S., & Botvin, G. (1989). School-based smoking prevention: The teacher training process. Preventive Medicine, 18, 280–289.

    PubMed  Google Scholar 

  165. *Vadasy, P. F., Jenkins, J. R., Antil, L. R., Phillips, N. B., & Pool, K. (1997). The research-to-practice ball game. Remedial and Special Education, 18, 143–156.

    Article  Google Scholar 

  166. *Vincent, M. L., Paine-Andrews, A., Fisher, J., Devereaux, R. S., Gonyear Dolan, H., Harris, K. J., et al. (2000). Replication of a community-based multicomponent teen pregnancy prevention model: Realities and challenges. Family and Community Health, 23, 28–45.

    Google Scholar 

  167. *Wall, M. A., Severson, H. H., Andrews, J. A., Lichtenstein, E., & Zoref, L. (1995). Pediatric office-based smoking intervention: Impact on maternal smoking and relapse. Pediatrics, 96, 622–628.

    PubMed  Google Scholar 

  168. Wandersman, A. (2003). Community science: Bridging the gap between science and practice with community-centered models. American Journal of Community Psychology, 31, 227–242.

    PubMed  Google Scholar 

  169. Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: The interactive systems framwork for dissemination and implementation. American Journal of Community Psychology, 37, (this issue). doi:10.1007/s10464-008-9174-z

  170. *Wandersman, A., Morissey, E., Davino, K., Seybolt, D., Crusto, C., Nation, M., et al. (1998). Comprehensive quality programming and accountability: Eight essential strategies for implementing successful prevention programs. The Journal of Primary Prevention, 19, 3–30.

    Google Scholar 

  171. Weisman, S. A., Womer, S. C., Kellstrom, M., Bryner, S., Kahler, A., Slocum, L. A., et al. (2003). Maryland after school grant program part 1: Report on the 2001–2002 school year evaluation of the phase 3 after school programs. Unpublished manuscript, University of Maryland, College Park.

  172. *Wiecha, J. L., El Ayadi, A. M., Fuemmeler, B. F., Carter, J. E., Handler, S., Johnson, S., et al. (2004). Diffusion of an integrated health education program in an urban school system: Planet Health. Journal of Pediatric Psychology, 29, 467–474.

    PubMed  Google Scholar 

  173. Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136–149.

    PubMed  Google Scholar 

  174. *Winfree, L. T., Lynsky, D. P., & Maupin, J. R. (1999). Developing local police and federal law enforcement partnerships: G.R.E.A.T. as a case study of policy implementation. Criminal Justice Review, 24, 145–168.

    Google Scholar 

  175. Wolff, T. (2001). A practitioner’s guide to successful coalitions. American Journal of Community Psychology, 29, 173–181.

    PubMed  Google Scholar 

Download references

Acknowledgement

Preparation of this paper was supported in part from a grant awarded to the first author by the William T. Grant Foundaton (Grant #2212).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Joseph A. Durlak.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Durlak, J.A., DuPre, E.P. Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation. Am J Community Psychol 41, 327 (2008). https://doi.org/10.1007/s10464-008-9165-0

Download citation

Keywords

  • Implementation
  • Youth programs
  • Prevention
  • Health promotion