Abstract
Although there is an established literature supporting the efficacy of a variety of prevention programs, there has been less empirical work on the translation of such research to everyday practice or when scaled-up state-wide. There is a considerable need for more research on factors that enhance implementation of programs and optimize outcomes, particularly in school settings. The current paper examines how the implementation fidelity of an increasingly popular and widely disseminated prevention model called, School-wide Positive Behavioral Interventions and Supports (SW-PBIS), relates to student outcomes within the context of a state-wide scale-up effort. Data come from a scale-up effort of SW-PBIS in Maryland; the sample included 421 elementary and middle schools trained in SW-PBIS. SW-PBIS fidelity, as measured by one of three fidelity measures, was found to be associated with higher math achievement, higher reading achievement, and lower truancy. School contextual factors were related to implementation levels and outcomes. Implications for scale-up efforts of behavioral and mental health interventions and measurement considerations are discussed.
Similar content being viewed by others
Notes
This scale measures pre-implementation readiness and therefore the variability and the internal consistency on this scale are low.
References
Flay BR, Biglan A, Boruch RF, et al. Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science. 2005;6(3):151–175.
Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research. 2003;18(2):237–256.
Ringwalt CL, Ennett S, Johnson R, et al. Factors associated with fidelity to substance use prevention curriculum guides in the nation's middle schools. Health Education & Behavior. 2003;30(3):375–391.
SPR MAPS II Task Force. Type 2 translational research: Overview and definitions. 2008; http://preventionscience.org/SPR_Type%202%20Translation%20Research_Overview%20and%20Definition.pdf.
Woolf SH. The meaning of translational research and why it matters. Journal of the American Medical Association. 2008;299:211–213.
Woolf SH. Potential health and economic consequences of misplaced priorities. Journal of the American Medical Association. 2007;297:523–526.
Spoth R. Translating family-focused prevention science into effective practice: Toward a translational impact paradigm. Current Directions in Psychological Science. 2008;17(6):415–421.
Ringwalt C, Vincus AA, Hanley S, et al. The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2008. Prevention Science. 2011;12(1):63–69.
Sloboda Z, Pyakuryal A, Stephens PC, et al. Reports of substance abuse prevention programming available in schools. Prevention Science. 2008;9(4):276–287.
Gottfredson GD, Gottfredson DC, Czeh ER, et al. National study of delinquency prevention in schools. Ellicott City, MD: Gottfredson Associates; 2000.
Gottfredson GD, Jones EM, Gore TW. Implementation and evaluation of a cognitive–behavioral intervention to prevent problem behavior in a disorganized school. Prevention Science. 2002;3(1):43–56.
Domitrovich CE, Greenberg MT. The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational & Psychological Consultation. 2000;11(2):193–221.
Durlak JA. Successful prevention programs for children and adolescents. New York: Plenum; 1997.
Sugai G, Horner RH, Gresham FM. Behaviorally effective school environments. In: Shinn MR, Walker HM, Stoner G, eds. Interventions for academic and behavior problems: II. Preventive and remedial approaches. Bethesda, MD US: National Association of School Psychologists; 2002:315–350.
Domitrovich CE, Bradshaw CP, Poduska JM, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion. 2008;1(3):6–28.
Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services. 2001;52(9):1190–1197.
Stuart E, Cole S, Bradshaw CP, et al. The use of propensity scores to assess the generalizability of results from randomized trials. The Journal of the Royal Statistical Society, Series A. 2011; 174(2):369–386.
O’Connell ME, Boat T, Warner KE. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press. Committee on the Prevention of Mental Disorders and Substance Abuse Among Children, Youth, and Young Adults: Research Advances and Promising Interventions; 2009.
Bradshaw CP, Koth CW, Bevans KB, et al. The impact of school-wide Positive Behavioral Interventions and Supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly. 2008;23(4):462–473.
Bradshaw CP, Koth CW, Thornton LA, et al. Altering school climate through school-wide Positive Behavioral Interventions and Supports: Findings from a group-randomized effectiveness trial. Prevention Science. 2009;10(2):100–115.
Bradshaw CP, Mitchell MM, Leaf PJ. Examining the effects of schoolwide Positive Behavioral Interventions and Supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions. 2010;12(3):133–148.
Horner RH, Sugai G, Smolkowski K, et al. A randomized, wait-list controlled effectiveness trial assessing School-Wide Positive Behavior Support in elementary schools. Journal of Positive Behavior Interventions. 2009;11(3):133–144.
Bradshaw CP, Waasdorp TE, Leaf PJ. Effects of School-Wide Positive Behavioral Interventions and Supports on child behavior problems and adjustment. Manuscript submitted for publication. 2012.
Waasdorp TE, Bradshaw CP, Leaf PJ. The impact of School-wide Positive Behavioral Interventions and Supports (SWPBIS) on bullying and peer rejection: A randomized controlled effectiveness trial. Archives of Child and Adolescent Medicine. 2012;116(2):149–156.
Barrett SB, Bradshaw CP, Lewis-Palmer T. Maryland statewide PBIS initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions. 2008;10(2):105–114.
Muscott HS, Mann EL, LeBrun MR. Positive behavioral interventions and supports in New Hampshire. Effects of large-scale implementation of schoolwide positive behavior support on student discipline and academic achievement. Journal of Positive Behavior Interventions. 2008;10(3):190–205.
Bradshaw CP, Pas ET. A state-wide scale-up of Positive Behavioral Interventions and Supports (PBIS): A description of the development of systems of support and analysis of adoption and implementation. School Psychology Review. 2011;40(4):530–548.
Adelman HS, Taylor L. Toward a scale-up model for replicating new approaches to schooling. Journal of Educational & Psychological Consultation. 1997;8(2):197.
Kreuter MW, Bernhardt JM. Reframing the dissemination challenge: a marketing and distribution perspective. American Journal of Public Health. 2009;99(12):2123–2127.
Rohrbach LA, Grana R, Sussman S, et al. Type II Translation: Transporting prevention interventions from research to real-world settings. Evaluation & the Health Professions. 2006;29(3):302–333.
Bloomquist ML, August GJ, Horowitz JL, et al. Moving from science to service: Transposing and sustaining the early risers prevention program in a community service system. The Journal of Primary Prevention. 2008;29(4):307–321.
Fixsen DL, Naoom SF, Blase KA, et al. Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.
Spoth RL, Greenberg, MT. Toward a comprehensive strategy for effective practitioner–scientist partnerships and larger-scale community health and well-being. American Journal of Community Psychology. 2005;35:107–126.
Rogers EM. Diffusion of preventive innovations. Addictive Behaviors. 2002;27(6):989–993.
Botvin GJ, Baker E, Dusenbury L, et al. Long-term follow-up results of a randomized drug abuse prevention trial in a White middle-class population. Journal of the American Medical Association.1995;273(14):1106–1112.
Derzon JH, Sale E, Springer JF, et al. Estimating intervention effectiveness: Synthetic projection of field evaluation results. The Journal of Primary Prevention. 2005;26(4):321–343.
Durlak JA, DuPre EP. Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology. 2008;41(3–4):327–350.
Hulleman CS, Cordray, DS. Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness. 2009;2(1):88–110.
Ialongo NS, Werthamer L, Kellam SG, et al. Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology. 1999;27(5):599–641.
Sanders MR. Triple P-Positive Parenting Program: Towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clinical Child & Family Psychology Review. 1999;2(2):71–90.
Nowak C, Heinrichs N. A comprehensive meta-analysis of Triple P-Positive Parenting Program using hierarchical linear modeling: Effectiveness and moderating variables. Clinical Child and Family Psychology Review. 2008;11(3):114–144.
Kam C-M, Greenberg MT, Walls CT. Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science. 2003;4(1):55–63.
Jowers KL, Bradshaw CP, Gately S. Taking school-based substance abuse prevention to scale: District-wide implementation of Keep A Clear Mind. Journal of Alcohol and Drug Education. 2007;51(3):73–91.
Gottfredson GD, Gottfredson DC, Payne AA, et al. School climate predictors of school disorder: Results from a national study of delinquency prevention in schools. Journal of Research in Crime and Delinquency. 2005;42(4):412–444.
Glisson C, Green P. The effects of organizational culture and climate on the access to mental health care in Child welfare and Juvenile Justice systems. Administration and Policy in Mental Health and Mental Health Services Research. 2006;33(4):433–448.
Bradshaw CP, Debnam K, Koth CW, et al. Preliminary validation of the Implementation Phases Inventory for assessing fidelity of schoolwide positive behavior supports. Journal of Positive Behavior Interventions. 2009;11(3):145–160.
Sugai G, Lewis-Palmer T, Todd A, et al. School-wide Evaluation Tool (SET). Eugene, OR: Center for Positive Behavioral Supports, University of Oregon; 2001.
Kincaid D, Childs K, George H. School-wide Benchmarks of Quality (Revised). Tampa, Florida: University of South Florida; 2010.
Cohen R, Kincaid D, Childs KE. Measuring school-wide positive behavior support implementation: Development and validation of the benchmarks of quality. Journal of Positive Behavior Interventions. 2007;9(4):203–213.
Bond GR, Drake RE, McHugo GJ, et al. Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice. 2009;19(5):569–581.
Horner RH, Todd AW, Lewis-Palmer T, et al. The School-Wide Evaluation Tool (SET): A research instrument for assessing School-Wide Positive Behavior Support. Journal of Positive Behavior Interventions. 2004;6(1):3–12.
Bradshaw CP, Reinke WM, Brown LD, et al. Implementation of School-Wide Positive Behavioral Interventions and Supports (PBIS) in elementary schools: Observations from a randomized trial. Education & Treatment of Children (West Virginia University Press). 2008:1–26.
Vincent C, Spaulding S, Tobin TJ. A reexamination of the psychometric properties of the School-Wide Evaluation Tool (SET). Journal of Positive Behavior Interventions. 2010;12(3):161–179.
Muthén LK, & Muthén BO. Mplus user’s guide. Los Angeles, CA: Muthén and Muthén; 1997–2009.
Bollen KA. Structural equations with latent variables. Oxford, England: John Wiley & Sons; 1989.
Kline RB. Principles and practice of structural equation modeling. 2nd ed. New York, NY: Guilford; 2005.
Yu CY, Muthén BO. Evaluation of model fit indices for latent variable models with categorical and continuous outcomes. Los Angeles, CA; 2001.
McDonald RP, Ho MR. Principles and practice in reporting structural equation analyses. Psychological Methods. 2002;7(1):64–82.
Wilson SJ, Lipsey MW. School-Based Interventions for Aggressive and Disruptive Behavior: Update of a Meta-Analysis. American Journal of Preventive Medicine. 2007;33(2):S130–S143.
Thomas R, Zimmer-Gembeck MJ. Behavioral outcomes of Parent–Child Interaction Therapy and Triple P-Positive Parenting Program: A review and meta-analysis. Journal of Abnormal Child Psychology. 2007;35(3):475–495.
Birnbaum AS, Lytle LA, Hannan PJ, et al. School functioning and violent behavior among young adolescents: A contextual analysis. Health Education Research. 2003;18(3):389–403.
Hoagwood K, Burns BJ, Kiser L, et al. Evidence-based practice in child and adolescent mental health services. Psychiatric Services. 2001;52(9):1179–1189.
Sandler I, Ostrom A, Bitner MJ, et al. Developing effective prevention services for the real world: A prevention science development model. American Journal of Community Psychology. 2005;35:127–142.
Acknowledgements
Support for this project comes from the National Institute of Mental Health (R01 MH67948-1A1, T32 MH19545-11), the Centers for Disease Control and Prevention (1U49CE000728, K01CE001333-01), and the Institute of Education Sciences (R324A07118, R305A090307, R324A110107). The authors would like the thank the Maryland PBIS Management Team for their support of this project, with special thanks to Philip Leaf, the Maryland State Department of Education, and Sheppard Pratt Health System.
Conflicts of interest
The authors do not have any conflicts of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pas, E.T., Bradshaw, C.P. Examining the Association Between Implementation and Outcomes. J Behav Health Serv Res 39, 417–433 (2012). https://doi.org/10.1007/s11414-012-9290-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11414-012-9290-2