Skip to main content

Advertisement

SpringerLink
Log in

Examining the Association Between Implementation and Outcomes

State-wide Scale-up of School-wide Positive Behavior Intervention and Supports

  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

Although there is an established literature supporting the efficacy of a variety of prevention programs, there has been less empirical work on the translation of such research to everyday practice or when scaled-up state-wide. There is a considerable need for more research on factors that enhance implementation of programs and optimize outcomes, particularly in school settings. The current paper examines how the implementation fidelity of an increasingly popular and widely disseminated prevention model called, School-wide Positive Behavioral Interventions and Supports (SW-PBIS), relates to student outcomes within the context of a state-wide scale-up effort. Data come from a scale-up effort of SW-PBIS in Maryland; the sample included 421 elementary and middle schools trained in SW-PBIS. SW-PBIS fidelity, as measured by one of three fidelity measures, was found to be associated with higher math achievement, higher reading achievement, and lower truancy. School contextual factors were related to implementation levels and outcomes. Implications for scale-up efforts of behavioral and mental health interventions and measurement considerations are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

Similar content being viewed by others

Notes

  1. This scale measures pre-implementation readiness and therefore the variability and the internal consistency on this scale are low.

References

  1. Flay BR, Biglan A, Boruch RF, et al. Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science. 2005;6(3):151–175.

    Article  PubMed  Google Scholar 

  2. Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research. 2003;18(2):237–256.

    Article  PubMed  Google Scholar 

  3. Ringwalt CL, Ennett S, Johnson R, et al. Factors associated with fidelity to substance use prevention curriculum guides in the nation's middle schools. Health Education & Behavior. 2003;30(3):375–391.

    Article  Google Scholar 

  4. SPR MAPS II Task Force. Type 2 translational research: Overview and definitions. 2008; http://preventionscience.org/SPR_Type%202%20Translation%20Research_Overview%20and%20Definition.pdf.

  5. Woolf SH. The meaning of translational research and why it matters. Journal of the American Medical Association. 2008;299:211–213.

    Article  PubMed  CAS  Google Scholar 

  6. Woolf SH. Potential health and economic consequences of misplaced priorities. Journal of the American Medical Association. 2007;297:523–526.

    Article  PubMed  CAS  Google Scholar 

  7. Spoth R. Translating family-focused prevention science into effective practice: Toward a translational impact paradigm. Current Directions in Psychological Science. 2008;17(6):415–421.

    Article  PubMed  Google Scholar 

  8. Ringwalt C, Vincus AA, Hanley S, et al. The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2008. Prevention Science. 2011;12(1):63–69.

    Article  PubMed  Google Scholar 

  9. Sloboda Z, Pyakuryal A, Stephens PC, et al. Reports of substance abuse prevention programming available in schools. Prevention Science. 2008;9(4):276–287.

    Article  PubMed  Google Scholar 

  10. Gottfredson GD, Gottfredson DC, Czeh ER, et al. National study of delinquency prevention in schools. Ellicott City, MD: Gottfredson Associates; 2000.

    Book  Google Scholar 

  11. Gottfredson GD, Jones EM, Gore TW. Implementation and evaluation of a cognitive–behavioral intervention to prevent problem behavior in a disorganized school. Prevention Science. 2002;3(1):43–56.

    Article  PubMed  Google Scholar 

  12. Domitrovich CE, Greenberg MT. The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational & Psychological Consultation. 2000;11(2):193–221.

    Article  Google Scholar 

  13. Durlak JA. Successful prevention programs for children and adolescents. New York: Plenum; 1997.

    Google Scholar 

  14. Sugai G, Horner RH, Gresham FM. Behaviorally effective school environments. In: Shinn MR, Walker HM, Stoner G, eds. Interventions for academic and behavior problems: II. Preventive and remedial approaches. Bethesda, MD US: National Association of School Psychologists; 2002:315–350.

    Google Scholar 

  15. Domitrovich CE, Bradshaw CP, Poduska JM, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion. 2008;1(3):6–28.

    Article  Google Scholar 

  16. Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services. 2001;52(9):1190–1197.

    Article  PubMed  CAS  Google Scholar 

  17. Stuart E, Cole S, Bradshaw CP, et al. The use of propensity scores to assess the generalizability of results from randomized trials. The Journal of the Royal Statistical Society, Series A. 2011; 174(2):369–386.

    Article  Google Scholar 

  18. O’Connell ME, Boat T, Warner KE. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press. Committee on the Prevention of Mental Disorders and Substance Abuse Among Children, Youth, and Young Adults: Research Advances and Promising Interventions; 2009.

    Google Scholar 

  19. Bradshaw CP, Koth CW, Bevans KB, et al. The impact of school-wide Positive Behavioral Interventions and Supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly. 2008;23(4):462–473.

    Article  Google Scholar 

  20. Bradshaw CP, Koth CW, Thornton LA, et al. Altering school climate through school-wide Positive Behavioral Interventions and Supports: Findings from a group-randomized effectiveness trial. Prevention Science. 2009;10(2):100–115.

    Article  PubMed  Google Scholar 

  21. Bradshaw CP, Mitchell MM, Leaf PJ. Examining the effects of schoolwide Positive Behavioral Interventions and Supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions. 2010;12(3):133–148.

    Article  Google Scholar 

  22. Horner RH, Sugai G, Smolkowski K, et al. A randomized, wait-list controlled effectiveness trial assessing School-Wide Positive Behavior Support in elementary schools. Journal of Positive Behavior Interventions. 2009;11(3):133–144.

    Article  Google Scholar 

  23. Bradshaw CP, Waasdorp TE, Leaf PJ. Effects of School-Wide Positive Behavioral Interventions and Supports on child behavior problems and adjustment. Manuscript submitted for publication. 2012.

  24. Waasdorp TE, Bradshaw CP, Leaf PJ. The impact of School-wide Positive Behavioral Interventions and Supports (SWPBIS) on bullying and peer rejection: A randomized controlled effectiveness trial. Archives of Child and Adolescent Medicine. 2012;116(2):149–156.

    Article  Google Scholar 

  25. Barrett SB, Bradshaw CP, Lewis-Palmer T. Maryland statewide PBIS initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions. 2008;10(2):105–114.

    Article  Google Scholar 

  26. Muscott HS, Mann EL, LeBrun MR. Positive behavioral interventions and supports in New Hampshire. Effects of large-scale implementation of schoolwide positive behavior support on student discipline and academic achievement. Journal of Positive Behavior Interventions. 2008;10(3):190–205.

    Article  Google Scholar 

  27. Bradshaw CP, Pas ET. A state-wide scale-up of Positive Behavioral Interventions and Supports (PBIS): A description of the development of systems of support and analysis of adoption and implementation. School Psychology Review. 2011;40(4):530–548.

    Google Scholar 

  28. Adelman HS, Taylor L. Toward a scale-up model for replicating new approaches to schooling. Journal of Educational & Psychological Consultation. 1997;8(2):197.

    Article  Google Scholar 

  29. Kreuter MW, Bernhardt JM. Reframing the dissemination challenge: a marketing and distribution perspective. American Journal of Public Health. 2009;99(12):2123–2127.

    Article  PubMed  Google Scholar 

  30. Rohrbach LA, Grana R, Sussman S, et al. Type II Translation: Transporting prevention interventions from research to real-world settings. Evaluation & the Health Professions. 2006;29(3):302–333.

    Article  Google Scholar 

  31. Bloomquist ML, August GJ, Horowitz JL, et al. Moving from science to service: Transposing and sustaining the early risers prevention program in a community service system. The Journal of Primary Prevention. 2008;29(4):307–321.

    Article  PubMed  Google Scholar 

  32. Fixsen DL, Naoom SF, Blase KA, et al. Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.

    Google Scholar 

  33. Spoth RL, Greenberg, MT. Toward a comprehensive strategy for effective practitioner–scientist partnerships and larger-scale community health and well-being. American Journal of Community Psychology. 2005;35:107–126.

    Article  PubMed  Google Scholar 

  34. Rogers EM. Diffusion of preventive innovations. Addictive Behaviors. 2002;27(6):989–993.

    Article  PubMed  Google Scholar 

  35. Botvin GJ, Baker E, Dusenbury L, et al. Long-term follow-up results of a randomized drug abuse prevention trial in a White middle-class population. Journal of the American Medical Association.1995;273(14):1106–1112.

    Article  PubMed  CAS  Google Scholar 

  36. Derzon JH, Sale E, Springer JF, et al. Estimating intervention effectiveness: Synthetic projection of field evaluation results. The Journal of Primary Prevention. 2005;26(4):321–343.

    Article  PubMed  Google Scholar 

  37. Durlak JA, DuPre EP. Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology. 2008;41(3–4):327–350.

    Article  PubMed  Google Scholar 

  38. Hulleman CS, Cordray, DS. Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness. 2009;2(1):88–110.

    Article  Google Scholar 

  39. Ialongo NS, Werthamer L, Kellam SG, et al. Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology. 1999;27(5):599–641.

    Article  PubMed  CAS  Google Scholar 

  40. Sanders MR. Triple P-Positive Parenting Program: Towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clinical Child & Family Psychology Review. 1999;2(2):71–90.

    Article  CAS  Google Scholar 

  41. Nowak C, Heinrichs N. A comprehensive meta-analysis of Triple P-Positive Parenting Program using hierarchical linear modeling: Effectiveness and moderating variables. Clinical Child and Family Psychology Review. 2008;11(3):114–144.

    Article  PubMed  Google Scholar 

  42. Kam C-M, Greenberg MT, Walls CT. Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science. 2003;4(1):55–63.

    Article  PubMed  Google Scholar 

  43. Jowers KL, Bradshaw CP, Gately S. Taking school-based substance abuse prevention to scale: District-wide implementation of Keep A Clear Mind. Journal of Alcohol and Drug Education. 2007;51(3):73–91.

    Google Scholar 

  44. Gottfredson GD, Gottfredson DC, Payne AA, et al. School climate predictors of school disorder: Results from a national study of delinquency prevention in schools. Journal of Research in Crime and Delinquency. 2005;42(4):412–444.

    Article  Google Scholar 

  45. Glisson C, Green P. The effects of organizational culture and climate on the access to mental health care in Child welfare and Juvenile Justice systems. Administration and Policy in Mental Health and Mental Health Services Research. 2006;33(4):433–448.

    Article  PubMed  Google Scholar 

  46. Bradshaw CP, Debnam K, Koth CW, et al. Preliminary validation of the Implementation Phases Inventory for assessing fidelity of schoolwide positive behavior supports. Journal of Positive Behavior Interventions. 2009;11(3):145–160.

    Article  Google Scholar 

  47. Sugai G, Lewis-Palmer T, Todd A, et al. School-wide Evaluation Tool (SET). Eugene, OR: Center for Positive Behavioral Supports, University of Oregon; 2001.

  48. Kincaid D, Childs K, George H. School-wide Benchmarks of Quality (Revised). Tampa, Florida: University of South Florida; 2010.

    Google Scholar 

  49. Cohen R, Kincaid D, Childs KE. Measuring school-wide positive behavior support implementation: Development and validation of the benchmarks of quality. Journal of Positive Behavior Interventions. 2007;9(4):203–213.

    Article  Google Scholar 

  50. Bond GR, Drake RE, McHugo GJ, et al. Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice. 2009;19(5):569–581.

    Article  Google Scholar 

  51. Horner RH, Todd AW, Lewis-Palmer T, et al. The School-Wide Evaluation Tool (SET): A research instrument for assessing School-Wide Positive Behavior Support. Journal of Positive Behavior Interventions. 2004;6(1):3–12.

    Article  Google Scholar 

  52. Bradshaw CP, Reinke WM, Brown LD, et al. Implementation of School-Wide Positive Behavioral Interventions and Supports (PBIS) in elementary schools: Observations from a randomized trial. Education & Treatment of Children (West Virginia University Press). 2008:1–26.

  53. Vincent C, Spaulding S, Tobin TJ. A reexamination of the psychometric properties of the School-Wide Evaluation Tool (SET). Journal of Positive Behavior Interventions. 2010;12(3):161–179.

    Article  Google Scholar 

  54. Muthén LK, & Muthén BO. Mplus user’s guide. Los Angeles, CA: Muthén and Muthén; 1997–2009.

  55. Bollen KA. Structural equations with latent variables. Oxford, England: John Wiley & Sons; 1989.

    Google Scholar 

  56. Kline RB. Principles and practice of structural equation modeling. 2nd ed. New York, NY: Guilford; 2005.

    Google Scholar 

  57. Yu CY, Muthén BO. Evaluation of model fit indices for latent variable models with categorical and continuous outcomes. Los Angeles, CA; 2001.

  58. McDonald RP, Ho MR. Principles and practice in reporting structural equation analyses. Psychological Methods. 2002;7(1):64–82.

    Article  PubMed  Google Scholar 

  59. Wilson SJ, Lipsey MW. School-Based Interventions for Aggressive and Disruptive Behavior: Update of a Meta-Analysis. American Journal of Preventive Medicine. 2007;33(2):S130–S143.

    Article  PubMed  Google Scholar 

  60. Thomas R, Zimmer-Gembeck MJ. Behavioral outcomes of Parent–Child Interaction Therapy and Triple P-Positive Parenting Program: A review and meta-analysis. Journal of Abnormal Child Psychology. 2007;35(3):475–495.

    Article  PubMed  Google Scholar 

  61. Birnbaum AS, Lytle LA, Hannan PJ, et al. School functioning and violent behavior among young adolescents: A contextual analysis. Health Education Research. 2003;18(3):389–403.

    Article  PubMed  Google Scholar 

  62. Hoagwood K, Burns BJ, Kiser L, et al. Evidence-based practice in child and adolescent mental health services. Psychiatric Services. 2001;52(9):1179–1189.

    Article  PubMed  CAS  Google Scholar 

  63. Sandler I, Ostrom A, Bitner MJ, et al. Developing effective prevention services for the real world: A prevention science development model. American Journal of Community Psychology. 2005;35:127–142.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Support for this project comes from the National Institute of Mental Health (R01 MH67948-1A1, T32 MH19545-11), the Centers for Disease Control and Prevention (1U49CE000728, K01CE001333-01), and the Institute of Education Sciences (R324A07118, R305A090307, R324A110107). The authors would like the thank the Maryland PBIS Management Team for their support of this project, with special thanks to Philip Leaf, the Maryland State Department of Education, and Sheppard Pratt Health System.

Conflicts of interest

The authors do not have any conflicts of interest.

Author information

Authors and Affiliations

  1. Johns Hopkins Bloomberg School of Public Health, Johns Hopkins Center for the Prevention of Youth Violence, 624 N. Broadway, Baltimore, MD, 21205, USA

    Elise T. Pas PhD & Catherine P. Bradshaw PhD

Authors
  1. Elise T. Pas PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Catherine P. Bradshaw PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elise T. Pas PhD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pas, E.T., Bradshaw, C.P. Examining the Association Between Implementation and Outcomes. J Behav Health Serv Res 39, 417–433 (2012). https://doi.org/10.1007/s11414-012-9290-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-012-9290-2

Keywords

Search

Navigation