Abstract
We assessed the reporting of treatment integrity in school-wide prevention programs in K-12 schools. This review was designed to determine (a) the extent to which treatment integrity was reported in school-wide prevention and intervention programs and how the reporting varied by research design, year, and journal; and (b) the procedures (e.g., method, frequency, informant) used to collect treatment integrity data. Results indicated that fewer than half of the studies in the review (n = 36, 45.6 %) measured and reported treatment integrity. Those studies reporting treatment integrity often used multiple methods and informants. Reporting treatment integrity in this body of literature has increased steadily over time.
Similar content being viewed by others
References
* = articles included in the literature review.
*Araji, S. K., Fenton, R., & Straugh, T. (1995). Child sexual abuse: Description and evaluation of K-6 prevention curriculum. Journal of Primary Prevention, 16, 149–164.
*Ausems, M., Mesters, I., van Brewkelen, G., & De Vries, H. (2004). Effects of in-school and tailored out-of-school smoking prevention among Dutch vocational school students. Health Education Research, 19, 51–63.
*Barrett, S. B., Bradshaw, C. P., & Lewis-Palmer, T. (2008). Maryland Statewide PBIS Initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions, 10, 105–114.
*Beets, M. W., Flay, B. R., & Vuchinich, S. (2009). Use of a social and character development program to prevent substance use, violent behaviors, and sexual activity among elementary-school students in Hawaii. American Journal of Public Health, 99, 1438–1445.
Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23, 443–451. doi:10.1037/0278-6133.23.5.443.
Blonigen, B. A., Harbaugh, W. T., Singell, L. D., Horner, R. H., Irvin, L. K., & Smolkowski, K. S. (2008). Application of economic analysis to school-wide positive behavior support (SWPBS) programs. Journal of Positive Behavior Interventions, 10(1), 5–19.
*Bohanon, H., Fenning, P., Carney, K., Minnis, M., Anderson-Harriss, S., Moroz, K., et al. (2006). School-wide application of positive behavior support in an urban high school: A case study. Journal of Positive Behavior Interventions, 8, 131–145.
*Bond, L., Patton, G., Glover, S., Carlin, J. B., Butler, H., Thomas, L., & Bowes, G. (2004a). The gatehouse project: Can a multilevel school intervention affect emotional wellbeing and health risk behaviors? Journal of Epidemiology and Community Health, 58, 997–1003.
*Bond, L., Thomas, L., & Coffey, C. (2004b). Long-term impact of the gatehouse project on cannabis use of 16-year-olds in Australia. The Journal of School Health, 74, 23–29.
*Borawski, E. A., Trapl, E. S., Lovegreen, L. D., Colabianchi, N., & Block, T. (2005). Effectiveness of abstinence-only intervention in middle school teens. American Journal of Health Behavior, 29, 423–434.
*Bowllan, N. M. (2011). Implementation and evaluation of a comprehensive, school-wide bullying prevention program in an urban/suburban middle school. Journal of School Health, 81, 167–173.
*Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133–148.
Bruhn, A. L., Lane, K. L., & Hirsch, S. E. (2014). A review of tier 2 interventions conducted within multi-tiered models of prevention evidencing a primary behavioral plan. Journal of Emotional and Behavioral Disorders, 22, 171–189. doi:10.1177/1063426613476092.
*Calear, A. L., Christensen, J., Mackinnon, A., Griffiths, K. M., & O’Kearney, R. (2009). The youthmodd project: A cluster randomized controlled trial of an online cognitive behavioral program with adolescents. Journal of Consulting and Clinical Psychology, 77, 1021–1032.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. Washington, DC: American Educational Research Association.
*Chaparro, E. A., Smolkowski, K., Baker, S. K., Hanson, N., & Ryan-Jackson, K. (2012). A model for system-wide collaboration to support integrated social behavior and literacy evidence-based practices. Psychology in the Schools, 49, 465–482. doi:10.1002/pits.
Colvin, G., & Fernandez, E. (2000). Sustaining effective behavior support systems in an elementary school. Journal of Positive Behavior Interventions, 2(4), 251–253.
*Curtis, R., Van Horner, J., Robertson, P., & Karvonen, M. (2010). Outcomes of a schoolside positive behavioral support program. Professional School Counseling, 13, 159–164.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.
Duda, M. A., Dunlap, G., Fox, L., Lentini, R., & Clarke, S. (2004). An experimental evaluation of positive behavior support in a community preschool program. Topics in Early Childhood Special Education, 24(3), 143–155.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.
*Ervin, R. A., Schaughency, E., Goodman, S. D., McGlinchey, M. T., & Matthews, A. (2006). Merging research and practice agendas to address reading and behavior school-wide. School Psychology Review, 35, 198–223.
*Ervin, R. A., Schaughency, E., Matthews, A., Goodman, S. D., & McGlinchey, M. T. (2007). Primary and secondary prevention of behavior difficulties: Developing a data-informed problem-solving model to guide decision making at a school-wide level. Psychology in the Schools, 44, 7–18.
*Farkas, M. S., Simonsen, B., Migdole, S., Donovan, M. E., Clemens, K., & Cicches, V. (2012). Schoolwide positive behavior support in an alternative school setting: An evaluation of fidelity, outcomes, and social validity of tier 1 implementation. Journal of Emotional and Behavioral Disorders, 20, 275–288. doi:10.1177/1063426610389615.
Franzen, K., & Kamps, D. (2008). The utilization and effects of positive behavior support strategies on an urban school playground. Journal of Positive Behavior Interventions, 10(3), 150–161.
*Galloway, R., Panyan, M., Smith, C., & Wessendorf, S. (2008). Systems change with school-wide positive behavior supports: Iowa’s work in progress. Journal of Positive Behavior Interventions, 10, 129–135.
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 60, 294–309.
*Grant-Petersson, J., Dietrich, A. J., & Sox, C. H. (1999). Promoting sun protection in elementary schools and child care settings: the sunsafe project. The Journal of School Health, 69, 100–106.
Gresham, F. M., Gansle, K., & Noell, G. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257–263.
Hagermoser-Sanetti, L. M., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29–46.
Hagermoser-Sanetti, L. M., Gritter, K. L., & Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40(1), 72–84.
*Haines, J., Neumark-Sztainer, D., Perry, C. L., Hannan, P. J., & Levine, M. P. (2006). V.I. K. (Very important kids): A school-based program designed to reduce teaching and unhealthy weight-control behaviors. Health Education Research, 21, 884–895.
*Hampel, P., Meier, M., & Kummel, U. (2008). School-based stress management training for adolescents: Longitudinal results from an experimental study. Journal of Youth and Adolescence, 37, 1009–1024.
*Harris, E., McFarland, J., Siebold, W., Aguilar, R., & Sarmiento, A. (2007). Universal prevention program outcomes: Safe schools health students in a rural, multicultural setting. Journal of School Violence, 6, 75–91.
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.
*Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144.
Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. L., Sugai, G., & Boland, J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.
*Jagers, R. J., Morgan-Lopez, A. A., & Flay, B. R. (2009). The impact of age and type of intervention on youth violent behaviors. Journal of Primary Prevention, 30, 642–658.
*Johnson-Gros, K. N., Lyons, E. A., & Griffin, J. R. (2008). Active supervision: An intervention to reduce high school tardiness. Education and Treatment of Children, 31, 39–53.
*Kartub, D. T., Taylor-Greene, S., March, R. E., & Horner, R. H. (2000). Reducing hallway noise: A systems approach. Journal of Positive Behavior Interventions, 2, 170–182.
*Katz, D. L., Katz, C. S., True, J. A., Reynolds, J., Njike, V., Walker, J., & Michael, J. (2011). Teaching healthful food choices to elementary school students and their parents: The nutrition detectives program. Journal of School Health, 81, 21–28.
Keller-Margulis, M. A. (2012). Fidelity of implementation framework: A critical need for response to intervention models. Psychology in the Schools, 49, 342–352.
Lane, K. L., Bocian, K. M., MacMillan, D. L., & Gresham, F. M. (2004). Treatment integrity: An essential—but often forgotten—component of school-based interventions. Preventing School Failure: Alternative Education for Children and Youth, 48(3), 36–43.
*Lane, K. L., Kalberg, J. R., Bruhn, A. L., Mahoney, M. E., & Driscoll, S. A. (2008). Primary prevention programs at the elementary level: Issues of treatment integrity, systematic screening, and reinforcement. Education and Treatment of Children, 31, 465–494.
Lane, K. L., Kalberg, J. R., Lambert, E. W., Crnobori, M., & Bruhn, A. L. (2010). A comparison of systematic screening tools for emotional and behavioral disorders: A replication. Journal of Emotional and Behavioral Disorders, 18, 100–112.
*Lane, K. L., & Menzies, M. H. (2002). The effects of a school-based primary intervention program: Preliminary outcomes. Preventing School Failure, 47, 26–32.
*Lane, K. L., & Menzies, M. H. (2003). A school-wide intervention with primary and secondary levels of support for elementary students: outcomes and considerations. Education and Treatment of Children, 26, 431–451.
*Lane, K. L., Wehby, J. H., Robertson, E. J., & Rogers, L. A. (2007). How do different types of high school students respond to schoolwide positive behavior support programs? Characteristics and responsiveness of teacher-identified students. Journal of Emotional and Behavioral Disorders, 15, 3–20.
*Lassen, S., Steele, M., & Sailor, W. (2006). The relationship of school-wide positive behavior support to academic achievement in an urban middle school. Psychology in Schools, 43(6), 701–712.
*Leadbeater, B., & Hoglund, W. (2006). Changing the social contexts of peer victimization. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 15, 21–26.
*Lewis, T. J., Colvin, G., & Sugai, G. (2000). The effects of precorrection and active supervision on the recess behavior of elementary school students. Education and Treatment of Children, 23, 109–121.
*Lewis, T. J., Powers, L. J., Kelk, M. J., & Newcomer, L. (2002). Reducing problem behaviors on the playground: An investigation of the application of school-wide positive behavior supports. Psychology in the Schools, 39, 181–190.
*Linares, L. O., Rosbruch, N., Stern, M. B., Edwards, M. E., Walker, G., Abikoff, H. B., & Alvir, J. M. J. (2005). Developing cognitive-social-emotional competencies to enhance academic learning. Psychology in the Schools, 42, 405–417.
*Luiselli, J. K., Putnam, R. F., Handler, M. W., & Feinberg, A. B. (2005). Whole-school positive behaviour support: Effects on student discipline problems and academic performance. Educational Psychology, 25, 183–198.
*Luiselli, J., Putnam, R., & Sunderland, M. (2002). Longitudinal evaluation of behavior support interventions in public middle school. Journal of Positive Behavior Interventions, 4, 182–188.
*Mayer, G. R., Butterworth, T., Nafpaktitis, M., & Suzer-Azaroff, B. (1983). Preventing school vandalism and improving discipline: A three year study. Journal of Applied Behavior Analysis, 16, 355–369.
*McCurdy, B. L., Mannella, M. C., & Eldridge, N. (2003). Positive behavior support in urban schools: Can we prevent the escalation of antisocial behavior? Journal of Positive Behavior Interventions, 5, 158–170.
*McIntosh, K., Chard, D. J., Boland, J. B., & Horner, R. H. (2006a). Demonstration of combined efforts in school-wide academic and behavioral systems and incidence of reading and behavior challenges in early elementary grades. Journal of Positive Behavior Interventions, 8, 146–154.
*McIntosh, K., Horner, R. H., Chard, D., Boland, J., & Good, R. (2006b). The use of reading and behavior screening measures to predict non-response to school-wide positive behavior support: A longitudinal analysis. School Psychology Review, 35, 275–291.
McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the journal of applied behavior analysis. Journal of Applied Behavior Analysis, 30, 659–672.
*McVey, G., Tweed, S., & Blackmore, E. (2007). Healthy schools-healthy kids: A controlled evaluation of a comprehensive universal eating disorder prevention program. Body Image, 4, 115–136.
*Metzler, C. W., Biglan, A., Rusby, J. C., & Sprague, J. R. (2001). Evaluation of a comprehensive behavior management program to improve school-wide positive behavior support. Education and Treatment of Children, 24, 448–479.
*Mikami, A. Y., Boucher, M. A., & Humphreys, K. (2005). Prevention of peer rejection through a classroom-level intervention in middle school. The Journal of Primary Prevention, 26, 5–23.
*Milne, E., Jacoby, P., Giles-Corti, B., Cross, D., Johnston, R., & English, D. R. (2006). The impact of the kidskin sun protection intervention on summer suntan and reported sun exposure: Was it sustained? Preventive Medicine, 42, 14–20.
Moncher, F. J., & Prinz, F. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247–266.
*Muscott, H. S., Mann, E. L., & LeBrun, M. R. (2008). Positive behavioral interventions and supports in New Hampshire: Effects of large-scale implementation of schoolwide positive behavior support on student discipline and academic achievement. Journal of Positive Behavior Interventions, 10, 190–205.
*Nandha, B., & Krishnamoorthy, K. (2007). School-based health education campaign—a potential tool for social mobilization to promote the use of DEC-fortified salt towards elimination of lymphatic filariasis. Health Education Research, 22, 539–546.
*Nelson, J. R. (1996). Designing schools to meet the needs of students who exhibit disruptive behavior. Journal of Emotional and Behavioral Disorders, 4, 147–161.
*Nelson, J. R., Martella, R. M., & Marchand-Martella, N. (2002). Maximizing student learning: The effects of a comprehensive school-based program for preventing problem behaviors. Journal of Emotional and Behavioral Disorders, 10, 136–148.
*Nelson, J. R., Colvin, G., & Smith, D. J. (1996). The effects of setting clear standards on students’ social behavior in common areas of the school. Journal of Emotional and Behavioral Disorder, 4(3), 10–17.
*Nixon, C. L., & Wener, N. E. (2010). Reducing adolescents’ involvement with relational aggression: Evaluating the effectiveness of the creating a safe school (CASS) intervention. Psychology in the Schools, 47, 606–620.
*Orpinas, P., Kelder, S., Frankowski, R., Murray, N., Zhang, Q., & McAlister, A. (2000). Outcome evaluation of a multi-component violence-prevention program for middle schools: the students for peace project. Health Education Research, 15, 45–58.
*Oswald, K., Safran, S., & Johanson, G. (2005). Preventing trouble: Making schools safer places using positive behavior supports. Education and Treatment of Children, 28, 265–278.
*Putnam, R. F., Handler, M. W., Ramirez-Platt, C. M., & Luiselli, J. K. (2003). Improving student bus-riding behavior through a whole-school intervention. Journal of Applied Behavior Analysis, 36, 583–589.
*Putnam, R. F., Luiselli, J. K., & Sunderland, M. (2002). Longitudinal evaluation of behavior support intervention in a public middle school. Journal of Positive Behavior Interventions, 4, 182–188.
*Reinert, B., Carver, V., & Range, L. M. (2004). Tobacco use prevention in private high school. International Quarterly of Community Health Education, 22(1), 47–58.
Roach, A. T., & Elliott, S. N. (2008). Best practices in facilitating and evaluating intervention integrity. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (5th ed., pp. 195–208). Bethesda, MD: National Association of School Psychologists.
*Ross, S. W., & Horner, R. H. (2009). Bully prevention in positive behavior support. Journal of Applied Behavior Analysis, 42, 747–759.
*Sanchez, E., Robertson, T. R., Lewis, C. M., Rosenbluth, B., Bohman, T., & Casey, D. M. (2001). Preventing bullying and sexual harassment in elementary schools: The expect respect model. Journal of Emotional Abuse, 2, 157–180.
Schober, I., Sharpe, H., & Schmidt, U. (2013). The reporting of fidelity measures in primary prevention programmes for eating disorders in schools. European Eating Disorders Review, 21, 374–381. doi:10.1002/erv.2243.
*Schofield, M. J., Lynagh, M., & Mishra, G. (2003). Evaluation of a health promoting schools program to reduce smoking australian secondary schools. Health Education Research, 18, 678–692.
*Schonfeld, D. J., Bases, J., & Quackenbush, M. (2001). Pilot-testing a cancer education curriculum for grades K-6. The Journal of School Health, 71, 61–65.
Schulte, A. C., Easton, J. E., & Parker, J. (2009). Advances in treatment integrity research: Interdisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 39, 460–475.
*Scott, T. M. (2001). A schoolwide example of positive behavioral support. Journal of Positive Behavior Interventions, 3, 88–94.
*Scott, T. M., & Barrett, S. B. (2004). Using staff and student time engaged in disciplinary procedures to evaluate the impact of school-wide PBS. Journal of Positive Behavior Interventions, 6, 21–27.
*Sherrod, M. D., Getch, Y., & Ziomek-Daigle, J. (2009). The impact of positive behavior support to decrease discipline referrals with elementary students. Professional School Counseling, 12, 421–427.
*Simonsen, B., Britton, L., & Young, D. (2010). School-wide positive behavior support in an alternative school setting: A case study. Journal of Positive Behavior Interventions, 12, 180–191.
*Simons-Morton, B., Haynie, D., Saylor, K., Crump, A. D., & Chen, R. (2005). The effects of the going places program on early adolescent substance use and antisocial behavior. Prevention Science, 6, 187–197.
*Slater, M. D., Kelly, K. J., & Edwards, R. W. (2006). Combining in-school and community-based media efforts: reducing marijuana and alcohol uptake among younger adolescents. Health Education Research, 2, 157–167.
*Smith, J. S., Anderson, J. A., & Abell, A. K. (2008). Preliminary evaluation of the full-purpose partnership schoolwide model. Preventing School Failure, 53, 28–37.
*Snyder, F. J., Vuchinich, S., Acock, A., Washburn, I. J., & Flay, B. R. (2012). Improving elementary school quality through the use of a social-emotional and character development program: A matched-pair, cluster-randomized controlled trial in Hawaii. Journal of School Health, 82, 11–20.
Spence, S. H., Sheffield, J. K., & Donovan, C. L. (2005). Long-term outcome of a school-based, universal approach to prevention of depression in adolescents. Journal of Consulting and Clinical Psychology, 73(1), 160.
*Sprague, J., Walker, H., Golly, A., White, K., Myers, D. R., & Shannon, T. (2001). Translating research into effective practice: The effects of a universal staff and student intervention on indicators of discipline and school safety. Education & Treatment of Children, 24, 495–511.
Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner, R. H. (2001). School-wide evaluation tool (SET). Eugene, OR: Center for Positive Behavioral Supports, University of Oregon.
*Swaim, R. C., & Kelly, K. (2008). Efficacy of a randomized trial of a community and school-based anti-violence media intervention among small-town middle school youth. Prevention Science, 9, 202–214.
*Taub, J. (2002). Evaluation of the second step violence prevention program at a rural elementary school. School Psychology Review, 31, 186–200.
*Taylor-Greene, S., Brown, D., Nelson, L., Longton, J., Gassman, T., & Cohen, J. (1997). School-wide behavioral support: Starting the year off right. Journal of Behavioral Education, 7, 99–112.
*Todd, A., Haugen, L., Anderson, K., & Spriggs, M. (2002). Teaching recess: Low-cost efforts producing effective results. Journal of Positive Behavior Interventions, 4, 46–52.
*Vazsonyi, A. T., Belliston, L. M., & Flannery, D. (2004). Evaluation of a school-based, universal violence prevention program: low-, medium-, and high-risk children. Youth Violence and Juvenile Justice, 2, 185–206.
*Vigna-Taglianti, F., Vadrucci, S., Faggiano, F., Burkhart, G., Siliquini, R., & Galanti, M. R. (2009). Is universal prevention against youths’ substance misuse really universal? Gender-specific effects in the EU-Dap school-based prevention trial. Journal of Epidemiology and Community Health, 63, 722–728.
*Warren, J. R. (2006). Communicating prevention: The effects of the keepin’ it real classroom videotapes and televised PSAs on middle-school students’ substance us. Journal of Applied Communication Research, 34, 209–227.
*Waschubusch, D. A., Pelham, W. E., & Massetti, G. (2005). The behavior education support and treatment (BEST) school intervention program: pilot project data examining schoolwide, targeted-school, and targeted-home approaches. Journal of Attention Disorders, 9, 313–322.
*Wheatley, R. K., West, R. P., Charlton, C. T., Sanders, R. B., Smith, T. G., & Taylor, M. J. (2009). Improving behavior through differential reinforcement: A praise note system for elementary school students. Education and Treatment of Children, 32, 551–571.
Wickstrom, K. F., Jones, K. M., LaFleur, L. H., & Witt, J. C. (1998). An analysis of treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 13(2), 141–154.
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167.
*Zenere, F. J., & Lazarus, P. J. (2009). The sustained reduction of youth suicidal behavior in an urban, multicultural school district. School Psychology Review, 38, 189–199.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest.
Rights and permissions
About this article
Cite this article
Bruhn, A.L., Hirsch, S.E. & Lloyd, J.W. Treatment Integrity in School-Wide Programs: A Review of the Literature (1993–2012). J Primary Prevent 36, 335–349 (2015). https://doi.org/10.1007/s10935-015-0400-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10935-015-0400-9