Skip to main content
SpringerLink
Log in

Using Complier Average Causal Effect Estimation to Determine the Impacts of the Good Behavior Game Preventive Intervention on Teacher Implementers

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Complier average causal effect (CACE) analysis is a causal inference approach that accounts for levels of teacher implementation compliance. In the current study, CACE was used to examine one-year impacts of PAX good behavior game (PAX GBG) and promoting alternative thinking strategies (PATHS) on teacher efficacy and burnout. Teachers in 27 elementary schools were randomized to PAX GBG, an integration of PAX GBG and PATHS, or a control condition. There were positive overall effects on teachers’ efficacy beliefs, but high implementing teachers also reported increases in burnout across the school year. The CACE approach may offer new information not captured using a traditional intent-to-treat approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abenavoli, R. M., Jennings, P. A., Greenberg, M. T., Harris, A. R., & Katz, D. A. (2013). The protective effects of mindfulness against burnout among educators. The Psychology of Education Review, 37, 57–69.

    Google Scholar 

  • Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444–455.

    Article  Google Scholar 

  • Barnard, J., Frangakis, C. E., Hill, J. L., & Rubin, D. (2003). Principal stratification approach to broken randomized experiments. Journal of American Statistical Association, 98, 299–323. doi:10.1198/016214503000071.

    Article  Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA, 273, 1106–1112.

    Article  CAS  PubMed  Google Scholar 

  • Bradshaw, C. P., Koth, C. W., Bevans, K. B., Ialongo, N., & Leaf, P. J. (2008). The impact of school-wide Positive Behavioral Interventions and Supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly, 23, 462–473. doi:10.1037/a0012883.

    Article  Google Scholar 

  • Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009a). Altering school climate through school-wide Positive Behavioral Interventions and Supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10(2), 100–115. doi:10.1007/s11121-008-0114-9.

    Article  PubMed  Google Scholar 

  • Bradshaw, C. P., Zmuda, J. H., Kellam, S. G., & Ialongo, N. S. (2009b). Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school. Journal of Educational Psychology, 101(4), 926–937. doi:10.1037/a0016586.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bryk, A. S., & Schnieder, B. (2002). Trust in schools: a core resource for improvement. New York: Russell Sage.

    Google Scholar 

  • Chen, H. T. (1998). Theory-driven evaluations. Advances in Educational. Productivity, 7, 15–34.

    Google Scholar 

  • Conduct Problems Prevention Research Group. (1999). Initial impact of the fast track prevention trial for conduct problems: II. Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.

    Article  PubMed Central  Google Scholar 

  • Connell, A. M., Dishion, M. Y., & Kavanagh, K. (2007). An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent behavior. Journal of Consulting and Clinical Psychology, 75, 568–579. doi:10.1037/0022-006X.75.4.568.

    Article  PubMed  Google Scholar 

  • Cowen, J. M. (2008). School choice as a latent variable: Estimating the “complier average causal effect” of vouchers in Charlotte. The Policy Studies Journal, 36, 301–315.

    Article  Google Scholar 

  • Derzon, J. H., Sale, E., Springer, J. F., & Brounstein, P. (2005). Estimating intervention effectiveness: Synthetic projection of field evaluation results. Journal of Primary Prevention, 26, 321–343.

    Article  PubMed  Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Berg, J., Pas, E. T., Becker, K., Musci, R, Ialongo, N. (2016). How do school-based prevention programs impact teachers? Findings from a randomized trial of combined classroom management and social-emotional programs. Journal of Prevention Science, 1–13.

  • Domitrovich, C., Bradshaw, C., Greenberg, M., Embry, D., Poduska, J., & Ialongo, N. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47, 71–88.

    PubMed  PubMed Central  Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K. E., Buckley, J. A., Olin, S., & Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28.

    Article  PubMed  PubMed Central  Google Scholar 

  • Domitrovich, C. E., Gest, S. D., Gill, S., Jones, D. J., & DeRouise, R. S. (2009). Teacher factors related to the professional development process of the Head Start REDI intervention. Early Education and Development, 20, 402–430.

    Article  Google Scholar 

  • Domitrovich, C., Pas, E., Bradshaw, C. P., Becker, K., Keperling, J. P., Embry, D., & Ialongo, N. (2015). Individual and school organizational factors that influence implementation of the Pax Good Behavior Game intervention. Prevention Science, 6(8), 1064–1074.

    Article  Google Scholar 

  • Domitrovich, C. & Poduska, J. (2008). The social-emotional learning efficacy scale. Unpublished technical report.

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

    Article  PubMed  Google Scholar 

  • Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432.

    Article  PubMed  Google Scholar 

  • Embry, D., Staatemeier, G., Richardson, C., Lauger, K., & Mitich, J. (2003). The PAX Good Behavior Game (1st ed.). Center City : Hazelden.

    Google Scholar 

  • Fixsen, D., Naoom, S., Blasé, K., Friedman, R., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute. The National Implementation Research Network (FMHI Publication #231).

  • Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58, 21–29.

    Article  PubMed  PubMed Central  Google Scholar 

  • Frank, J. L., Jennings, P. A., & Greenberg, M. T. (2016). Validation of the mindfulness in teaching scale. Mindfulness, 1, 155–163.

    Article  Google Scholar 

  • Greenberg, M., & Kusché, C. (2006). Building social and emotional competence: The PATHS Curriculum. In S. R. Jimerson & M. J. Furlong (Eds.), Handbook of school violence and school safety: From research to practice (pp. 395–412). Mahwah: Erlbaum.

    Google Scholar 

  • Greenberg, M. T., Kusché, C. A., & Conduct Problems Prevention Research Group. (2011). Grade level PATHS (Grades 3-5). South Deerfield: Channing-Bete Co.

    Google Scholar 

  • Greenberg, M., Kusché, C., Cook, E., & Quamma, J. (1995). Promoting emotional competence in school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117–136.

    Article  Google Scholar 

  • Han, S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679.

    Article  PubMed  Google Scholar 

  • Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–970.

    Article  Google Scholar 

  • Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.

    Article  Google Scholar 

  • Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional and Behavioral Disorders, 9, 146–160.

    Article  Google Scholar 

  • Ialongo, N., Wethamer, L., Kellam, S., Brown, C., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and anti-social behavior. American Journal of Community Psychology, 27, 599–641.

    Article  CAS  PubMed  Google Scholar 

  • Jo, B. (2002). Estimation of intervention effects with noncompliance: Alternative model specifications. Journal of Educational and Behavioral Statistics, 27, 385–409.

    Article  Google Scholar 

  • Jo, B., Asparouhov, T., Muthén, B. O., Ialongo, N. S., & Brown, C. H. (2008). Cluster randomized trials with treatment noncompliance. Psychological Methods, 13, 1–18.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jo, B., & Muthén, B. (2003). Longitudinal studies with intervention and noncompliance: Estimation of causal effects in growth mixture modeling. In N. Duan & S. Reise (Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 112–139). Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Jo, B., Wang, C. P., & Ialongo, N. S. (2009). Using latent outcome trajectory classes in causal inference. Statistics Interface, 2, 403–412.

    Article  Google Scholar 

  • Kellam, S., Brown, C. H., Poduska, J., Ialongo, N., Wang, W., Toyinbo, P., & Wilcox, H. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95S, S5–S28.

    Article  Google Scholar 

  • Kusché, C. A., Greenberg, M. T., &, & Conduct Problems Prevention Research Group. (2011). Grade level PATHS (Grades 1-2). South Deerfield: Channing-Bete Co.

    Google Scholar 

  • Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. Hoboken: Wiley.

    Book  Google Scholar 

  • Little, R. J. A., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin’s causal model. Psychological Methods, 3, 147–159.

    Article  Google Scholar 

  • Main, S., & Hammond, L. (2008). Best practice or most practiced? Pre-service teachers’ beliefs about effective behavior management strategies and reported self-efficacy. Australian Journal of Teacher Education, 33, 28–39.

    Article  Google Scholar 

  • Maslach, C., Jackson, S. E., & Leiter, M. P. (1997). Maslach burnout inventory. In R. J. Wood (Ed.), Evaluating stress: A book of resources (3rd ed., pp. 191–218). Lanham: Scarecrow Education.

    Google Scholar 

  • O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33–84.

    Article  Google Scholar 

  • Pas, E., & Bradshaw, C. P. (2014). What affects teacher ratings of student behaviors? The potential influence of teachers’ perceptions of the school environment and experiences. Prevention Science, 15, 940–950. doi:10.1007/s11121-013-0432-4.

    Article  PubMed  Google Scholar 

  • Payne, A. A., & Eckert, R. (2010). The relative importance of provider, program, school, and community predictors of the implementation quality of school-based prevention programs. Prevention Science, 11(2), 126–141.

    Article  PubMed  Google Scholar 

  • Ringeisen, H., Henderson, K., & Hoagwood, K. (2003). Context matters: Schools and the “research to practice gap” in children’s mental health. School Psychology Review, 32(2), 153–169.

    Google Scholar 

  • Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.

    Article  CAS  PubMed  Google Scholar 

  • Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014-4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, national Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from http://ies.ed.gov/ncee/edlabs.

  • Sobel, M. E. (2006). What do randomized studies of housing mobility demonstrate: Causal inference in the face of interference. Journal of the American Statistical Association, 101, 1398–1407.

    Article  CAS  Google Scholar 

  • Stanger, C., Ryan, S. R., Fu, H., & Budney, A. J. (2011). Parent training plus contingency management for substance abuse families: A Complier Average Causal Effects (CACE) analysis. Drug and Alcohol Dependence, 118, 119–126. doi:10.1016/j.drugalcdep.2011.03.007.

    Article  PubMed  PubMed Central  Google Scholar 

  • Stuart, E. A., Perry, D. F., Le, H., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science, 9, 288–298. doi:10.1007/s11121-008-0104-y.

    Article  PubMed  PubMed Central  Google Scholar 

  • Weiss, M. J., Bloom, H. S., & Brock, T. (2013). A conceptual framework for studying the sources of variation in program effects. MDRC. New York: MDRC.

    Google Scholar 

Download references

Acknowledgments

This research was supported in part by grants from the Institute of Education [R305A080326; R305A130060] and the National Institute of Mental Health [P30 MH08643]. We also thank Celene Domitrovich for her contribution to the project.

Author information

Authors and Affiliations

  1. Curry School of Education, University of Virginia, Charlottesville, VA, 22904, USA

    Juliette K. Berg & Catherine P. Bradshaw

  2. Department of Psychiatry and Behavioral Sciences, School of Medicine, Stanford University, Stanford, CA, 94305, USA

    Booil Jo

  3. Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, 21218, USA

    Nicholas S. Ialongo

  4. American Institutes for Research, 1000 Thomas Jefferson Street, NW, Washington, DC, 20007, USA

    Juliette K. Berg

Authors
  1. Juliette K. Berg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Catherine P. Bradshaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Booil Jo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicholas S. Ialongo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juliette K. Berg.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Berg, J.K., Bradshaw, C.P., Jo, B. et al. Using Complier Average Causal Effect Estimation to Determine the Impacts of the Good Behavior Game Preventive Intervention on Teacher Implementers. Adm Policy Ment Health 44, 558–571 (2017). https://doi.org/10.1007/s10488-016-0738-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-016-0738-1

Keywords

Search

Navigation