Abstract
A growing body of research has documented a link between variation in implementation dosage and outcomes associated with preventive interventions. Complier Average Causal Effect (CACE; Jo in J Educ Behav Stat 27:385–409, 2002) analysis allows for estimating program impacts in light of variation in implementation. This study reports intent-to-treat (ITT) and CACE findings from a randomized controlled trial (RCT) testing the impacts of the universal PAX Good Behavior Game (PAX GBG) integrated with Promoting Alternative Thinking Strategies (i.e., PATHS to PAX) and PAX GBG only compared to a control. This study used ratings by 318 K-5 teachers of 1526 at-risk children who, at baseline, were rated as displaying the top 33rd percentile of aggressive-disruptive behavior. Leveraging a prior study on these data (Berg et al. in Admin Policy Ment Health Ment Health Serv Res 44:558–571, https://doi.org/10.1007/s10488-016-0738-1, 2017), CACE was defined as the effect of intervention assignment for compliers, using two compliance cut points (50th and 75th percentile), on posttest ratings of student academic engagement, social competence, peer relations, emotion regulation, hyperactivity, and aggressive-disruptive behavior. The ITT analyses indicated improvements for students in the integrated condition on ratings of social competence compared to the control condition. The CACE analyses also indicated significant effects of the integrated intervention on social competence, as well as academic engagement and emotion regulation for students in high compliance classrooms. These findings illustrate the importance of considering variation in implementation within the context of RCTs.
Similar content being viewed by others
References
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444–455.
Barnard, J., Frangakis, C. E., Hill, J. L., & Rubin, D. (2003). Principal stratification approach to broken randomized experiments. Journal of American Statistical Association, 98, 299–323. https://doi.org/10.1198/016214503000071.
Berg, J., Bradshaw, C. P., Jo, B., & Ialongo, N. S. (2017). Using complier average causal effect estimation to determine the impacts of the Good Behavior Game preventive intervention on teacher implementers. Administration and Policy in Mental Health and Mental Health Services Research, 44, 558–571. https://doi.org/10.1007/s10488-016-0738-1.
Bradshaw, C. P., & Kush, J. M. (2019). Teacher observation of classroom adaptation-checklist: Measuring children’s social, emotional, and behavioral functioning. Children & Schools, 42(1), 29–40. https://doi.org/10.1093/cs/cdz022.
Bradshaw, C. P., Waasdorp, T. E., & Leaf, P. J. (2015). Examining variation in the impact of school-wide positive behavioral interventions and supports: Findings from a randomized controlled effectiveness trial. Journal of Educational Psychology, 107(2), 546–557.
Bradshaw, C. P., Zmuda, J. H., Kellam, S. G., & Ialongo, N. S. (2009). Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school. Journal of Educational Psychology, 101(4), 926–937. https://doi.org/10.1037/a0016586.
Chen, H. T. (1998). Theory-driven evaluations. Advances in Educational Productivity, 7, 15–34.
Conduct Disorder Problems Research Group. (1999). Initial impact of the Fast Track prevention trial for conduct problems II: Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.
Connell, A. M., Dishion, M. Y., Yasui, M., & Kavanagh, K. (2007). An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent behavior. Journal of Consulting and Clinical Psychology, 75, 568–579. https://doi.org/10.1037/0022-006X.75.4.568.
Cowen, J. M. (2008). School choice as a latent variable: Estimating the “complier average causal effect” of vouchers in Charlotte. The Policy Studies Journal, 36, 301–315.
Domitrovich, C., Bradshaw, C., Greenberg, M., Embry, D., Poduska, J., & Ialongo, N. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47, 71–88.
Domitrovich, C., Bradshaw, C. P., Berg, J., Pas, E., Becker, K., Musci, R., et al. (2016). How do school-based prevention programs impact teachers? Findings from a randomized trial of an integrated classroom management and social-emotional program. Prevention Science, 17, 325–337.
Domitrovich, C., Pas, E., Bradshaw, C. P., Becker, K., Keperling, J. P., Embry, D., et al. (2015). Individual and school organizational factors that influence implementation of the Pax Good Behavior Game intervention. Prevention Science, 6(8), 1064–1074.
Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K. E., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.
Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students' social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432.
Embry, D., Staatemeier, G., Richardson, C., Lauger, K., & Mitich, J. (2003). The PAX good behavior game (1st ed.). Center City, MN: Hazelden.
Fixsen, D., Naoom, S., Blasé, K., Friedman, R., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute. The National Implementation Research Network (FMHI Publication #231).
Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58, 21–29.
Greenberg, M., & Kusché, C. (2006). Building social and emotional competence: the PATHS Curriculum. In S. R. Jimerson & M. J. Furlong (Eds.), Handbook of school violence and school safety: From research to practice (pp. 395–412). Mahwah, NJ: Erlbaum.
Greenberg, M., Kusché, C., Cook, E., & Quamma, J. (1995). Promoting emotional competence in school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117–136.
Greenberg, M. T., Kusché, C. A., & Conduct Problems Prevention Research Group. (2011). Grade level PATHS (Grades 3–5). South Deerfield, MA: Channing-Bete Co.
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–970.
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.
Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional and Behavioral Disorders, 9, 146–160.
Ialongo, N., Wethamer, L., Kellam, S., Brown, C., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and anti-social behavior. American Journal of Community Psychology, 27, 599–641.
Ialongo, N. S., Domitrovich, C. E., Embry, D., Greenberg, M., Lawson, A., Becker, C., et al. (2019). A randomized controlled trial of the combination of two school-based universal preventive Interventions. Developmental Psychology, 56(6), 1313–1325. https://doi.org/10.1037/dev0000715.
Jo, B. (2002). Estimation of intervention effects with noncompliance: Alternative model specifications. Journal of Educational and Behavioral Statistics, 27, 385–409.
Jo, B., Asparouhov, T., & Muthén, B. O. (2008a). Intention-to-treat analysis in cluster randomized trials with noncompliance. Statistics in Medicine, 27, 5565–5577. https://doi.org/10.1002/sim.3370.
Jo, B., Asparouhov, T., Muthén, B. O., Ialongo, N. S., & Brown, C. H. (2008b). Cluster randomized trials with treatment noncompliance. Psychological Methods, 13, 1–18.
Kellam, S., Brown, C. H., Poduska, J., Ialongo, N., Wang, W., Toyinbo, P., et al. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95S, S5–S28.
Kellam, S. G., Ling, X., Merisca, R., Brown, C. H., & Ialongo, N. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–185.
Kellam, S. G., Wang, W., Mackenzie, A. C. L., Brown, C. H., Ompad, D. C., Or, F., et al. (2014). The impact of the Good Behavior Game, a universal classroom-based preventive intervention in first and second grades, on high-risk sexual behaviors and drug abuse and dependence disorders into young adulthood. Prevention Science, 15, 6–18. https://doi.org/10.1007/s11121-012-0296-z.
Koth, C. W., Bradshaw, C. P., & Leaf, P. J. (2009). Teacher Observation of Classroom Adaptation-Checklist (TOCA-C): Development and factor structure. Measurement and Evaluation in Counseling and Development, 42, 15–30. https://doi.org/10.1177/0748175609333560.
Kusché, C. A., Greenberg, M. T., & Conduct Problems Prevention Research Group. (2011). Grade level PATHS (Grades 1–2). South Deerfield, MA: Channing-Bete Co.
Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. Hoboken, NJ: Wiley.
Little, R. J. A., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin’s causal model. Psychological Methods, 3, 147–159.
Musci, R., Pas, E. T., Bettencourt, A., Masyn, K., Ialongo, N. S., & Bradshaw, C. P. (2019). How does collective student behavior and other classroom contextual factors relate to implementation of an evidence-based intervention? A multilevel SEM. Development and Psychopathology, 31, 1827–1835. https://doi.org/10.1017/s095457941900097x.
O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33–84.
Pas, E., & Bradshaw, C. P. (2014). What affects teacher ratings of student behaviors? The potential influence of teachers’ perceptions of the school environment and experiences. Prevention Science, 15, 940–950. https://doi.org/10.1007/s11121-013-0432-4.
Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014-4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, national Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from https://ies.ed.gov/ncee/edlabs.
Sobel, M. E. (2006). What do randomized studies of housing mobility demonstrate: Causal inference in the face of interference. Journal of the American Statistical Association, 101, 1398–1407.
Stanger, C., Ryan, S. R., Fu, H., & Budney, A. J. (2011). Parent training plus contingency management for substance abuse families: A Complier Average Causal Effects (CACE) analysis. Drug and Alcohol Dependence, 118, 119–126. https://doi.org/10.1016/j.drugalcdep.2011.03.007.
Stuart, E. A., Perry, D. F., Le, H., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science, 9, 288–298. https://doi.org/10.1007/s11121-008-0104-y.
Werthamer-Larsson, L., Kellam, S., & Wheeler, L. (1991). Effect of first grade classroom environment on shy behavior, aggressive behavior, and concentration problems. American journal of community psychology, 19, 585–602.
Acknowledgements
This research was supported in part by grants from the Institute of Education [R305A080326; R305A130060] and the National Institute of Mental Health [P30 MH08643]. The authors would like to thank Celene Domitrovich for her contributions to the project and Booil Jo for consultation on the analyses.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bradshaw, C.P., Shukla, K.D., Pas, E.T. et al. Using Complier Average Causal Effect Estimation to Examine Student Outcomes of the PAX Good Behavior Game When Integrated with the PATHS Curriculum. Adm Policy Ment Health 47, 972–986 (2020). https://doi.org/10.1007/s10488-020-01034-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-020-01034-1