Prevention Science

, Volume 14, Issue 2, pp 121–133 | Cite as

Methodological Challenges Examining Subgroup Differences: Examples from Universal School-based Youth Violence Prevention Trials

  • Albert D. FarrellEmail author
  • David B. Henry
  • Amie Bettencourt


This article reviews the literature on school-based universal violence prevention programs to illustrate key methodological challenges for investigating subgroup differences in prevention effects. The variety of potential moderating factors examined within this literature is discussed within the context of a social-ecological model. Our review of this literature identified the following methodological issues: the need for a clear a priori theoretical basis for selecting potential moderators, inflated Type I error rates that result from large numbers of comparisons, the absence of explicit tests of moderation, interpretive issues arising from a restricted range on moderator variables, the failure to report effect size estimates, the presence of potential confounding factors, and the importance of examining factors that might operate at multiple ecological levels. These points are illustrated using examples of studies, primarily within youth violence prevention research, that have identified factors within the individual, school, and community that moderate the outcomes of preventive interventions. We conclude with general recommendations for future work. These include the benefits of using the social-ecological model to provide a basis for moving from exploratory to more theory-driven confirmatory models of subgroup differences, the potential merits of qualitative research designed to identify factors that may influence the effectiveness of intervention efforts for specific subgroups of individuals, and the provision of effect size estimates and confidence intervals for effect sizes in prevention reports.


Prevention Moderation Subgroup analyses Violence prevention 


  1. Aber, J. L., Jones, S. M., Brown, J. L., Chaundry, N., & Samples, F. (1998). Resolving conflict creatively: Evaluating the developmental effects of a school-based violence prevention program in neighborhood and classroom context. Development and Psychopathology, 10, 187–214.PubMedCrossRefGoogle Scholar
  2. Aber, J. L., Brown, J. L., & Jones, S. M. (2003). Developmental trajectories toward violence in middle childhood: Course, demographic differences, and response to school-based intervention. Developmental Psychology, 3, 324–348.CrossRefGoogle Scholar
  3. Asch, S. E. (1955). Opinions and social pressure. Scientific American, 193, 31–35.CrossRefGoogle Scholar
  4. Bronfenbrenner, U. (1979). Contexts of child rearing: Problems and prospects. American Psychologist, 34, 844–850.CrossRefGoogle Scholar
  5. Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., et al. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48, 1013–1022.PubMedCrossRefGoogle Scholar
  6. Conduct Problems Prevention Research Group. (1999a). Initial impact of the Fast Track prevention trial for conduct problems: I. The high-risk sample. Journal of Consulting and Clinical Psychology, 67, 631–647.CrossRefGoogle Scholar
  7. Conduct Problems Prevention Research Group. (1999b). Initial impact of the Fast Track prevention trial for conduct problems II: Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.CrossRefGoogle Scholar
  8. Conduct Problems Prevention Research Group. (2010). The effects of a multiyear universal social—emotional learning program: The role of student and school characteristics. Journal of Consulting and Clinical Psychology, 78, 156–168.CrossRefGoogle Scholar
  9. Dahlberg, L. L., & Krug, E. G. (2002). Violence—a global public health problem. In E. Krug, L. L. Dahlberg, J. A. Mercy, A. B. Zwi, & R. Lozano (Eds.), World report on violence and health (pp. 1–56). Geneva, Switzerland: World Health Organization.Google Scholar
  10. Davis, C. H., MacKinnon, D. P., Schultz, A., & Sandler, I. (2003). Cumulative risk and population attributable fraction in prevention. Journal of Clinical Child and Adolescent Psychology, 32, 228–235.PubMedGoogle Scholar
  11. Dishion, T. J., Patterson, G. R., & Griesler, P. C. (1994). Peer adaptation in the development of antisocial behavior: A confluence model. In L. R. Huesmann (Ed.), Aggressive behavior: Current perspectives (pp. 61–95). New York: Plenum Press.Google Scholar
  12. Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm: Peer groups and problem behavior. American Psychologist, 54, 755–764.PubMedCrossRefGoogle Scholar
  13. Dishion, T. J., Kavanagh, K., Schneiger, A., Nelson, S., & Kaufman, N. K. (2002). Preventing early adolescent substance use: A family-centered strategy for the public middle school. Prevention Science, 3, 191–201.PubMedCrossRefGoogle Scholar
  14. Dodge, K. A., Lansford, J. E., & Dishion, T. J. (2006). The problem of deviant peer influences in intervention programs. In K. A. Dodge, T. J. Dishion, & J. E. Lansford (Eds.), Deviant peer influences in programs for youth: Problems and solutions (pp. 3–13). New York: Guilford.Google Scholar
  15. Fagan, J., & Wilkinson, D. L. (1998). Social contexts and functions of adolescent violence. In D. S. Elliott, B. A. Hamburg, & K. R. Williams (Eds.), Violence in American schools: A new perspective (pp. 55–93). New York: Cambridge University Press.Google Scholar
  16. Farrell, A. D., & Meyer, A. L. (1997). The effectiveness of a school-based curriculum for reducing violence among urban sixth-grade students. American Journal of Public Health, 87, 979–984.PubMedCrossRefGoogle Scholar
  17. Farrell, A. D., & Vulin-Reynolds, M. (2007). Violent behavior and the science of prevention. In D. J. Flannery, A. T. Vazsonyi, & I. D. Waldman (Eds.), The Cambridge handbook of violent behavior and aggression (pp. 767–786). New York: Cambridge University Press.CrossRefGoogle Scholar
  18. Farrell, A. D., Meyer, A. L., Kung, E. M., & Sullivan, T. N. (2001a). Development and evaluation of school-based violence prevention programs. Journal of Clinical Child Psychology, 30, 207–220.CrossRefGoogle Scholar
  19. Farrell, A. D., Meyer, A. L., & White, K. S. (2001b). Evaluation of Responding in Peaceful and Positive Ways (RIPP): A school-based prevention program for reducing violence among urban adolescents. Journal of Clinical Child Psychology, 30, 451–463.CrossRefGoogle Scholar
  20. Farrell, A. D., Erwin, E. H., Bettencourt, A., Mays, S., Vulin-Reynolds, M., Sullivan, T. N., et al. (2008). Individual factors influencing effective nonviolent behavior and fighting in peer situations: A qualitative study with urban African American adolescents. Journal of Clinical Child and Adolescent Psychology, 37, 397–411.PubMedGoogle Scholar
  21. Farrell, A. D., Mays, S., Bettencourt, A., Erwin, E. H., Vulin-Reynolds, M., & Allison, K. W. (2010). Environmental influences on fighting versus nonviolent behavior in peer situations: A qualitative study with urban African American adolescents. American Journal of Community Psychology, 46, 19–35.PubMedCrossRefGoogle Scholar
  22. Farrell, A. D., Henry, D., Mays, S., & Schoeny, M. (2011). Parents as moderators of the impact of peers and school environment. Child Development, 82, 146–161.PubMedCrossRefGoogle Scholar
  23. Flannery, D. J., Vazsonyi, A. T., & Waldman, I. D. (Eds.). (2007). The Cambridge handbook of violent behavior and aggression. New York: Cambridge University Press.Google Scholar
  24. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.PubMedCrossRefGoogle Scholar
  25. Foshee, V. A., Bauman, K. E., Ennett, S. T., Suchindran, C., Benefield, T., & Linder, G. F. (2005). Assessing the effects of the dating violence prevention program "Safe Dates" using random coefficient regression modeling. Prevention Science, 6, 245–258.PubMedCrossRefGoogle Scholar
  26. Fraley, C., & Raferty, A. E. (1998). How many clusters? Which clustering method? Answers via model-based cluster analysis. The Computer Journal, 41, 478–488.CrossRefGoogle Scholar
  27. Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., et al. (2005). Social information-processing skills training to promote social competence and prevent aggressive behavior in the third grade. Journal of Consulting and Clinical Psychology, 73, 1045–1055.PubMedCrossRefGoogle Scholar
  28. Frey, K. S., Hirchstein, M. K., & Guzzo, B. A. (2000). Second Step: Preventing aggression by promoting social competence. Journal of Emotional and Behavioral Disorders, 8, 102–112.CrossRefGoogle Scholar
  29. Gottfredson, D. C., Gottfredson, G. D., & Hybl, L. G. (1993). Managing adolescent behavior: A multiyear, multischool study. American Educational Research Journal, 30, 179–215.Google Scholar
  30. Grabosky, P. N. (1996). Unintended consequences of crime prevention. In R. Homel (Ed.), Crime prevention studies, vol 5: The politics and practice of situational crime prevention (pp. 25–57). Monsey, NY: Criminal Justice Press.Google Scholar
  31. Gregory, A., Henry, D., Schoeny, M. & The Metropolitan Area Child Study Research Group. (2007). School climate and implementation of a preventive intervention. American Journal of Community Psychology, 40, 250–260.CrossRefGoogle Scholar
  32. Griffin, J. P., Jr., Chen, D., Eubanks, A., Brantley, K. M., & Willis, L. A. (2007). A randomized violence prevention trial with comparison: Responses by gender. Journal of School Violence, 6, 65–81.CrossRefGoogle Scholar
  33. Hanish, L. D., & Guerra, N. G. (2000). The roles of ethnicity and school context in predicting children's victimization by peers. American Journal of Community Psychology. Special Issue: Minority Issues in Prevention, 28, 201–223.CrossRefGoogle Scholar
  34. Hawkins, J. D., Herrenkohl, T., Farrington, D. P., Brewer, D., Catalano, R. F., Harachi, T. W., et al. (1998). A review of predictors of youth violence. In R. Loeber & D. P. Farrington (Eds.), Serious & violent juvenile offenders: Risk factors and successful interventions (pp. 106–146). Thousand Oaks, CA: Sage Publications.Google Scholar
  35. Henry, D., & Chan, W. Y. (2010). Cross-sectional and longitudinal effects of sixth grade setting-level norms for nonviolent problem solving on aggression and associated attitudes. Journal of Community Psychology, 38, 1007–1022.CrossRefGoogle Scholar
  36. Henry, D. B., & Schoeny, M. E. (2007, June). Moderation of universal intervention effects by norms and ethnicity (S-11372: “Violence and poverty: Testing theoretical models for African American youth”). Paper presented at the Bienniel Meeting of the Society for Community Research and Action, Pasadena, CA.Google Scholar
  37. Henry, D. B., Farrell, A. D., Schoeny, M. E., Tolan, P. H., & Dymnicki, A. B. (2010). Influence of school-level variables on aggression and associated attitudes of middle school students. Manuscript submitted for publication.Google Scholar
  38. Huesmann, L. R., Maxwell, C., Eron, L., Dahlberg, L. L., Guerra, N. G., Tolan, P. H., et al. (1996). Evaluating a cognitive/ecological program for the prevention of aggression in urban children. Preventive Medicine, 12, 120–128.Google Scholar
  39. Jackson, J. (1966). A conceptual and measurement model for norms and roles. Pacific Sociological Review, 9, 35–47.CrossRefGoogle Scholar
  40. Jaycox, L. H., McCaffrey, D., Eiseman, B., Aronoff, J., Shelley, G. A., & Collins, R. L. (2006). Impact of a school-based dating violence prevention program among Latino teens: Randomized controlled effectiveness trial. Journal of Adolescent Health, 39, 694–704.PubMedCrossRefGoogle Scholar
  41. Jo, B. (2002). Estimation of intervention effects with noncompliance: Alternative model specifications. Journal of Educational and Behavioral Statistics, 27, 385–409.CrossRefGoogle Scholar
  42. Kam, C., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.PubMedCrossRefGoogle Scholar
  43. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.CrossRefGoogle Scholar
  44. Kellam, S. G., Ling, X., Merisca, R., Brown, C. H., & Ialongo, N. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–185.PubMedCrossRefGoogle Scholar
  45. Kiesler, D. J. (1966). Some myths of psychotherapy research and the search for a paradigm. Psychological Bulletin, 65, 110–136.CrossRefGoogle Scholar
  46. Leadbeater, B., Hoglund, W., & Woods, T. (2003). Changing contents? The effects of a primary prevention program on classroom levels of peer relational and physical victimization. Journal of Community Psychology, 31, 397–418.CrossRefGoogle Scholar
  47. Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.CrossRefGoogle Scholar
  48. Lipsey, M. W., & Derzon, J. H. (1998). Predictors of violent or serious delinquency in adolescence and early adulthood: A synthesis of longitudinal research. In R. Loeber & D. P. Farrington (Eds.), Serious & violent juvenile offenders: Risk factors and successful interventions (pp. 86–105). Thousand Oaks, CA: Sage Publications, Inc.Google Scholar
  49. Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767–778.CrossRefGoogle Scholar
  50. MacKinnon, D. P. (2008). Introduction to statistical mediation analysis. New York: Taylor & Francis Group/Lawrence Erlbaum Associates.Google Scholar
  51. MacKinnon, D. P., & Lockwood, C. M. (2003). Advances in statistical methods for substance abuse prevention research. Prevention Science, 4, 155–171.PubMedCrossRefGoogle Scholar
  52. Metropolitan Area Child Study Research Group (MACS). (2002). A cognitive-ecological approach to preventing aggression in urban settings: Initial outcomes for high risk children. Journal of Consulting and Clinical Psychology, 70, 179–194.CrossRefGoogle Scholar
  53. Meyer, A. L., Farrell, A. D., Northup, W., Kung, E., & Plybon, L. (2000a). Promoting nonviolence in early adolescence: Responding in peaceful and positive ways. New York: Kluwer.CrossRefGoogle Scholar
  54. Meyer, A., Nicholson, R., Danish, S., Fries, E., & Polk, V. (2000b). A model to measure program integrity of peer-led health promotion programs in rural middle schools: Assessing the implementation of the sixth grade goals for health program. Journal of Educational and Psychological Consultation, 11, 223–252.CrossRefGoogle Scholar
  55. Multisite Violence Prevention Project. (2008). Impact of a universal school-based violence prevention program on social-cognitive outcomes. Prevention Science, 9, 231–244.CrossRefGoogle Scholar
  56. Multisite Violence Prevention Project. (2009). The ecological effects of universal and selective violence prevention programs for middle school students: A randomized trial. Journal of Consulting and Clinical Psychology, 77, 526–542.CrossRefGoogle Scholar
  57. Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109–118.PubMedCrossRefGoogle Scholar
  58. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2004). Classroom Assessment Scoring System [CLASS]. Unpublished measure. University of Virginia.Google Scholar
  59. Raudenbush, S. W., & Sampson, R. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Sociological Methodology, 29, 1–41.CrossRefGoogle Scholar
  60. Reid, J. B., Eddy, J. M., Fetrow, R. A., & Stoolmiller, M. (1999). Description and immediate impacts of a preventive intervention for conduct problems. American Journal of Community Psychology, 27, 483–517.PubMedCrossRefGoogle Scholar
  61. Rosenthal, R. (1979). The 'file drawer problem' and tolerance for null results. Psychological Bulletin, 86, 638–641.CrossRefGoogle Scholar
  62. Rousseau, D. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1–37.Google Scholar
  63. Segawa, E., Ngwe, J. E., Li, Y., & Flay, B. R. (2005). Evaluation of the effects of the Aban Aya Youth Project in reducing violence among African American adolescent males using latent class growth mixture modeling techniques. Evaluation Review, 29, 128–148.PubMedCrossRefGoogle Scholar
  64. Shinn, M., & Rapkin, B. D. (2000). Cross-level research without cross-ups in community psychology. In J. Rappaport & E. Seidman (Eds.), Handbook of community psychology (pp. 669–695). Dordrecht, The Netherlands: Kluwer Academic Publishers.CrossRefGoogle Scholar
  65. Smith, B., & Sechrest, L. (1991). Treatment of aptitude × treatment interactions. Journal of Consulting and Clinical Psychology, 59, 233–244.PubMedCrossRefGoogle Scholar
  66. Smokowski, P. R., Fraser, M. W., Day, S. H., Galinsky, M. J., & Bacallao, M. L. (2004). School-based skills training to prevent aggressive behavior and peer rejection in childhood: Evaluating the making choices program. Journal of Primary Prevention, 25, 233–251.CrossRefGoogle Scholar
  67. Spoth, R., Redmond, C., Shin, C., Lepper, H., Haggerty, K., & Wall, M. (1998). Risk moderation of parent and child outcomes in a preventive intervention: A test and replication. American Journal of Orthopsychialry, 68, 565–579.CrossRefGoogle Scholar
  68. Steiger, J. H. (2004). Beyond the F Test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9, 164–182.PubMedCrossRefGoogle Scholar
  69. Stoolmiller, M., Eddy, J. M., & Reid, J. B. (2000). Detecting and describing preventive intervention effects in a universal school-based randomized trial targeting delinquent and violent behavior. Journal of Consulting and Clinical Psychology, 68, 296–306.PubMedCrossRefGoogle Scholar
  70. Stuart, E. A., Perry, D. F., Le, H.-N., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science, 9, 288–298.PubMedCrossRefGoogle Scholar
  71. Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 24–31.CrossRefGoogle Scholar
  72. Tolan, P. H., & Brown, C. H. (1998). Evaluation research on violence interventions: Issues and strategies for design. In P. K. Trickett & C. J. Schellenbach (Eds.), Violence against children in the family and the community (pp. 439–464). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  73. Tolan, P. H., Gorman-Smith, D., & Henry, D. (2004). Supporting families in a high-risk setting: proximal effects of the SAFEChildren preventive intervention. Journal of Consulting and Clinical Psychology, 72, 855–869.PubMedCrossRefGoogle Scholar
  74. Tseng, V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology. Special Issue: Systems Change, 39, 217–228.CrossRefGoogle Scholar
  75. US Department of Health and Human Services (USDHHS) Public Health Service (PHS); Office of the Surgeon General. (2001). Youth violence: A report of the Surgeon General. Rockville MD: US Department of Health and Human Services.Google Scholar
  76. Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136–149.PubMedCrossRefGoogle Scholar
  77. Wright, J. C., & Zimmerman, M. A. (2006). Culturally sensitive interventions to prevent youth violence. In N. G. Guerra & E. P. Smith (Eds.), Preventing youth violence in a multicultural society (pp. 221–269). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  78. Yale, M. E., Scott, K. G., Gross, M., & Gonzalez, A. (2003). Using developmental epidemiology to choose the target population for an intervention program in a high-risk neighborhood. Journal of Clinical Child and Adolescent Psychology, 32, 236–242.PubMedGoogle Scholar
  79. Yau, L. H. Y., & Little, R. J. (2001). Inference for the complier-average causal effect from longitudinal data subject to noncompliance and missing data, with application to a job training assessment for the unemployed. Journal of the American Statistical Association, 96, 1232–1244.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2011

Authors and Affiliations

  • Albert D. Farrell
    • 1
    Email author
  • David B. Henry
    • 2
  • Amie Bettencourt
    • 3
  1. 1.Department of PsychologyVirginia Commonwealth UniversityRichmondUSA
  2. 2.School of Public HealthUniversity of Illinois at ChicagoChicagoUSA
  3. 3.Bloomberg School of Public HealthJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations