American Journal of Community Psychology

, Volume 27, Issue 5, pp 711–731 | Cite as

Opening the Black Box: Using Process Evaluation Measures to Assess Implementation and Theory Building

  • Tracy W. Harachi
  • Robert D. Abbott
  • Richard F. Catalano
  • Kevin P. Haggerty
  • Charles B. Fleming


The past decade has seen increasing recognition in prevention science of the need to move away from a black box approach to intervention evaluation and toward an approach that can elaborate on the mechanisms through which changes in the outcomes operate (Chen & Rossi, 1989; Durlak & Wells, 1997; Spoth et al., 1995). An approach that examines issues of program implementation is particularly critical in the design of efficacy studies of school-based preventive interventions. Numerous preventive intervention strategies are now delivered within the schools, often by regular classroom teachers. The extent to which teachers faithfully deliver a particular curriculum or incorporate instructional strategies emphasized by an intervention is a critical question for the overall project evaluation. This article illustrates the utilization of process measures from a multicomponent school-based prevention program to examine implementation of a teaching staff development intervention, and the program's underlying theoretical basis. Given the nested study design, the analyses utilize hierarchical linear models (Bryk & Raudenbush, 1992) to examine changes in teaching strategies by condition and investigate the hypothesized relationships between teaching practices and student behaviors based on the program's theoretical framework. Results suggest that teaching practices in two of the six intervention focus areas were positively impacted in the first 18 months of the project. Findings also support th relationships between teachers' instructional practices and students' behaviour.

program implementation school-based prevention 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Achenbach, T. M. (1991). Manual for the teacher's report form and 1991 profile. Burlington, VT: University of Vermont, Department of Psychiatry.Google Scholar
  2. Altman, D. G. (1986). A framework for evaluation of community based heart disease prevention programs. Social Science and Medicine, 22, 479–487.PubMedCrossRefGoogle Scholar
  3. Arnold, C. L. (1992). An introduction to hierarchical linear models. Measurement and Evaluation in Counseling and Development, 25, 58–90.Google Scholar
  4. Aronson, E., Bridgeman, D. L., & Geffner, R. (1978). Interdependent interactions and prosocial behavior. Journal of Research and Development in Education, 12, 16–27.Google Scholar
  5. Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the Child Development Project: Early findings from an ongoing multisite demonstration trial. Special Issue: Preventing adolescent substance abuse. Journal of Adolescent Research, 11, 12–35.Google Scholar
  6. Brophy, J. (1987). Synthesis of research on strategies for motivating students to learn. Educational Leadership, 45, 40–48.Google Scholar
  7. Brophy, J. (1990). Teaching social studies for understanding and higher-order applications. Elementary School Journal, 90, 351–417.CrossRefGoogle Scholar
  8. Brophy, J., & Good, T. L. (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed.), Handbook of research on training (pp. 328–375). New York: Macmillan.Google Scholar
  9. Bryk, A. S., & Raudenbush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 1011, 147–158.CrossRefGoogle Scholar
  10. Bryk, A. S., and Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications.Google Scholar
  11. Caplan, M. Z., & Weissberg, R. P. (1989). Promoting social competence in early adolescence: Developmental considerations. In B. H. Schneider, G. Attili, J. Nadel, & R. P. Weissberg (Eds.), Social competence in developmental perspective (pp. 371–385). Boston: Kluwer Academic.CrossRefGoogle Scholar
  12. Catalano, R. F., & Hawkins, J. D. (1996). The social development model: A theory of antisocial behavior. In J. D. Hawkins (Ed.), Delinquency and crime: Current theories (pp. 149–197). New York: Cambridge University Press.Google Scholar
  13. Catalano, R. F., Kosterman, R., Hawkins, J. D., Newcomb, M. D., & Abbott, R. D. (Spring, 1996). Modeling the etiology of adolescent substance use: A test of the social development model. Journal of Drug Issues: Empirical Validity of Theories of Drug Abuse, 26, 429–455.Google Scholar
  14. Chen, H., & Rossi, P. H. (1989). Issues in the theory-driven perspective. Evaluation and Program Planning, 12, 299–306.CrossRefGoogle Scholar
  15. Dobson, L. D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3, 269–376.CrossRefGoogle Scholar
  16. Doyle, W. (1986). Classroom organization and management. In M. E. Wittrock (Ed.), Handbook of research on teaching, 392–431. New York: Macmillan.Google Scholar
  17. Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. Special Issue: Meta-analysis of primary prevention programs. American Journal of Community Psychology, 25, 115–152.PubMedCrossRefGoogle Scholar
  18. Dusenbury, L., & Falco, M. (1997). School-based drug abuse prevention strategies: From research to policy and practice. In R. P. Weissberg, T. P. Gullotta, R. L. Hampton, B. A. Ryan, & G. R. Adams (eds.), Health children 2010: Enhancing children's wellness. (pp. 47–75). Thousand Oaks, CA: Sage Publications.Google Scholar
  19. Farrington, D. P., & Hawkins, J. D. (1991). Predicting participation, early onset, and later persistence in officially recorded offending. Criminal Behavior and Mental Health, 1, 1–33.Google Scholar
  20. Fielding, L. G., & Pearson, D. P. (1994). Reading comprehension: What works? Educational Leadership, 1, 62–68.Google Scholar
  21. Finney, J. W., & Moos, R. H. (1989). Theory and method in treatment evaluation. Evaluation and Program Planning, 12, 307–316.CrossRefGoogle Scholar
  22. Goodstadt, M. J. (1988). School based drug education in North America: What is wrong? What can be done? Journal of School Health, 56, 278–281.CrossRefGoogle Scholar
  23. Gottfredson, D. C. (1990). Changing school structures to benefit high-risk youths. In P. E. Leone (Ed.), Understanding troubled and troubling youth (pp. 246–271). Newbury Park, CA: Sage Publications.Google Scholar
  24. Haggerty, K. P., Catalano, R. F., Harachi, T. W., & Abbott, R. D. (1998). Preventing adolescent problem behaviors: A comprehensive intervention description. Criminologie, 3(1), 25–47.CrossRefGoogle Scholar
  25. Harachi, T. W., Haggerty, K. P., Catalano, R. E., Cummings, C., Gangnes, D., & Abbott, R. A. (1993). RHC classroom observation system manual. Unpublished instrument.Google Scholar
  26. Hawkins, J. D., Abbott, R. A., Catalano, R. F., & Gillmore, M. R. (1991). Assessing effectiveness of drug abuse prevention: Long-term effects and replication. In C. Leukfeld & W. Bukoski (Eds.), Drug abuse prevention research: Methodological issues. NIDA Research Monograph 107, DHHS Publication No. 91-1761:195-212. Washington, DC: U.S. Government Printing Office.Google Scholar
  27. Hawkins, J. D., Catalano, R. F., & Miller, J. Y. (1992a). Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: Implications for substance abuse prevention. Psychological Bulletin, 54, 661–664.Google Scholar
  28. Hawkins, J. D., Catalano, R. F., Morrison, D. M., O'Donnell, J., Abbott, R. D., & Day, L. E. (1992b). The Seattle Social Development Project: Effects of the first four years on protective factors and problem behaviors. In J. McCord & R. Tremblay (Eds.), The prevention of antisocial behavior in children (pp. 139–161). New York: Guilford.Google Scholar
  29. Hawkins, J. D., Doueck, H. J., & Lishner, D. M. (1988). Changing teaching practices in mainstream classes to improve bonding and behavior of low achievers. American Educational Research Journal, 25, 31–50.Google Scholar
  30. Hawkins, J. D., & Weis, J. G. (1985). The social development model: An integrated approach to delinquency prevention. Journal of Primary Prevention, 6, 73–97.CrossRefGoogle Scholar
  31. Johnson, D. W., & Johnson, R. T. (1980). Effects of cooperative, competitive, and individualistic learning experiences on cross-ethnic interaction and friendships. Journal of Social Psychology, 118, 47–58.CrossRefGoogle Scholar
  32. Kassebaum, G., Ward, D. A., & Wilner, D. M. (1971). Prison treatment and parole survival. New York: John Wiley.Google Scholar
  33. Kerr, D. M., Kent, L., & Lam, T. C. M. (1985). Measuring program implementation with a classroom observation instrument. The interactive teaching map. Evaluation Review, 9, 461–482.CrossRefGoogle Scholar
  34. Leithwood, K. A., & Montgomery, D. J. (1980). Evaluating program implementation. Evaluation Review, 4, 193–214.CrossRefGoogle Scholar
  35. Lipps, G., & Grant, P. R. (1990). A participatory method of assessing program implementation. Evaluation Review, 14, 427–434.CrossRefGoogle Scholar
  36. McGraw, S. A., McKinlay, S. M., McClements, L., Lasater, T. M., Assaf, A., & Carleton, R. A. (1989). Methods in program evaluation: The process evaluation system of the Pawtucket Heart Health Program. Evaluation Review, 13, 459–483.CrossRefGoogle Scholar
  37. McLaughlin, M. W. (1987). Implementation realities and evaluation design. Evaluation Studies Review Annual, 12, 73–97.Google Scholar
  38. Norman, S. A., Greenberg, R., Marconi, K., Novelli, W., Felix, M., Schechter, C., Stolley, P., & Stunkard, A. (1990). A process evaluation of a two-year community, cardiovascular risk reduction program: What was done and who knew about it? Health Education Research, 5, 87–97.CrossRefGoogle Scholar
  39. Patterson, G. R. (1986). Performance models for antisocial boys. American Psychologist, 41, 432–444.PubMedCrossRefGoogle Scholar
  40. Patton, M. Q. (1979). Evaluation of program implementation. Evaluation Studies Review Annual, 4, 318–345.Google Scholar
  41. Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. P., Dwyer, J. H., & Johnson, C. A. (1990). Effects of program implementation on adolescent drug use behavior. The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264–289.CrossRefGoogle Scholar
  42. Quay, H. C. (1979). The three faces of evaluation: What can be expected to work. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual, (Vol. 4). Beverly Hills, CA: Sage Publications.Google Scholar
  43. Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.PubMedCrossRefGoogle Scholar
  44. Rotheram, M. J. (1982). Social skills training for underachievers, disruptive, and exceptional children. Psychology in the Shools, 19, 532–539.CrossRefGoogle Scholar
  45. Routman, R. (1991). Invitations. Portsmouth, NH: Heinemann.Google Scholar
  46. Schaps, E., Moskowitz, J. M., Malvin, J. H., & Schaeffer, G. A. (1986). Evaluation of seven school-based prevention programs: A final report on the Napa project. International Journal of the Addictions, 21, 1081–1112.PubMedGoogle Scholar
  47. Slavin, R. E. (1990). Cooperative learning theory, research and practice. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  48. Spoth, R., & Redmond, C. (1996). Illustrating a framework for prevention research: Project Family Studies of rural family participation and outcomes. In R. Peters & R. McMahon (Eds.), Childhood disorders, substance abuse, and delinquency: Prevention and early intervention approaches (pp. 299–328). Newbury Park, CA: Sage.Google Scholar
  49. Spoth, R., Redmond, C., Haggerty, K., & Ward, T. (1995). A controlled outcome study examining individual difference and attendance effects. Journal of Primary Prevention, 57, 449–464.Google Scholar
  50. Walberg, H. J. (1988). Synthesis of research on time and learning. Educational Leadership, 45, 76–85.Google Scholar
  51. Walker, H. M., & McConnell, S. R. (1988). The Walker McConnell Scale of social competence and school adjustment. Austin, TX: Proc-Ed.Google Scholar
  52. Weissberg, R. P., Gesten, E. L., Carnrike, C. L., Toro, P. A. Rapkin, B. D., Davidson, E., & Cowen, E. L. (1981). Social problem-solving skills training: A competence building intervention with second-to fourth-grade students. American Journal of Community Psychology, 9, 411–423.CrossRefGoogle Scholar
  53. Wentzel, K. R. (1991). Social competence at school: Relation between social responsibility and academic achievement. Review of Education Research, 61, 1–24.Google Scholar
  54. Werthamer-Larsson, L., Kellam, S. G., & Ovesen-McGregor, K. E. (1990). Teacher interview: Teacher Observation of Classroom Adaptation—Revised (TOCA-R). In S. G. Kellam (Ed.), Johns Hopkins prevention training manual. Baltimore: Johns Hopkins University.Google Scholar

Copyright information

© Plenum Publishing Corporation 1999

Authors and Affiliations

  • Tracy W. Harachi
    • 1
  • Robert D. Abbott
    • 1
  • Richard F. Catalano
    • 1
  • Kevin P. Haggerty
    • 1
  • Charles B. Fleming
    • 1
  1. 1.Social Development Research GroupUniversity of WashingtonSeattle

Personalised recommendations