Skip to main content

Drug and violence prevention: Rediscovering the critical rational dimension of evaluation research

Journal of Experimental Criminology volume 1, pages 39–62 (2005)Cite this article

Abstract

Following criticism of government-funded drug prevention activities of the early 1990s, a spate of “best practice” or “science-based” lists of alcohol, drug and violence prevention programs have been produced by federal agencies in recent years. The writings of Donald T. Campbell on validity have had a profound influence on the development of the methodological quality scales that have been utilized in the review processes used to generate these lists. Implicit in this approach to the identification of science-based prevention programs is the idea that science is equivalent to research methodology and study design. Following Karl Popper and Campbell, I contend that, while certain designs are clearly better than others in dealing with threats to internal validity and allow for better generalization of results beyond the study population, utilization of these designs in and of itself is not sufficient to designate an evaluation study as “scientific.” Nor can the accumulation of data from such studies be used to proclaim an entire area of research a “science,” as has occurred with the field of so-called “prevention science.” Rather, the fundamental criterion by which to judge the scientific status of a theory is falsifiability. If the field of drug and violence prevention is truly a science, then it should be subjecting its predictions about the effects of intervention programs to genuinely critical tests and not attempting to verify these hypotheses. It is argued that it has failed to do this, and two specific examples of prevention programs that appear on a number of science-based lists of prevention programs are discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

  • Alkin, M. C. (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Ashton, M. (2003). The American STAR comes to England. Alcohol and Drug Findings 8, 21–26.

    Google Scholar 

  • Bacon, F. (1620). Novum Organum. Edited and translated by Jardine, L. and Silverthorne, M. 2000. The New Organon. New York: Cambridge University Press.

  • Biglan, A., Mrazek, P. J., Carnine, D. & Flay, B. R. (2003). The integration of research and practice in the prevention of youth problem behaviors. American Psychologist 58, 433–440.

    Google Scholar 

  • Botvin, G. J. (1986). Substance abuse prevention research: Recent developments and future directions. Journal of School Health 56, 369–374.

    Google Scholar 

  • Botvin, G. J. (2000). Inaugural editorial. Prevention Science 1, 1–2.

    Google Scholar 

  • Botvin, G. J. & Kantor, L. W. (2000). Preventing alcohol and tobacco use through Life Skills Training. Alcohol Research and Health 24, 250–257.

    Google Scholar 

  • Botvin, G. J., Eng, A. & Williams, C. L. (1980). Preventing the onset of cigarette smoking through life skills training. Preventive Medicine 9, 135–143.

    Google Scholar 

  • Botvin, G. J., Renick, N. L. & Baker, E. (1983). The effects of scheduling format and booster sessions on a broad-spectrum psychosocial smoking prevention program. Journal of Behavioral Medicine 6, 359–379.

    Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S. & Botvin, E. M. (1990a). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology 58, 437–446.

    Google Scholar 

  • Botvin, G. J., Baker, E., Filazzola, A. D. & Botvin, E. M. (1990b). A cognitive-behavioral approach to substance abuse prevention: One-year follow-up. Addictive Behaviors 15, 47–63.

    Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M. & Diaz, T. (1995). Long-term follow-up of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association 273, 1106–1112.

    Google Scholar 

  • Botvin, G. J., Griffin, K. W., Diaz, T., Schwier, L. M., Williams, C. & Epstein, J. A. (2000). Preventing illicit drug use in adolescents: Long-term follow-up data from a randomized control trial of a school population. Addictive Behaviors 25, 769–774.

    Google Scholar 

  • Brown, J. H. (2001). Youth, drugs and resilience education. Journal of Drug Education 31, 83–122.

    Google Scholar 

  • Bunge, M. (1998). Philosophy of Science: From problem to theory, vol. 1 (Revised edition). New Brunswick, NJ: Transaction Press.

    Google Scholar 

  • Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin 54, 297–312.

    Google Scholar 

  • Campbell, D. T. (1974). Evolutionary epistemology. In P. A. Schilpp (Ed.), The philosophy of Karl Popper (pp. 413–463). La Salle, IL: Open Court.

    Google Scholar 

  • Campbell, D. T. (1984). Can we be scientific in applied Social Science? In R. F. Connor, D. G. Altman & C. Jackson (Eds.), Evaluation studies: Review annual, vol. 9 (pp. 26–48). Beverly Hills, CA: Sage.

    Google Scholar 

  • Campbell, D. T. (1988). The experimenting society. In E. S. Overman (Ed.), Methodology and epistemology for social science: Selected Papers–Donald T. Campbell (pp. 290–314). Chicago: University of Chicago Press.

    Google Scholar 

  • Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching (pp. 171–246). Chicago: Rand McNally.

    Google Scholar 

  • Catalano, R. F., Kosterman, R., Hawkins, J. D., Newcomb, M. D. & Abbott, R. D. (1996). Modeling the etiology of adolescent substance use: A test of the social development model. Journal of Drug Issues 26, 429–455.

    Google Scholar 

  • Center for Substance Abuse Prevention. (2001). 2001 Annual report of the science-based prevention programs. Rockville, MD: Center for Substance Abuse Prevention.

    Google Scholar 

  • Coalition for Evidence-Based Policy. (2003). Bringing evidence-driven progress to crime and substance-abuse policy: A recommended federal strategy. Washington, DC: Council for Excellence in Government.

    Google Scholar 

  • Cook, T. D. & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.

    Google Scholar 

  • Elliott, D. S. (1998). Prevention programs that work for youth: Violence prevention. Boulder, CO: Center for the Study & Prevention of Violence, University of Colorado.

    Google Scholar 

  • Farrington, D. P. (2003). Methodological quality standards for evaluation research. Annals of the American Academy of Political and Social Science 587, 49–58.

    Google Scholar 

  • Farrington, D. P. & Welsh, B. C. (1999). Delinquency prevention using family-based interventions. Children and Society 13, 287–303.

    Google Scholar 

  • Gerstein D. R. & Green, L. W. (1993). Preventing drug abuse: What do we know? Washington, DC: National Academy Press.

    Google Scholar 

  • Gillmore, M. R., Hawkins, J. D., Catalano, R. F., Day, L. E., Moore, M. & Abbott, R. (1991). Structure of problem behaviors in preadolescence. Journal of Consulting and Clinical Psychology 59, 499–506.

    Google Scholar 

  • Goodstadt, M. S. (1980). Drug education–A turn on or a turn off. Journal of Drug Education 10, 89–99.

    Google Scholar 

  • Gorman, D. M. (1995). On the difference between statistical and practical significance in school-based drug abuse prevention. Drugs: Education, Prevention, and Policy 2, 275–283.

    Google Scholar 

  • Gorman, D. M. (1996). Etiological theories and the primary prevention of drug use. Journal of Drug Issues 26, 505–520.

    Google Scholar 

  • Gorman, D. M. (1997). The failure of drug education. The Public Interest No. 129, 50–60.

    Google Scholar 

  • Gorman, D. M. (1998). The irrelevance of evidence in the development of school-based drug prevention policy, 1986–1996. Evaluation Review 22, 118–146.

    Google Scholar 

  • Gorman, D. M. (2002a). The “Science” of drug and alcohol prevention: The case of the randomized trial of the Life Skills Training (LST) Program. International Journal of Drug Policy 13, 21–26.

    Google Scholar 

  • Gorman, D. M. (2002b). Defining and operationalizing “research-based” prevention: A critique (with case studies) of the US Department of Education’s Safe, Disciplined and Drug-Free Schools Exemplary Programs. Evaluation & Program Planning 25, 295–302.

    Google Scholar 

  • Gorman, D. M. (2002c). Overstating the behavioral effects of the Seattle Social Development Project (letter). Archives of Pediatrics & Adolescent Medicine 156, 1155.

    Google Scholar 

  • Gorman, D. M. (2003a). Prevention programs and scientific nonsense. Policy Review No. 117, 65–75.

    Google Scholar 

  • Gorman, D. M. (2003b). The best of practices, the worst of practices: The making of science-based primary prevention programs. Psychiatric Services 54, 1087–1089.

    Google Scholar 

  • Gorman, D. M. (2003c). Assessing outcomes of prevention interventions: In reply. Psychiatric Services 54, 1548.

    Google Scholar 

  • Gorman, D. M. (2005). Does measurement dependence explain the effects of the Life Skills Training program on smoking outcomes? Preventive Medicine 40, 479–487.

    Google Scholar 

  • Gorman, D. M. & Speer, P. W. (1996). Preventing alcohol abuse and alcohol-related problems through community interventions: A review of evaluation studies. Psychology & Health 11, 95–131.

    Google Scholar 

  • Gottfredson, D. C., Wilson, D. B. & Najaka, S. S. (2002). School-based crime prevention. In L. W. Sherman, D. P. Farrington, B. C. Welsh & D. L. MacKenzie (Eds.), Evidence-based crime prevention (pp. 56–164). New York: Routledge.

    Google Scholar 

  • Gruner, A., Murphy-Graham, E., Petrosino, A., Remignanti, S., & Weiss, C. H. (in review). The devil is in the details: Examining the evidence for “proven” school-based drug abuse prevention programs. Prevention Science.

  • Hawkins, J. D., Catalano, R. F., Jones, G. & Fine, D. (1987). Delinquency prevention through parent training: Results and issues from work in progress. In J. Q. Wilson & G. C. Lowry (Eds.), From Children to Citizens: Families, Schools, and Delinquency Prevention, vol. 3 (pp. 186–204). New York: Springer-Verlag.

    Google Scholar 

  • Hawkins, J. D., Von Cleve, E. & Catalano, R. F. (1991). Reducing early childhood aggression: Results of a primary prevention program. Journal of the American Academy of Child Adolescent Psychiatry 30, 208–217.

    Google Scholar 

  • Hawkins, J. D., Catalano, R. F., Jones, G. & Fine, D. (1992). The Seattle Social Development Project: Effects of the first four years on protective factors and problem behaviors. In J. McCord & R. E. Tremblay (Eds.), Preventing Antisocial Behavior: Interventions from Birth through Adolescence (pp. 139–161). New York: Guilford Press.

    Google Scholar 

  • Hawkins, J. D., Arthur, M. W. & Catalano, R. F. (1995). Preventing substance abuse. In M. Tonry & D. Farrington (Eds.), Building a Safer Society: Strategic Approaches to Crime Prevention (pp. 343–427). Chicago: University of Chicago Press.

    Google Scholar 

  • Hawkins, J. D., Catalano, R. F., Kosterman, R., Abbott, R. & Hill, K. G. (1999). Preventing adolescent health-risk behaviors by strengthening protection during childhood. Archives of Pediatric & Adolescent Medicine 153, 226–234.

    Google Scholar 

  • Hawkins, J. D., Lonczak, H. S., Abbott, R. D., Kosterman, R. & Catalano, R. F. (2002). Overstating the behavioral effects of the Seattle Social Development Project: In reply, Archives of Pediatrics & Adolescent Medicine 156, 1155–1156.

    Google Scholar 

  • Lincoln, Y. S. & Guba, E. G. (2004). The roots of fourth generation evaluation: theoretical and methodological origins. In M. C. Alkin (Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 225–241). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Kinder, B. N., Pape, N. E, & Walfish, S. (1980). Drug and alcohol education programs: A review of outcome studies. International Journal of the Addictions 15, 1035–1054.

    Google Scholar 

  • MacCoun, R. J. & Reuter, P. (2001). Drug war heresies: Learning from other vices, times, and places. New York: Cambridge University Press.

    Google Scholar 

  • Meehl, P. E. (1997). Credentialed persons, credentialed knowledge. Clinical Psychology 4, 91–97.

    Google Scholar 

  • Mihalic, S., Irwin, K., Elliott, D., Fagan, A. & Hansen, D. (2001). Blueprints for Violence Prevention (OJJDP Juvenile Justice Bulletin). Washington, DC: US Department of Justice.

    Google Scholar 

  • Minkler, M. & Wallerstein, N. (2003). Community based participatory research in health. San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • National Institute on Drug Abuse (2003). Preventing drug use among children and adolescents: A research-based guide for parents, educators, and community leaders. 2nd edn. Bethesda, MD: US Department of Health and Human Services.

    Google Scholar 

  • Petrosino, A. (2003). Standards for evidence and evidence for standards: The case of school-based drug prevention. Annals of the American Academy of Political and Social Science 587, 180–207.

    Google Scholar 

  • Popper, K. R. (1963). Conjectures and Refutations: The growth of scientific knowledge. New York: Routledge.

    Google Scholar 

  • Popper, K. R. (1966). The open society and its enemies: Volume II–Hegel and Marx. 5th edn. revised. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Popper, K. R. (1985). The problem of demarcation. In D. Miller (Ed.), Popper Selections (pp. 118–130). Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Rossi, P. H. (1987). The iron law of evaluation and other metallic rules. Research in Social Problems & Public Policy 4, 3–20.

    Google Scholar 

  • Rossi, P. H. & Freeman, H. E. (1979). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Schinke, S., Brounstein, P. & Gardner, S. (2002). Science-based prevention programs and principles, 2002. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration.

    Google Scholar 

  • Shadish, W. R. & Luellen, J. K. (2004). Donald Campbell: The accidental evaluator. In M.C. Alkin (Ed.). Evaluation roots: Tracing theorists’ views and influences (pp. 80–87). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Sherman, L. W. (2003). Misleading evidence and evidence-led policy: Making social science more experimental. Annals of the American Academy of Political and Social Science 589, 6–19.

    Google Scholar 

  • Sherman. L. W., Gottfredson, D. C., MacKenzie, D. L., Eck, J., Reuter, P. & Bushway, S. D. (1998). Preventing crime: What works, what doesn’t, what’s promising (National Institute of Justice Research in Brief). Washington, DC: US Department of Justice.

    Google Scholar 

  • Sherman, L. W., Farrington, D. P., Welsh, B. C. & MacKenzie, D. L. (2002). Evidence-based crime prevention. New York: Routledge.

    Google Scholar 

  • US Department of Education Safe, Disciplined, and Drug-Free Schools Expert Panel. (2001). Exemplary and promising safe, disciplined, and drug-free schools programs 2001. Jessup, MD: US Department of Education.

    Google Scholar 

  • US Department of Health and Human Services. (2001). Youth violence: A report of the surgeon general. Rockville, MD: US Department of Health and Human Services.

    Google Scholar 

  • US General Accounting Office. (1997). Safe and drug-free schools: Balancing accountability and local flexibility. Washington, DC: US General Accounting Office. Washington, DC: US General Accounting Office.

    Google Scholar 

  • Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  • Weisburd, D., Lum, C. M. & Yang, S. -M. (2003). When can we conclude that treatments or programs “don’t work”? Annals of the American Academy of Political and Social Science 587, 31–48.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Epidemiology and Biostatistics, School of Rural Public Health, Texas A&M Health Science Center, Texas, USA

    Dennis M. Gorman

Authors
  1. Dennis M. Gorman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dennis M. Gorman.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Gorman, D.M. Drug and violence prevention: Rediscovering the critical rational dimension of evaluation research. J Exp Criminol 1, 39–62 (2005). https://doi.org/10.1007/s11292-004-6461-z

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-004-6461-z

Key words

  • best practices
  • drug and violence prevention
  • evaluation
  • evidence-based
  • prevention science