Trends and Developments in Program Evaluation in General and Criminal Justice Programs in Particular

Abstract

The paper first describes general trends in evaluation in (mainly) western societies. Why is evaluation growing, what are the characteristics of this ‘growth industry’ and what developments are occurring outside western societies? Trends in the evaluation of criminal justice programs in the USA, the UK and the Netherlands are then discussed. Two important developments are therafter highlighted: experimental evaluations and theory-driven evaluations. Both approaches are discussed, and some pros and cons are listed. Finally, the paper outlines some challenges for future work in the evaluation of criminal justice programs, stressing the importance of combining good designs with both program and social science theories.

This is a preview of subscription content, access via your institution.

References

  1. Bennet, T., The effectiveness of a police-initiated fear-reducing strategy. British Journal of Criminology, 31(1), pp. 1–14, 1991.

    Google Scholar 

  2. Boruch, R., P. Davies, H. Soydan, I. Chalmers, H. Cooper, A. Petrosino, D. DeMoya, D. Myers, A. Oxman, G. Macdonald, L. Hedges, R. Harden, R. Lilley et al., Progress on Developing the Campbell Collaboration: July 1999 and February 2000. Philadelphia: Center for Research & Evaluation in Social Policy, University of Pennsylvania, 2000.

  3. Campbell, D.T. and J.C. Stanley, Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally, 1963.

    Google Scholar 

  4. Chelimsky, E., The politics of program evaluation. Social Science and Modern Society, 25(24–32), 1987.

    Google Scholar 

  5. Cook, T.D., Why have educational evaluators chosen not to do randomized experiments? The Annals of the American Academy of Political and Social Science, 589, pp. 114–148, 2003.

  6. Cook, T.D. and C.S. Reichardt (Eds.), Qualitative and Quantitative Methods in Evaluation Research. Beverly Hills, CA: Sage, 1979.

    Google Scholar 

  7. Cooksy, L. and F.L. Leeuw, Evaluating the performance of development agencies: The role of meta-evaluations. In: G.K. Pitman, O.N. Feinstein and G.K. Ingram (Eds.), Evaluating development effectiveness, World Bank Series on Evaluation and Development, pp. 95–108. New Brunswick: Transaction Publishers, 2004.

  8. Derlien, H.U., Genesis and structure of evaluation efforts in comparative perspective. In: R.C. Rist (Ed.), Program evaluation and the management of government, Patterns and prospects across eight nations, pp. 147–177. New Brunswick, NJ: Transaction Publishers, 1989.

  9. Dickens, B., Contribution to the Symposium of Learning What Works: Evaluating Complex Social Interventions, Washington, DC: The Brookings Institution, 1997.

  10. Duflo, E. and M. Kramer, Use of randomization in the evaluation of development effectiveness. In: G.K. Pitman, O.N. Feinstein and G.K. Ingram (Eds.), Evaluating development effectiveness, World Bank Series on Evaluation and Development, pp. 205–232. New Brunswick, NJ: Transaction Publishers, 2004.

  11. Eckblom, P. and K. Pease, Evaluation crime prevention. In: M.H. Tonry and M.H. Farrington (Eds.), Building a Safer Society: Strategic Approaches to Crime Prevention. Chicago: University of Chicago Press, 1995.

    Google Scholar 

  12. Farrington, D.P., Methodological quality standards for evaluation. The Annals of the American Academy of Political and Social Science Studies, 587, pp. 49–68, 2003.

  13. Farrington, D.P. and B.C. Welsh, Home Office Research Study 251: Effects of improved street lighting on crime: A systematic review. London: Home Office, Development and Statistics Directorate, 2002.

    Google Scholar 

  14. Freeman, H.E., The present status of evaluation research. In: M.A. Guttenberg and S. Saar (Eds.), Evaluation Studies Review Annual, 2, pp. 17–51, 1977.

  15. Furubo, J.E., R.C. Rist and R. Sandahl (Eds.), The International Atlas of Evaluation. New Brunswick, NJ: Transaction Publishers, 2002a.

    Google Scholar 

  16. Furubo, J.E. and R. Sandahl, Introduction: a diffusion perspective on global developments in evaluation. In: J.E. Furubo, R.C. Rist and R. Sandahl (Eds.), The International Atlas of Evaluation, pp. 1–27. New Brunswick, NJ: Transaction Publishers, 2002b.

  17. Gill, M. and A. Spriggs, Assessing the impact of CCTV. London: Home Office, Development and Statistics Directorate, 2005.

    Google Scholar 

  18. Goldblatt, P. and C. Lewis, Home Office Research Study 187: Reducing offending: An assessment of research evidence on ways of dealing with offending behaviour. London: Home Office, Development and Statistics Directorate, 1998.

    Google Scholar 

  19. Gray, A., B. Jenkins and F.L. Leeuw (Eds.), Collaboration in public services: the challenge for evaluation. New Brunswick, NJ: Transaction Publishers, 2003.

    Google Scholar 

  20. Hedstrom, P. and R. Swedberg (Eds.), Social Mechanisms. An Analytical Approach to Social Theory. Cambridge, UK: Cambridge University Press, 1998.

    Google Scholar 

  21. Hovland, C.I., et al., Experiments in Mass communication. Vol. III: The American Soldier. Princeton, NJ: Princeton University Press, 1949.

    Google Scholar 

  22. Johnson, S.D., K.J. Bowers, P. Jordan, N. Davidson and A. Hirschfield, Estimating crime reduction outcomes: how many crimes were prevented? Evaluation 10(3), pp. 327–348, 2004.

    Article  Google Scholar 

  23. Klein Haarhuis, C., M. van Ooyen-Houben, E.R. Kleemans and F.L. Leeuw, Kennis voor beleid; een synthese van 58 (evaluatie)onderzoeken op het gebied van rechtshandhaving, The Hague: WODC, 2005.

  24. Latour, B. and S. Woolgar, Laboratory Life. Beverly Hills, CA: Sage, 1979.

    Google Scholar 

  25. Leeuw, F.L., Policy theories, knowledge utilization, and evaluation. Knowledge and Policy, 4, pp. 73–92, 1991.

  26. Leeuw, F.L., Unintended side effects of auditing: The relationship between performance auditing and performance improvement and the role of trust. In: J. Weesie and W. Raub (Eds.), The management of durable relations, pp. 95–97. Amsterdam: Thela, 2000.

  27. Leeuw, F.L., Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24(1), pp. 5–20, 2003.

    Google Scholar 

  28. Leeuw, F.L., Managing evaluations in the Netherlands and types of knowledge. In: R. Rist and N. Stame (Eds.), From Studies to Streams. New Brunswick, NJ: Transaction Publishers, in press.

  29. Leeuw, F.L., R.C. Rist and R.C. Sonnichsen (Eds.), Can governments learn: comparative perspectives on evaluation and organizational learning. New Brunswick, NJ: Transaction Publishers, 1994.

    Google Scholar 

  30. Leeuw, F.L. and J. Toulemonde, Evaluation activities in Europe: A quick scan of the market in 1998. Evaluation, 5, pp. 487–496, 1999.

    Article  Google Scholar 

  31. Leeuw, F.L. and S. Van Thiel, The performance paradox in the public sector. Public Productivity and Management Review, 25, pp. 123–143, 2002.

    Google Scholar 

  32. Mayne, J. and R. Schwartz (Eds.), Quality Matters. New Brunswick, NJ: Transaction Publishers, 2004.

    Google Scholar 

  33. Muir, E., They blinded me with political science: On the use of on-peer reviewed research in education policy. Political Science and Politics, 32(4), pp. 762–764, 1999.

    Article  Google Scholar 

  34. Nuttal, C., The home office and random allocation experiments. Evaluation Review, 27(3), pp. 267–290, 2003.

    Article  PubMed  Google Scholar 

  35. Oakley, A., Experiments in knowing. London: Polity Press, 2000.

    Google Scholar 

  36. Orr, L.L., Social Experiments: Evaluating Public Programs with Experimental Methods. Thousand Oaks, CA: Sage, 1999.

    Google Scholar 

  37. Pawson, R., Does Megan's Law Work? A theory-driven systematic review, London: Queen Mary University of London, Working Paper 8, ESRC UK Centre for Evidence Based Policy and Practice, 2002a.

  38. Pawson, R., Evidence-based policy: The promise of ‘realist synthesis'. Evaluation, 8(3), pp. 340–358, 2002b.

    Article  Google Scholar 

  39. Pawson, R., Simple principles for the evaluation of complex programmes. Working Paper, Leeds: Leeds University, 2004.

  40. Pawson, R. and N. Tilley, What works in evaluation research. British Journal of Criminology, 34(3), pp. 291–307, 1994.

    Google Scholar 

  41. Pawson, R. and N. Tilley, Realistic evaluation. London: Sage, 1997.

    Google Scholar 

  42. Pease, K., A review of street lighting evaluations: Crime reductions effects. Crime Prevention Studies, 10, pp. 47–76, 1999.

    Google Scholar 

  43. Petrosino, A., et al., A Proposal for a Campbell Criminal Justice Group Paper developed for the February 2000 meeting of the Campbell Collaboration Draft 3, February 8th 2000. Campbell Collaboration, 2000.

  44. Picciotto, R., Evaluation in the World Bank: antecedents, methods, and instruments. In: J. E. Furubo, R. C. Rist and R. Sandahl (Eds.), The International Atlas of Evaluation, pp. 425–439. New Brunswick, NJ: Transaction Publishers, 2002.

  45. Power, M., The Audit Society. London: Penguin, 1997.

    Google Scholar 

  46. Power, M., The Risk Management of Everything. Rethinking the Politics of Uncertainty. London: Demos, 2004.

    Google Scholar 

  47. Powers, E. and H. Witmer, An experiment in the prevention of juvenile delinquency: The Cambridge-Somerville Youth Study. New York: Columbia University Press, 1951.

    Google Scholar 

  48. Rist, R.C. (Ed.), Program evaluation and the management of government. Patterns and prospects across eight nations. New Brunswick, NJ: Transaction Publishers, 1989.

    Google Scholar 

  49. Rist, R. and K.L. Paliokas, The rise and fall (and rise again?) of the evaluation function in the U.S. Government. In: J.E. Furubo, R.C. Rist and R. Sandahl (Eds.), International Atlas of Evaluation. New Brunswick, NJ: Transaction Publishers, 2002.

    Google Scholar 

  50. Rist, R. and N. Stame (Eds.), From Studies to Streams. New Brunswick, NJ: Transaction Publishers, in press.

  51. Rossi, P.H., H.E. Freeman and M.W. Lipsey, Evaluation. A Systematic Approach. Thousand Oaks: Sage Publication, 1999.

    Google Scholar 

  52. Scriven, M., Goal-free evaluation. In: E.R. House (Ed.), School Evaluation: The Politics and Processes (pp. 319–328). Berkeley, CA: McCutchan, 1973.

    Google Scholar 

  53. Scriven, M., The Logic of Evaluatin. Inverness, CA: Edgepress, 1980.

    Google Scholar 

  54. Shadish, W.R., Program Evaluation: A Pluralistic Enterprise. San Francisco: Jossey Bass, 1993.

    Google Scholar 

  55. Shadish, W.R., T.D. Cook and L.C. Leviton, Foundations of program evaluation, theories of practice. London: Sage, 1995.

    Google Scholar 

  56. Sherman, L.W., D.P. Farrington, B.C. Welsh and D. Layton Mackenzie (Eds.), Evidence-based crime prevention. Routledge: London, 2002.

    Google Scholar 

  57. Sherman, L.W., D. Gottfredson, D. Mackenzie, J. Eck, P. Reuter and S. Bushway. Preventing Crime: What Works, What Doesn't, What's Promising, a Report to the United States Congress. Prepared for the National Institute of Justice, Department of Criminology and Criminal Justice, University of Maryland, 1997.

  58. Stame, N., Theory-based evaluation and types of complexity. Evaluation, 10(1), pp. 58–76, 2004.

    Article  Google Scholar 

  59. Stame, N., Streams of evaluative knowledge: Introduction. In: R. Rist and N. Stame (Eds.), From studies to streams, pp. 1–23. New Brunswick, NJ: Transaction Publishers, in press.

  60. Summa, H. and J. Toulemonde, Evaluation in the EU: Addressing complexity and ambiguity. In: J.E. Furubo, R.C. Rist and R. Sandahl (Eds.), The International Atlas of Evaluation, pp. 407–425. New Brunswick, NJ: Transaction Publishers, 2002.

    Google Scholar 

  61. The Economist, Try it and see, 28 February 2002.

  62. Tilley, N., Applying theory-driven evaluation to the British Crime Reduction Programme. Criminal Justice, 4(3), pp. 255–276, 2004.

    Article  Google Scholar 

  63. US GAO, Prospective Evaluation Methods: The prospective evaluation synthesis. Washington: US General Accounting Office, 1990.

    Google Scholar 

  64. US GAO, Teenage pregnancy: 500,000 births a year but few tested programs. Washington: US General Accounting Office, 1996.

    Google Scholar 

  65. Vischer, C.A. and D. Weisburd, Identifying what works: Recent trends in crime prevention strategies. Crime, Law and Social Change, 28, pp. 223–242, 1998.

    Google Scholar 

  66. Wartna, B., N.J. Baas and E. Beenakkers, Beter, anders en goedkoper: een literatuurverkenning ten behoeve van het traject Modernisering Sanctietoepassing. Memorandum 2004-1. The Hague: WODC, 2004.

  67. Weeks, H.A., Youthful offenders at Highfields. Ann Arbor, MI.: University of Michigan Press, 1958.

    Google Scholar 

  68. Weiss, C.H., Where politics and evaluation research meet. Evaluation, 1, pp. 37–45, 1973.

    Google Scholar 

  69. Weisburd, D. and J.E. Eck, What can police do to reduce crime, disorder, and fear? The Annals of the American Academy of Political and Social Science, 593, pp. 42–65, 2004.

    Article  Google Scholar 

  70. Wholey, J.S., Evaluability assessment: developing program theory. New Directions for Program Evaluation, 33, pp. 77–92, 1987.

    Article  Google Scholar 

  71. Wildavsky, A., The self-evaluating organization. Public Administration Review, 32, pp. 509–520, 1972.

    Article  Google Scholar 

  72. Wittebrood, K. and M. Van Beem, Sociale veiligheid vergroten door gelegenheidsbeperking: wat werkt en wat niet. Raad voor de Maatschappelijke Ontwikkeling, bijlage 6, pp. 271–368, 2004.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Frans L. Leeuw.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Leeuw, F.L. Trends and Developments in Program Evaluation in General and Criminal Justice Programs in Particular. Eur J Crim Policy Res 11, 233–258 (2005). https://doi.org/10.1007/s10610-005-0835-2

Download citation

Keywords

  • criminal justice evaluations
  • experimental designs
  • future challenges regarding the combination of theory-driven approaches and experimental ones
  • history of evaluations
  • international developments
  • theory-driven evaluations