Skip to main content
SpringerLink
Log in

How you can tell if the simulations in computational criminology are any good

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Computational criminology applies computer simulations to study topics of interest for criminologists. Just as for all computer modelling in science, the validity of the simulations ultimately depends on whether they are able to reproduce empirical phenomena with sufficient accuracy. The only way in which this can be determined is by comparing model output to real observations. This paper provides an overview of how such model evaluations can be undertaken.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. It is also interesting how many otherwise excellent treatments of computer simulation models make little or no mention of model evaluation or validation (e.g., Cellier 1991; Gilbert and Troitzsch 2005; Miller and Page 2007).

References

  • Austin, J., Naro, W., & Fabelo, T. (2007). Public safety, public spending: Forecasting America’s prison population 2007–2011. Washington, DC: The Pew Charitable Trust, Public Safety Performance Project.

    Google Scholar 

  • Auerhahn, K. (2007). Do you know who your probationers are? Using simulation modeling to estimate the composition of California’s probation population, 1980–2000. Justice Quarterly, 23(4), 28–47.

    Article  Google Scholar 

  • Bayarri, M. J., Berger, J. O., Cafeo, J. A., Cavendish, J., Lin, J. C., Paulo, R., et al. (2007a). A framework for validation of computer models. Technometrics, 49(2), 138–154.

    Article  Google Scholar 

  • Bayarri, M. J., Walsh, D.,Berger, J. O., Cafeo, J., Garcia-Donato, G., Liu, F., et al. (2007b). Computer model validation with functional output. Annals of Statistics, 35(5), 1874–1906.

    Article  Google Scholar 

  • Berk, R. A., & MacDonald, J. (2007). The dynamics of crime regimes. Working paper, Department of Criminology, University of Pennsylvania.

  • Berk, R. A., Messinger, S. L., Rauma, D., & Berecochea, J. E. (1983). Prisons as self-regulating systems: A comparison of historical patterns in california for male and female offenders. Law and Society Review, 17(4), 547–586.

    Article  Google Scholar 

  • Berk, R. A., Bond, J., Lu, R., Turco, R., & Weiss, R. E. (2000a). Computer simulations as experiments: Using program evaluation tools to assess the validity of interventions in virtual worlds. In L. Bickman (Ed.), Donald Campbell’s legacy (Volume II): Contributions to research design. Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Berk, R. A., Schoenberg, F., Fovell, R., Lu, R., & Weiss, R. (2000b). Approximation and inversion of a complex meteorological system via local linear filters. Journal of Applied Meteorology, 40, 446–458.

    Google Scholar 

  • Berk, R. A., Fovell, R. G., Schoenberg, F., & Weiss, R. E. (2001). The use of statistical tools for evaluating computer simulations. Climatic Change, 51, 119–130.

    Article  Google Scholar 

  • Berk, R. A., Bickel, P., Campbell, K., Keller-McNulty, S., Kelly, E., & Sacks, J. (2002). Workshop on statistical approaches for the evaluation of complex computer models. Statistical Science, 17, 173–192.

    Article  Google Scholar 

  • Blumstein, A. (1967). Systems analysis of the criminal justice system. Annals of the American Academy of Political and Social Science, 374, 92–100.

    Article  Google Scholar 

  • Breiman, L. (2001a). Statistical modeling: Two cultures (with discussion). Statistical Science, 16, 199–231.

    Article  Google Scholar 

  • Breiman, L. (2001b). Random forests. Machine Learning, 45, 5–32.

    Article  Google Scholar 

  • Carley, K. M. (1996). Validating computational models. Pittsburgh, PA: Carnegie Mellon University, Department of Sociology.

    Google Scholar 

  • Clark, M., & Holm, E. (1987). Microsimulation methods and spatial analysis planning. Geografiska Annaler, Series B, Human Geography, 69(2), 145–164.

    Article  Google Scholar 

  • Cellier, F. E. (1991). Continuous system modeling. New York: Springer-Verlag.

    Google Scholar 

  • Cressie, N. A. (1993). Statistics for spatial data. New York: John Wiley.

    Google Scholar 

  • Edmunds, B. (2006). How are physical and social spaces related? — Cognitive agents as necessary glue. In F. C. Billari, T. Fent, A. Prskawetz, & J. Scheffran (Eds.), Agent-based computational modeling: Applications in demography, social, economic, and environmental sciences. New York: Physica-Verlag, Springer.

    Google Scholar 

  • Gass, S. I., & Thompson, B. W. (1980). Guidelines for model evaluation: An abridged version of the US General Accounting Office Exposure Draft. Operations Research, 28(2), 431–439.

    Google Scholar 

  • Fawcett, K. R., Anderson, M. G., Bates, P. D., Jordon, J.-P. & Bathhurst, J. J. (1995). The importance of internal validation in the assessment of physically based distributed models. Transactions of the Institute of British Geographers, New Series, 20(2), 248–265.

    Article  Google Scholar 

  • Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics and Data Analysis, 38, 367–378.

    Article  Google Scholar 

  • Gilbert, N. (2008). Agent-based models. Los Angeles: Sage Publications.

    Google Scholar 

  • Gilbert, N., & Troitzsch, K. G. (2005). Simulation for the Social Scientist (2nd ed.). New York: Open University Press.

    Google Scholar 

  • Goldstein, J. (1999). Emergence as a construct: History and issues. Emergence: Complexity, and Organization, 1, 49–82.

    Google Scholar 

  • Grassl, H. (2000). Status and improvement of coupled circulation models. Science, 288, 1991–1997.

    Article  Google Scholar 

  • Groff, L. R. (2007). Simulation for theory testing and experimentation: An example using routine activity theory and street robbery. Journal of Quantitative Criminology, 23, 75–103.

    Article  Google Scholar 

  • Hänninen, H. (1995). Assessing ecological implications of climate change: Can we rely on our simulation models? Climate Change, 31, 1–4.

    Article  Google Scholar 

  • Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning: Data mining, inference, and prediction. New York: Springer-Verlag.

    Google Scholar 

  • Haylock, R. G., & O’Hagan, A. (1996). On inference for outputs of computationally expensive algorithms with uncertainty on the inputs. In J. M. Bernardo, et al. (Eds.), Bayesian statistics 5 (pp. 629–637). London: Oxford University Press.

    Google Scholar 

  • Hamilton, J. D. (1994). Time series analysis. Princeton: Princeton University Press.

    Google Scholar 

  • Hanneman, R. A. (1995). Simulation modeling and theoretical analysis in sociology. Sociological Perspectives, 38(4), 457–462.

    Google Scholar 

  • Jones, G. T., Mezic, I., Gruenwald, P. J., & Gorman, D. M. (2007). Agent-based modeling: Use with necessary caution. American Journal of Public Health, 97(5), 780–781.

    Article  Google Scholar 

  • Kelter, D. (1973). A random walk epidemic simulation. Journal of the American Statistical Association, 68(344), 821–823.

    Article  Google Scholar 

  • Kennedy, M. C., & O’Hagan, A. (2001). Bayesian calibration of computer models (with discussion). Journal of the Royal Statistical Society, Series B, 63, 425–464.

    Article  Google Scholar 

  • Kirman, A. P., & Vriend, N. J. (2001). Evolving market structure: An ACE Model of price dispersion and loyalty. Journal of Economic Dynamics and Control, 25(3–4), 459–502.

    Article  Google Scholar 

  • Kleindorfer, G. B., O’Neill, L., & Ganeshan, R. (1998). Validation in simulation: Various positions in the philosophy of science. Management Science, 44(8), 1087–1099.

    Article  Google Scholar 

  • Kleijen, J. P. C. (1995). Verification and validation of simulation models. European Journal of Operations Research, 82, 145–162.

    Article  Google Scholar 

  • Leik, R. K., & Meeker, B. F. (1995). Computer simulation for exploring theories: Models of interpersonal cooperation and competition. Sociological Perspectives, 38(4), 463–482.

    Google Scholar 

  • Lim, Y. B., Sacks, J., Studden, W. J., & Welch, W. J. (2002). Design and analysis of computer experiments when the output is highly correlated over the input space. The Canadian Journal of Statistics, 30(1), 109–126.

    Article  Google Scholar 

  • Linkletter, C., Bingham, D., Hengartner, N., Higdon, D., & Ye, K. Q. (2006). Variable selection for Gaussian process models in computer experiments. Technometrics, 48(4), 478–489.

    Article  Google Scholar 

  • Love, G., & Back, G. (2000). Model verification and validation for rapidly developed simulation models: Balancing cost and theory. In Proceedings of the 18th international conference of the system dynamics society, Bergen, Norway.

  • McKay, M. D., Conover, W. J., & Beckman, R. J. (1979). A comparison of three methods for selecting values of input variables in the analysis of output from computer code. Technometrics, 21(45), 239–245.

    Article  Google Scholar 

  • Mease, D., & Bingham, D. (2006). Latin hypercube sampling for computer experiments. Technometrics, 48(4), 467–477.

    Article  Google Scholar 

  • Miller, J. H., & Page, S. E. (2007). Complex adaptive systems: An introduction to computational models of social life. Princeton: Princeton University Press.

    Google Scholar 

  • Mihram, G. A. (1972). Some practical aspects of the verification and validation of simulation models. Operations Research Quarterly, 23(1), 17–29.

    Google Scholar 

  • Oakley, J. E., & O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. Journal of the Royal Statistical Society, Series B, 66, 751–769.

    Article  Google Scholar 

  • O’Hagan, A., Kennedy, M. C., & Oakley, J. E. (1998). Uncertainty analysis and other inference tools for complex computer codes. In J. M. Bernardo, J. O. Berger, A. P. David, & A. F. M. Smith (Eds.), Bayesian statistics, 6. Oxford: University Press.

    Google Scholar 

  • Perez, P., & Dray A. (2005). SimDrug: Exploring the complexity of heroin use in Melbourne. Drug Policy Modeling Project: Monograph 11. Fitzroy, Australia: Turning Point Alcohol and Drug Centre.

  • Petit, J. R., Jouzel, J., Raynaud, D., Barkov, N. I., Barnola, J. M., Basile I., et al. (1999). Climate and Atmospheric History of the past 420,000 years from the Vostok Ice Core, Antarctica. Nature, 399, 429–436.

    Article  Google Scholar 

  • Randall, D. A. (Ed.) (2000). General circulation model development: Past present and future. New York: Academic Press.

    Google Scholar 

  • Sacks, J., Welch, W., Mitchell, T. J., & Wynn, H. P. (1989). Design and analysis of computer experiments. Statistical Science, 4, 409–435.

    Article  Google Scholar 

  • Sacks, J., Rouphail, N. M., Park, B. B., & Thakuriah, P. (2000). Statistically-based validation of computer simulation models in traffic operations and management. Paper submitted to the Journal of Transportation and Statistics.

  • Saltelli, A., Chan, K., & Scott, M. (Eds.) (2000). Sensitivity analysis. Chichester, UK: Wiley.

    Google Scholar 

  • Santner, T. J., Williams, B. J., & Notz, W. I. (2003). The design and analysis of computer experiments. New York: Springer.

    Google Scholar 

  • Schoenberg, F., Berk, R. A., Fovell, R. G., Li, C., Lu, R., & Weiss, R. E. (2000). Approximation and inversion of a complex meteorological system via local linear filters. Journal of Applied Meteorology, 40(3), 446–458.

    Article  Google Scholar 

  • Sargent, R. G. (1998). Verification and validation of simulation models. In D. J. Medekros, E. F. Watson, J. S. Carson, & M. S. Manivannan (Eds.), Proceedings of the 1998 winter simulation conference, Piscataway, New Jersey, IEEE, 500–503.

  • Tabor, C. S., & Timpone, R. J. (1996). Computational modeling. Los Angeles: Sage Publications.

    Google Scholar 

  • Tesfatsion, L. (2002). Agent-based computational economics: Growing economies from the bottom up. Artificial Life, 8, 55–82.

    Article  Google Scholar 

  • Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (Ed.), 18th European simulation multiconference: networked simulation and simulation networks (pp. 265–270). SCS Publishing House: Erlangen, San Diego CA.

  • van Baal, P. V. (2004). Computer simulations of criminal deterrence: From public policy to local interaction to individual behavior. Den Hagg, Netherlands: BJU.

    Google Scholar 

  • Van Dyke, P., Savit, R., & Riolo, R. L. (1998). Agent-based modeling vs. equation-based modeling: A case study and user’s guide. In J. A. Sichman, R. Conte, & N. Gilbert (Eds.), Multi-agent systems and agent-based simulation. First International Workshop, MABS 98, Paris, France, July 4–6, 1998. Proceedings. Heidelberg: Springer.

  • Vicek, T. (2000). Complexity: The bigger picture. Nature, 418, 131.

    Article  Google Scholar 

  • Vilaand, B. J., & Cohen, L. E. (1993). Crime as strategy: Testing an evolutionary ecological theory of expropriative behavior. American Journal of Sociology, 98(4), 873–912.

    Article  Google Scholar 

  • Windrum, P., Fagiolo G., & Moneta, A. (2007). Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulations, 10(2), 8. http://jasss.soc.surrey.ac.uk/10/2/8.html.

    Google Scholar 

  • Younger, S (2005). Violence and revenge in Egalitarian Societies. Journal of Artificial Societies and Social Simulation, 8(4), 11. http://jasss.soc.surrey.ac.uk/8/4/11.html.

    Google Scholar 

  • Ziegler, B. P., Praefofer, H., & Kim, T. G. (2000). Theory of modeling and simulation: Integrating discrete event and continuous complex dynamic systems (2nd ed.). New York: Academic Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Criminology, University of Pennsylvania, McNeil Building, Suite 483 3718 Locust Walk, Philadelphia, PA, 19104-6286, USA

    Richard Berk

  2. Department of Statistics, University of Pennsylvania, McNeil Building, Suite 483 3718 Locust Walk, Philadelphia, PA, 19104-6286, USA

    Richard Berk

Authors
  1. Richard Berk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Berk.

Additional information

Work on this paper was funded by a grant from the National Science Foundation: SES-0437169, “Ensemble methods for Data Analysis in the Behavioral, Social and Economic Sciences.” This support is gratefully acknowledged. Very useful comments on an earlier draft were provided by Liz Groff and Larraine Mazerolle.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Berk, R. How you can tell if the simulations in computational criminology are any good. J Exp Criminol 4, 289–308 (2008). https://doi.org/10.1007/s11292-008-9053-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-008-9053-5

Keywords

Search

Navigation