Skip to main content
Log in

The perils of tweaking: how to use macrodata to set parameters in complex simulation models

  • Published:
Synthese Aims and scope Submit manuscript

Abstract

When can macroscopic data about a system be used to set parameters in a microfoundational simulation? We examine the epistemic viability of tweaking parameter values to generate a better fit between the outcome of a simulation and the available observational data. We restrict our focus to microfoundational simulations—those simulations that attempt to replicate the macrobehavior of a target system by modeling interactions between microentities. We argue that tweaking can be effective but that there are two central risks. First, tweaking risks overfitting the simulation to the data and thus compromising predictive accuracy; and second, it risks compromising the microfoundationality of the simulation. We evaluate standard responses to tweaking and propose strategies to guard against these risks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Axelsen B. E., Anker-Nilssen T., Fossum P., Kvamme C., Nøttestad L. (2001) Pretty patterns but a simple strategy: Predator–prey interactions between juvenile herring and Atlantic puffins observed with multibeam sonar. Canadian Journal of Zoology 79: 1586–1596

    Article  Google Scholar 

  • Barrett, C. L., Beckman, R. J., Berkbigler, K. P., Eubank, S. G., Henson, K. M., Kubicek, D. A., et. al. (2000). TRANSIMS: Transportation analysis simulation system. Los Alamos Unlimited Release (LAUR) 00-1725.

  • Bearman P. S., Moody J., Stovel K. (2004) Chains of affection: The structure of adolescent romantic and sexual networks. The American Journal of Sociology 110(1): 44–91

    Article  Google Scholar 

  • Bishop C. (2006) Pattern recognition and machine learning. Springer, New York

    Google Scholar 

  • Bondi A. (1964) Van der Waals volumes and radii. Journal of Physical Chemistry 68(3): 441–451

    Article  Google Scholar 

  • Burnham K. P., Anderson D. R. (2002) Model selection and multimodal inference: A practical information-theoretic approach. Springer, New York

    Google Scholar 

  • Cetin N., Nagel K., Raney B., Voellmy A. (2002) Large-scale multi-agent transportation simulations. Computer Physics Communications 147: 559–564

    Article  Google Scholar 

  • Dawkins C., Srinivasan T. N., Whalley J. (2001) Calibration. In: Heckman J. J., Leamer E. (eds) Handbook of econometrics. North-Holland, Amsterdam, pp 3653–3701

    Google Scholar 

  • Eubank S., Guclu H., Anil Kumar V. S., Marathe M. V., Srinivasan A., Toroczkai Z. et al (2004) Modeling disease outbreaks in realistic urban social networks. Nature 429: 180–184

    Article  Google Scholar 

  • Forster M. R., Sober E. (1994) How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. British Journal for the Philosophy of Science 45: 1–35

    Article  Google Scholar 

  • Friedman, M. (1953). The methodology of positive economics. In M. Friedman (Ed.), Essays in positive economics (pp. 3–43). Chicago: University of Chicago Press.

  • Glymour C. (1980) Theory and evidence. Princeton University Press, Princeton

    Google Scholar 

  • Hausman D. (1992) Why look under the hood?. In: Hausman D. (eds) Essays on philosophy and economic methodology. Cambridge University Press, Cambridge, pp 70–73

    Chapter  Google Scholar 

  • Hitchcock C., Sober E. (2004) Prediction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science 55: 1–34

    Article  Google Scholar 

  • Hoover K. (2006) A NeoWicksellian in a new classical world: The methodology of Michael Woodford’s interest and prices. Journal of the History of Economic Thought 28(2): 143–149

    Article  Google Scholar 

  • Kincaid H. (1986) Reduction, explanation, and individualism. Philosophy of Science 53(4): 492–513

    Article  Google Scholar 

  • Kirman A. (1992) Whom or what does the representative individual represent?. Journal of Economic Perspectives 6(2): 117–136

    Article  Google Scholar 

  • Kleindorfer G., O’Neill L., Ganeshan R. (1998) Validation in simulation: Various positions in the philosophy of science. Management Science 44(8): 1087–1099

    Article  Google Scholar 

  • Lucas R. E. (1976) Econometric policy evaluation: A critique. In: Brunner K., Meltzer A. H. (eds) The Phillips curve and labor markets. North-Holland, Amsterdam, pp 19–45

    Google Scholar 

  • Machamer P., Darden L., Craver C. (2000) Thinking about mechanisms. Philosophy of Science 67(1): 1–25

    Article  Google Scholar 

  • Mayo D. (2008) How to discount double-counting when it counts: Some clarifications. British Journal for the Philosophy of Science 59: 857–879

    Article  Google Scholar 

  • Müller P., von Storch H. (2004) Computer modelling in atmospheric and oceanic sciences: Building knowledge. Springer, New York

    Book  Google Scholar 

  • Nottestad L., Axelsen B. E. (1999) Herring schooling manoeuvers in response to killer whale attack. Canadian Journal of Zoology 77: 1540–1546

    Article  Google Scholar 

  • Oreskes N., Shrader-Frechette K., Belitz K. (1994) Verification, validation, and confirmation of numerical models in the Earth sciences. Science 263: 641–646

    Article  Google Scholar 

  • Pierce, S., van Gieson, E. J., & Skalak, T. (2004). Multicellular simulation predicts microvascular patterning. The FASEB Journal, express article 10.1096/fj.03-0933fje. Retrieved December 13, 2010, from http://www.fasebj.org/content/early/2004/03/31/fj.03-0933fje.full.pdf.

  • Randall D. A., Wielicki B. A. (1997) Measurements, models, and hypotheses in the atmospheric sciences. Bulletin of the American Meteorological Society 78: 399–406

    Article  Google Scholar 

  • Royall R. (1997) Statistical evidence: A likelihood paradigm. Chapman and Hall/CRC, New York

    Google Scholar 

  • Rykiel E. (1996) Testing ecological models: The meaning of validation. Ecological Modelling 90: 229–244

    Article  Google Scholar 

  • Satz D., Ferejohn J. (1994) Rational choice and social theory. Journal of Philosophy 91(2): 71–87

    Article  Google Scholar 

  • Sokal R. R., Rohlf F. J. (1994) Biometry. W.H. Freeman, New York

    Google Scholar 

  • van der Waals, J. (1910). The equation of state for gases and liquids (1910 Nobel Prize lecture).

  • Woodford M. (2006) Comments on the symposium on interest and prices. Journal of the History of Economic Thought 28(2): 187–198

    Article  Google Scholar 

  • Worrall J. (2002) New evidence for old. In: Gardenfors P. (eds) In the scope of logic, methodology and philosophy of science. Kluwer, Dordrecht, pp 191–209

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Brian Epstein.

Additional information

B. Epstein and P. Forber have contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Epstein, B., Forber, P. The perils of tweaking: how to use macrodata to set parameters in complex simulation models. Synthese 190, 203–218 (2013). https://doi.org/10.1007/s11229-012-0142-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11229-012-0142-7

Keywords

Navigation