Synthese

, Volume 190, Issue 2, pp 203–218 | Cite as

The perils of tweaking: how to use macrodata to set parameters in complex simulation models

Article

Abstract

When can macroscopic data about a system be used to set parameters in a microfoundational simulation? We examine the epistemic viability of tweaking parameter values to generate a better fit between the outcome of a simulation and the available observational data. We restrict our focus to microfoundational simulations—those simulations that attempt to replicate the macrobehavior of a target system by modeling interactions between microentities. We argue that tweaking can be effective but that there are two central risks. First, tweaking risks overfitting the simulation to the data and thus compromising predictive accuracy; and second, it risks compromising the microfoundationality of the simulation. We evaluate standard responses to tweaking and propose strategies to guard against these risks.

Keywords

Microfoundations Overfitting Tuning Calibration Simulation Estimation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Axelsen B. E., Anker-Nilssen T., Fossum P., Kvamme C., Nøttestad L. (2001) Pretty patterns but a simple strategy: Predator–prey interactions between juvenile herring and Atlantic puffins observed with multibeam sonar. Canadian Journal of Zoology 79: 1586–1596CrossRefGoogle Scholar
  2. Barrett, C. L., Beckman, R. J., Berkbigler, K. P., Eubank, S. G., Henson, K. M., Kubicek, D. A., et. al. (2000). TRANSIMS: Transportation analysis simulation system. Los Alamos Unlimited Release (LAUR) 00-1725.Google Scholar
  3. Bearman P. S., Moody J., Stovel K. (2004) Chains of affection: The structure of adolescent romantic and sexual networks. The American Journal of Sociology 110(1): 44–91CrossRefGoogle Scholar
  4. Bishop C. (2006) Pattern recognition and machine learning. Springer, New YorkGoogle Scholar
  5. Bondi A. (1964) Van der Waals volumes and radii. Journal of Physical Chemistry 68(3): 441–451CrossRefGoogle Scholar
  6. Burnham K. P., Anderson D. R. (2002) Model selection and multimodal inference: A practical information-theoretic approach. Springer, New YorkGoogle Scholar
  7. Cetin N., Nagel K., Raney B., Voellmy A. (2002) Large-scale multi-agent transportation simulations. Computer Physics Communications 147: 559–564CrossRefGoogle Scholar
  8. Dawkins C., Srinivasan T. N., Whalley J. (2001) Calibration. In: Heckman J. J., Leamer E. (eds) Handbook of econometrics. North-Holland, Amsterdam, pp 3653–3701Google Scholar
  9. Eubank S., Guclu H., Anil Kumar V. S., Marathe M. V., Srinivasan A., Toroczkai Z. et al (2004) Modeling disease outbreaks in realistic urban social networks. Nature 429: 180–184CrossRefGoogle Scholar
  10. Forster M. R., Sober E. (1994) How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. British Journal for the Philosophy of Science 45: 1–35CrossRefGoogle Scholar
  11. Friedman, M. (1953). The methodology of positive economics. In M. Friedman (Ed.), Essays in positive economics (pp. 3–43). Chicago: University of Chicago Press.Google Scholar
  12. Glymour C. (1980) Theory and evidence. Princeton University Press, PrincetonGoogle Scholar
  13. Hausman D. (1992) Why look under the hood?. In: Hausman D. (eds) Essays on philosophy and economic methodology. Cambridge University Press, Cambridge, pp 70–73CrossRefGoogle Scholar
  14. Hitchcock C., Sober E. (2004) Prediction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science 55: 1–34CrossRefGoogle Scholar
  15. Hoover K. (2006) A NeoWicksellian in a new classical world: The methodology of Michael Woodford’s interest and prices. Journal of the History of Economic Thought 28(2): 143–149CrossRefGoogle Scholar
  16. Kincaid H. (1986) Reduction, explanation, and individualism. Philosophy of Science 53(4): 492–513CrossRefGoogle Scholar
  17. Kirman A. (1992) Whom or what does the representative individual represent?. Journal of Economic Perspectives 6(2): 117–136CrossRefGoogle Scholar
  18. Kleindorfer G., O’Neill L., Ganeshan R. (1998) Validation in simulation: Various positions in the philosophy of science. Management Science 44(8): 1087–1099CrossRefGoogle Scholar
  19. Lucas R. E. (1976) Econometric policy evaluation: A critique. In: Brunner K., Meltzer A. H. (eds) The Phillips curve and labor markets. North-Holland, Amsterdam, pp 19–45Google Scholar
  20. Machamer P., Darden L., Craver C. (2000) Thinking about mechanisms. Philosophy of Science 67(1): 1–25CrossRefGoogle Scholar
  21. Mayo D. (2008) How to discount double-counting when it counts: Some clarifications. British Journal for the Philosophy of Science 59: 857–879CrossRefGoogle Scholar
  22. Müller P., von Storch H. (2004) Computer modelling in atmospheric and oceanic sciences: Building knowledge. Springer, New YorkCrossRefGoogle Scholar
  23. Nottestad L., Axelsen B. E. (1999) Herring schooling manoeuvers in response to killer whale attack. Canadian Journal of Zoology 77: 1540–1546CrossRefGoogle Scholar
  24. Oreskes N., Shrader-Frechette K., Belitz K. (1994) Verification, validation, and confirmation of numerical models in the Earth sciences. Science 263: 641–646CrossRefGoogle Scholar
  25. Pierce, S., van Gieson, E. J., & Skalak, T. (2004). Multicellular simulation predicts microvascular patterning. The FASEB Journal, express article 10.1096/fj.03-0933fje. Retrieved December 13, 2010, from http://www.fasebj.org/content/early/2004/03/31/fj.03-0933fje.full.pdf.
  26. Randall D. A., Wielicki B. A. (1997) Measurements, models, and hypotheses in the atmospheric sciences. Bulletin of the American Meteorological Society 78: 399–406CrossRefGoogle Scholar
  27. Royall R. (1997) Statistical evidence: A likelihood paradigm. Chapman and Hall/CRC, New YorkGoogle Scholar
  28. Rykiel E. (1996) Testing ecological models: The meaning of validation. Ecological Modelling 90: 229–244CrossRefGoogle Scholar
  29. Satz D., Ferejohn J. (1994) Rational choice and social theory. Journal of Philosophy 91(2): 71–87CrossRefGoogle Scholar
  30. Sokal R. R., Rohlf F. J. (1994) Biometry. W.H. Freeman, New YorkGoogle Scholar
  31. van der Waals, J. (1910). The equation of state for gases and liquids (1910 Nobel Prize lecture).Google Scholar
  32. Woodford M. (2006) Comments on the symposium on interest and prices. Journal of the History of Economic Thought 28(2): 187–198CrossRefGoogle Scholar
  33. Worrall J. (2002) New evidence for old. In: Gardenfors P. (eds) In the scope of logic, methodology and philosophy of science. Kluwer, Dordrecht, pp 191–209Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.Philosophy DepartmentTufts UniversityMedfordUSA

Personalised recommendations