Advertisement

Some Pitfalls to Beware When Applying Models to Issues of Policy Relevance

Chapter
Part of the Understanding Complex Systems book series (UCS)

Abstract

This chapter looks at some of the ways things can go wrong when mathematical or computational models are applied to inform policy on important issues. It looks at some of the pitfalls in the model construction and development phase, including choosing assumptions, the effect of ‘theoretical spectacles’, oversimplified models, not understanding model limitations, and not testing a model enough. It then goes on to discuss the pitfalls that can occur when a model is applied to inform policy, including entrenched policies based on models with little or no evidential support and how models can narrow the evidential base considered. It also looks at confusions concerning model purpose and kinds of question they may answer, when models are used out of context, asking unreasonable things of models, when the uncertainties are too great, when models give a false sense of security, and when the focus should be on values rather than facts. This discussion is then illustrated with two examples, one economic and one from fisheries. It concludes that most of these problems stem from the interface between the modelling and policy worlds. It ends with some simple recommendations to reduce these mistakes.

Keywords

Policy Pitfalls Mental model Abstraction Formal model Transparency Political process Assumptions Reality Theoretical spectacles Oversimplification Model limitations Entrenched models Evidential support Evil models Model purpose Prediction Context of development Context of application Uncertainty False security Austerity policies Newfoundland cod 

References

  1. Bavington, D. (2009). Managing to endanger: Creating manageable cod fisheries in Newfoundland & Labrador, Canada. Maritime Studies (MAST), 7, 99–119.Google Scholar
  2. Bavington, D. (2010). Managed annihilation: An unnatural history of the Newfoundland cod collapse. UBC press.Google Scholar
  3. Bavington, D. (2015). Marine and freshwater fisheries in Canada: Uncertainties, conflicts, and hope on the water. In B. Mitchell (Ed.), Resource and environmental management in Canada. Canada: OUP.Google Scholar
  4. Benessia, A., Funtowicz, S., Giampietro, M., Guimarães Pereira, A., Ravetz, J., Saltelli, A., et al. (2016). Science on the Verge. Tempe, AZ: The Consortium for Science, Policy and Outcomes at Arizona State University.Google Scholar
  5. Black, F., & Scholes, M. (1973). The pricing of options and corporate liabilities. Journal of Political Economy, 81(3), 637–654.MathSciNetCrossRefMATHGoogle Scholar
  6. Bruff, I. (2016). Scandalous obfuscations of crisis in Europe. In: Progress in political economy (PPE). Available from: http://ppesydney.net/scandalous-obfuscations-of-crisis-in-europe/.
  7. Cavero, T., & Poinasamy, K. (2013). A cautionary tale: The true cost of austerity and inequality in Europe. Oxford: Oxfam International.Google Scholar
  8. Costello, C., Ovando, D., Clavelle, T., Strauss, C. K., Hilborn, R., Melnychuk, M. C., et al. (2016). Global fishery prospects under contrasting management regimes. Proceedings of the National Academy of Sciences, 113(18), 5125–5129.CrossRefGoogle Scholar
  9. Edmonds, B. (2017). Five different modelling purposes. doi:https://doi.org/10.1007/978-3-319-66948-9_4.CrossRefGoogle Scholar
  10. Finlayson, A. C. (1994). Fishing for truth: A sociological analysis of northern cod stock assessments from 1977 to 1990. St. John’s, Newfoundland: Institute of social and economic research, Memorial University of Newfoundland.Google Scholar
  11. Finley, C. (2008). A political history of maximum sustainable yield, 1945–1955. In D. Starkey, P. Holm, & M. Barnard (Eds.), Ocean's past: Management insights from the history of marine animal populations (pp. 1989–1206). London: Earthscan.Google Scholar
  12. Giampietro, M., & Saltelli, A. (2014). Footprints to nowhere. Ecological Indicators, 46, 610–621.CrossRefGoogle Scholar
  13. Glynn, P. D. (2015). Integrated environmental modelling: Human decisions, human challenges. Geological Society, London, Special Publications, 408(1), 161–182.CrossRefGoogle Scholar
  14. Harris, L. (1990). Independent review of the northern cod stock: Executive summary, and recommendations, Available at: http://www.dfo-mpo.gc.ca/Library/114277.pdf.
  15. Herndon, T., Ash, M., & Pollin, R. (2013). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2), 257–279.CrossRefGoogle Scholar
  16. Holbraad, M., Pedersen, M. A., & de Castro, E. V. (2014). The politics of ontology: Anthropological positions. Cultural Anthropology, 13. Available at: https://culanth.org/fieldsights/462-the-politics-of-ontology-anthropological-positions.
  17. Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago: University of Chicago Press.Google Scholar
  18. Moore, J. W. (2017). The Capitalocene, Part I: On the nature and origins of our ecological crisis. The Journal of Peasant Studies, 44(3), 594–630.CrossRefGoogle Scholar
  19. Pearce, W., Wesselink, A., & Colebatch, H. (2014). Evidence and meaning in policy making. Evidence & Policy, 10(2), 161–165.CrossRefGoogle Scholar
  20. Rogoff, K., & Reinhart, C. (2010). Growth in a time of debt. American Economic Review, 100(2), 573–578.CrossRefGoogle Scholar
  21. Rosewell, B. (2017). Complexity science and the art of policy making. In J. Johnson, P. Ormerod, B. Rosewell, A. Nowak, & Y. C. Zhang (Eds.), Non-equilibrium social science and policy (pp. 159–178). Switzerland: Springer.Google Scholar
  22. Saltelli, A., & Funtowicz, S. (2014). When all models are wrong. Issues in Science and Technology, 30(2), 79–85.Google Scholar
  23. Saltelli, A., Stark, P. B., Becker, W., & Stano, P. (2015). Climate models as economic guides scientific challenge or quixotic quest? Issues in Science and Technology, 31(3), 79–84.Google Scholar
  24. Saltelli, A., & Giampietro, M. (2017). What is wrong with evidence based policy, and how can it be improved? Futures, 91, 62–71, Available at: https://doi.org/10.1016/j.futures.2016.11.012.
  25. Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press.Google Scholar
  26. Sterman, J. D. (2002). All models are wrong: Reflections on becoming a systems scientist. System Dynamics Review, 18(4), 501–531.CrossRefGoogle Scholar
  27. Stern, N. (2016). Economics: Current climate models are grossly misleading. Nature, 530(7591), 407–409.CrossRefGoogle Scholar
  28. Stewart, I. (2012). The mathematical equation that caused the banks to crash. The Observer, 12 February 2012, https://www.theguardian.com/science/2012/feb/12/black-scholes-equation-credit-crunch. Accessed 01 June 2017.
  29. Stiglitz, J. E. (2011). Rethinking macroeconomics: What failed, and how to repair it. Journal of the European Economic Association, 9(4), 591–645.CrossRefGoogle Scholar
  30. Weinberg, A. M. (1972). Science and trans-science. Minerva, 10(2), 209–222.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Centre for Policy ModellingManchester Metropolitan UniversityManchesterUK

Personalised recommendations