Some Pitfalls to Beware When Applying Models to Issues of Policy Relevance
This chapter looks at some of the ways things can go wrong when mathematical or computational models are applied to inform policy on important issues. It looks at some of the pitfalls in the model construction and development phase, including choosing assumptions, the effect of ‘theoretical spectacles’, oversimplified models, not understanding model limitations, and not testing a model enough. It then goes on to discuss the pitfalls that can occur when a model is applied to inform policy, including entrenched policies based on models with little or no evidential support and how models can narrow the evidential base considered. It also looks at confusions concerning model purpose and kinds of question they may answer, when models are used out of context, asking unreasonable things of models, when the uncertainties are too great, when models give a false sense of security, and when the focus should be on values rather than facts. This discussion is then illustrated with two examples, one economic and one from fisheries. It concludes that most of these problems stem from the interface between the modelling and policy worlds. It ends with some simple recommendations to reduce these mistakes.
KeywordsPolicy Pitfalls Mental model Abstraction Formal model Transparency Political process Assumptions Reality Theoretical spectacles Oversimplification Model limitations Entrenched models Evidential support Evil models Model purpose Prediction Context of development Context of application Uncertainty False security Austerity policies Newfoundland cod
- Bavington, D. (2009). Managing to endanger: Creating manageable cod fisheries in Newfoundland & Labrador, Canada. Maritime Studies (MAST), 7, 99–119.Google Scholar
- Bavington, D. (2010). Managed annihilation: An unnatural history of the Newfoundland cod collapse. UBC press.Google Scholar
- Bavington, D. (2015). Marine and freshwater fisheries in Canada: Uncertainties, conflicts, and hope on the water. In B. Mitchell (Ed.), Resource and environmental management in Canada. Canada: OUP.Google Scholar
- Benessia, A., Funtowicz, S., Giampietro, M., Guimarães Pereira, A., Ravetz, J., Saltelli, A., et al. (2016). Science on the Verge. Tempe, AZ: The Consortium for Science, Policy and Outcomes at Arizona State University.Google Scholar
- Bruff, I. (2016). Scandalous obfuscations of crisis in Europe. In: Progress in political economy (PPE). Available from: http://ppesydney.net/scandalous-obfuscations-of-crisis-in-europe/.
- Cavero, T., & Poinasamy, K. (2013). A cautionary tale: The true cost of austerity and inequality in Europe. Oxford: Oxfam International.Google Scholar
- Finlayson, A. C. (1994). Fishing for truth: A sociological analysis of northern cod stock assessments from 1977 to 1990. St. John’s, Newfoundland: Institute of social and economic research, Memorial University of Newfoundland.Google Scholar
- Finley, C. (2008). A political history of maximum sustainable yield, 1945–1955. In D. Starkey, P. Holm, & M. Barnard (Eds.), Ocean's past: Management insights from the history of marine animal populations (pp. 1989–1206). London: Earthscan.Google Scholar
- Harris, L. (1990). Independent review of the northern cod stock: Executive summary, and recommendations, Available at: http://www.dfo-mpo.gc.ca/Library/114277.pdf.
- Holbraad, M., Pedersen, M. A., & de Castro, E. V. (2014). The politics of ontology: Anthropological positions. Cultural Anthropology, 13. Available at: https://culanth.org/fieldsights/462-the-politics-of-ontology-anthropological-positions.
- Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago: University of Chicago Press.Google Scholar
- Rosewell, B. (2017). Complexity science and the art of policy making. In J. Johnson, P. Ormerod, B. Rosewell, A. Nowak, & Y. C. Zhang (Eds.), Non-equilibrium social science and policy (pp. 159–178). Switzerland: Springer.Google Scholar
- Saltelli, A., & Funtowicz, S. (2014). When all models are wrong. Issues in Science and Technology, 30(2), 79–85.Google Scholar
- Saltelli, A., Stark, P. B., Becker, W., & Stano, P. (2015). Climate models as economic guides scientific challenge or quixotic quest? Issues in Science and Technology, 31(3), 79–84.Google Scholar
- Saltelli, A., & Giampietro, M. (2017). What is wrong with evidence based policy, and how can it be improved? Futures, 91, 62–71, Available at: https://doi.org/10.1016/j.futures.2016.11.012.
- Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press.Google Scholar
- Stewart, I. (2012). The mathematical equation that caused the banks to crash. The Observer, 12 February 2012, https://www.theguardian.com/science/2012/feb/12/black-scholes-equation-credit-crunch. Accessed 01 June 2017.