Forecasting in Light of Big Data
- 536 Downloads
Predicting the future state of a system has always been a natural motivation for science and practical applications. Such a topic, beyond its obvious technical and societal relevance, is also interesting from a conceptual point of view. This owes to the fact that forecasting lends itself to two equally radical, yet opposite methodologies. A reductionist one, based on first principles, and the naïve-inductivist one, based only on data. This latter view has recently gained some attention in response to the availability of unprecedented amounts of data and increasingly sophisticated algorithmic analytic techniques. The purpose of this note is to assess critically the role of big data in reshaping the key aspects of forecasting and in particular the claim that bigger data leads to better predictions. Drawing on the representative example of weather forecasts we argue that this is not generally the case. We conclude by suggesting that a clever and context-dependent compromise between modelling and quantitative analysis stands out as the best forecasting strategy, as anticipated nearly a century ago by Richardson and von Neumann.
KeywordsForecasting Big data Epistemology
- Collet, P., & Eckmann, J.-P. (2006). Concepts and results in chaotic dynamics: A short course. Berlin: Springer.Google Scholar
- Coveney, P.V., Dougherty, E.R., & Highfield, R.R. (2016). Big data need big theory too. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 280(374), 1–11.Google Scholar
- de Finetti, B. (1974). Theory of probability Vol. 1. New York: Wiley.Google Scholar
- de Finetti, B. (2008). Philosophical lectures on probability In A. Mura (Ed.), Translated by H. Hosni. Berlin: Springer.Google Scholar
- Domingos, P. (2015). The master algorithm: How the quest for the ultimate learning machine will remake our world. New York: Basic Books.Google Scholar
- Halmos, P. R. (1956). Lectures on Ergodic Theory. London: Chelsea Publishing.Google Scholar
- Lorenz, E. N. (1996). Predictability- A problem partly solved, Proceedings of the Seminar on Predictability (pp. 1–18). Reading: ECMWF.Google Scholar
- Lynch, P. (2006). The Emergence of Numerical Weather Prediction: Richardson’s Dream. Cambridge: Cambridge University Press.Google Scholar
- Mayer-Schönberger, V., & Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think, (p. 2013). New York: Houghton Mifflin.Google Scholar
- Nowotny, E. (2016). The Cunning of Uncertainty. London: Polity.Google Scholar
- Nural, M. , Cotterell, M.E., & Miller, J. (2015). Using Semantics in Predictive Big Data Analytics, Proceedings - 2015 IEEE International Congress on Big Data, BigData Congress, (Vol. 2015 pp. 254–261).Google Scholar
- Pasquale, F. (2015). The Black Box Society Vol. 36. Harvard: Harvard University Press.Google Scholar
- Perry, W.L., McInnes, B., Price, C.C., Smith, S.C., & Hollywood, J.S. (2013). Predictive Policing: The role of crime forecasting in law enforcement operations. RAND Corporation, Santa Monica.Google Scholar
- Poincaré, H. (1890). Sur le problème des trois corps et les équations de la dynamique. Acta Mathematica, 13, 1–270.Google Scholar
- Richardson, L.F. (1922). Weather Prediction by Numerical Methods. Cambridge: Cambridge University Press.Google Scholar
- Robbins, M. (2016). Has a rampaging AI algorithm really killed thousands in Pakistan? The Guardian. http://www.theguardian.com/science/the-lay-scientist/2016/feb/18/has-a-rampaging-ai-algorithm-really-killed-thousands-in-pakistan .
- SKYNET (2005). Applying Advanced Cloud-based Behavior Analytics. The Intercept. https://theintercept.com/document/2015/05/08/skynet-applying-advanced-cloud-based-behavior-analytics/.
- Takens, F. (1981). Detecting strange attractors in turbulence. In Rand, D., & Young, L.-S. (Eds.), Dynamical Systems and Turbulence, Lecture Notes in Mathematics, (Vol. 898 pp. 366–381).Google Scholar
- Weigend, A.S., & Gershenfeld, N.A. (Eds.) (1994). Time Series Prediction: Forecasting the Future and Understanding the Past Addison-Wesley, Reading.Google Scholar