Machine Learning Strategies for Time Series Forecasting

  • Gianluca Bontempi
  • Souhaib Ben Taieb
  • Yann-Aël Le Borgne

Abstract

The increasing availability of large amounts of historical data and the need of performing accurate forecasting of future behavior in several scientific and applied domains demands the definition of robust and efficient techniques able to infer from observations the stochastic dependency between past and future. The forecasting domain has been influenced, from the 1960s on, by linear statistical methods such as ARIMA models. More recently, machine learning models have drawn attention and have established themselves as serious contenders to classical statistical models in the forecasting community. This chapter presents an overview of machine learning techniques in time series forecasting by focusing on three aspects: the formalization of one-step forecasting problems as supervised learning tasks, the discussion of local learning techniques as an effective tool for dealing with temporal data and the role of the forecasting strategy when we move from one-step to multiple-step forecasting.

Keywords

Time series forecasting machine learning local learning lazy learning MIMO 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ahmed, N.K., Atiya, A.F., El Gayar, N., El-Shishiny, H.: An empirical comparison of machine learning models for time series forecasting. Econometric Reviews 29(5-6) (2010)Google Scholar
  2. 2.
    Allen, D.M.: The relationship between variable selection and data agumentation and a method for prediction. Technometrics 16(1), 125–127 (1974)CrossRefGoogle Scholar
  3. 3.
    Alpaydin, E.: Introduction to Machine Learning, 2nd edn. Adaptive Computation and Machine Learning. The MIT Press (February 2010)Google Scholar
  4. 4.
    Anderson, T.W.: The statistical analysis of time series. J. Wiley and Sons (1971)Google Scholar
  5. 5.
    Atkeson, C.G., Moore, A.W., Schaal, S.: Locally weighted learning. AIR 11(1-5), 11–73 (1997)Google Scholar
  6. 6.
    Ben Taieb, S., Bontempi, G.: Recursive multi-step time series forecasting by perturbing data. In: Proceedings of IEEE-ICDM 2011(2011)Google Scholar
  7. 7.
    Ben Taieb, S., Bontempi, G., Atiya, A., Sorjamaa, A.: A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition. ArXiv e-prints (August 2011)Google Scholar
  8. 8.
    Ben Taieb, S., Bontempi, G., Sorjamaa, A., Lendasse, A.: Long-term prediction of time series by combining direct and mimo strategies. In: Proceedings of the 2009 IEEE International Joint Conference on Neural Networks, Atlanta, U.S.A., pp. 3054–3061 (June 2009)Google Scholar
  9. 9.
    Ben Taieb, S., Sorjamaa, A., Bontempi, G.: Multiple-output modelling for multi-step-ahead forecasting. Neurocomputing 73, 1950–1957 (2010)CrossRefGoogle Scholar
  10. 10.
    Ben Taieb, S., Bontempi, G., Sorjamaa, A., Lendasse, A.: Long-term prediction of time series by combining direct and mimo strategies. In: International Joint Conference on Neural Networks (2009)Google Scholar
  11. 11.
    Birattari, M., Bontempi, G., Bersini, H.: Lazy learning meets the recursive least-squares algorithm. In: Kearns, M.S., Solla, S.A., Cohn, D.A. (eds.) NIPS 11, pp. 375–381. MIT Press, Cambridge (1999)Google Scholar
  12. 12.
    Bontempi, G.: Local Learning Techniques for Modeling, Prediction and Control. PhD thesis, IRIDIA- Université Libre de Bruxelles (1999)Google Scholar
  13. 13.
    Bontempi, G.: Long term time series prediction with multi-input multi-output local learning. In: Proceedings of the 2nd European Symposium on Time Series Prediction (TSP), ESTSP 2008, Helsinki, Finland, pp. 145–154 (February 2008)Google Scholar
  14. 14.
    Bontempi, G., Birattari, M., Bersini, H.: Lazy learners at work: the lazy learning toolbox. In: Proceeding of the 7th European Congress on Intelligent Techniques and Soft Computing, EUFIT 1999 (1999)Google Scholar
  15. 15.
    Bontempi, G., Birattari, M., Bersini, H.: Local learning for iterated time-series prediction. In: Bratko, I., Dzeroski, S. (eds.) Machine Learning: Proceedings of the Sixteenth International Conference, pp. 32–38. Morgan Kaufmann Publishers, San Francisco (1999)Google Scholar
  16. 16.
    Bontempi, G., Ben Taieb, S.: Conditionally dependent strategies for multiple-step-ahead prediction in local learning. International Journal of Forecasting (2011) (in press, corrected proof)Google Scholar
  17. 17.
    Casdagli, M., Eubank, S., Farmer, J.D., Gibson, J.: State space reconstruction in the presence of noise. PHYD 51, 52–98 (1991)Google Scholar
  18. 18.
    Cheng, H., Tan, P.-N., Gao, J., Scripps, J.: Multistep-Ahead Time Series Prediction. In: Ng, W.-K., Kitsuregawa, M., Li, J., Chang, K. (eds.) PAKDD 2006. LNCS (LNAI), vol. 3918, pp. 765–774. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Crone, S.F.: NN3 Forecasting Competition, http://www.neural-forecasting-competition.com/NN3/index.html (last update May 26, 2009) (visited on July 05, 2010)
  20. 20.
    Crone, S.F.: NN5 Forecasting Competition, http://www.neural-forecasting-competition.com/NN5/index.html (last update May 27, 2009) (visited on July 05, 2010)
  21. 21.
    Crone, S.F.: Mining the past to determine the future: Comments. International Journal of Forecasting 5(3), 456–460 (2009); Special Section: Time Series MonitoringCrossRefGoogle Scholar
  22. 22.
    Engle, R.F.: Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica 50(4), 987–1007 (1982)CrossRefGoogle Scholar
  23. 23.
    Farmer, J.D., Sidorowich, J.J.: Predicting chaotic time series. Physical Review Letters 8(59), 845–848 (1987)CrossRefGoogle Scholar
  24. 24.
    Farmer, J.D., Sidorowich, J.J.: Exploiting chaos to predict the future and reduce noise. Technical report, Los Alamos National Laboratory (1988)Google Scholar
  25. 25.
    De Gooijer, J.G., Hyndman, R.J.: 25 years of time series forecasting. International Journal of Forecasting 22(3), 443–473 (2006)CrossRefGoogle Scholar
  26. 26.
    De Gooijer, J.G., Kumar, K.: Some recent developments in non-linear time series modelling, testing, and forecasting. International Journal of Forecasting 8(2), 135–156 (1992)CrossRefGoogle Scholar
  27. 27.
    Guo, M., Bai, Z., An, H.Z.: Multi-step prediction for nonlinear autoregressive models based on empirical distributions. In: Statistica Sinica, pp. 559–570 (1999)Google Scholar
  28. 28.
    Hand, D.: Mining the past to determine the future: Problems and possibilities. International Journal of Forecasting (October 2008)Google Scholar
  29. 29.
    Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning: data mining, inference and prediction, 2nd edn. Springer (2009)Google Scholar
  30. 30.
    Hsu, W., Lee, M.L., Wang, J.: Temporal and spatio-temporal data mining. IGI Pub. (2008)Google Scholar
  31. 31.
    Ikeguchi, T., Aihara, K.: Prediction of chaotic time series with noise. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E78-A(10) (1995)Google Scholar
  32. 32.
    Kline, D.M.: Methods for multi-step time series forecasting with neural networks. In: Peter Zhang, G. (ed.) Neural Networks in Business Forecasting, pp. 226–250. Information Science Publishing (2004)Google Scholar
  33. 33.
    Lapedes, A., Farber, R.: Nonlinear signal processing using neural networks: prediction and system modelling. Technical Report LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, NM (1987)Google Scholar
  34. 34.
    Lendasse, A. (ed.): ESTSP 2007: Proceedings (2007)Google Scholar
  35. 35.
    Lendasse, A. (ed.): ESTSP 2008: Proceedings. Multiprint Oy/Otamedia (2008) ISBN: 978-951-22-9544-9Google Scholar
  36. 36.
    Lorenz, E.N.: Atmospheric predictability as revealed by naturally occurring analogues. Journal of the Atmospheric Sciences 26, 636–646 (1969)CrossRefGoogle Scholar
  37. 37.
    Matías, J.M.: Multi-output Nonparametric Regression. In: Bento, C., Cardoso, A., Dias, G. (eds.) EPIA 2005. LNCS (LNAI), vol. 3808, pp. 288–292. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  38. 38.
    McNames, J.: A nearest trajectory strategy for time series prediction. In: Proceedings of the International Workshop on Advanced Black-Box Techniques for Nonlinear Modeling, pp. 112–128. K.U. Leuven, Belgium (1998)Google Scholar
  39. 39.
    Micchelli, C.A., Pontil, M.A.: On learning vector-valued functions. Neural Comput. 17(1), 177–204 (2005)CrossRefGoogle Scholar
  40. 40.
    Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)Google Scholar
  41. 41.
    Owen, S.: Mahout in action. Manning (2012)Google Scholar
  42. 42.
    Packard, N.H., Crutchfeld, J.P., Farmer, J.D., Shaw, R.S.: Geometry from a time series. Physical Review Letters 45(9), 712–716 (1980)CrossRefGoogle Scholar
  43. 43.
    Palit, A.K., Popovic, D.: Computational Intelligence in Time Series Forecasting: Theory and Engineering Applications. Advances in Industrial Control. Springer-Verlag New York, Inc., Secaucus (2005)Google Scholar
  44. 44.
    Poskitt, D.S., Tremayne, A.R.: The selection and use of linear and bilinear time series models. International Journal of Forecasting 2(1), 101–114 (1986)CrossRefGoogle Scholar
  45. 45.
    Price, S.: Mining the past to determine the future: Comments. International Journal of Forecasting 25(3), 452–455 (2009)CrossRefGoogle Scholar
  46. 46.
    Priestley, M.B.: Non-linear and Non-stationary time series analysis. Academic Press (1988)Google Scholar
  47. 47.
    Saad, E., Prokhorov, D., Wunsch, D.: Comparative study of stock trend prediction using time delay, recurrent and probabilistic neural networks. IEEE Transactions on Neural Networks 9(6), 1456–1470 (1998)CrossRefGoogle Scholar
  48. 48.
    Schuster, H.G.: Deterministic Chaos: An Introduction. Weinheim Physik (1988)Google Scholar
  49. 49.
    Sorjamaa, A., Hao, J., Reyhani, N., Ji, Y., Lendasse, A.: Methodology for long-term prediction of time series. Neurocomputing 70(16-18), 2861–2869 (2007)CrossRefGoogle Scholar
  50. 50.
    Sorjamaa, A., Lendasse, A.: Time series prediction using dirrec strategy. In: Verleysen, M. (ed.) European Symposium on Artificial Neural Networks, ESANN 2006, Bruges, Belgium, April 26-28, pp. 143–148 (2006)Google Scholar
  51. 51.
    Sorjamaa, A., Lendasse, A., Verleysen, M.: Pruned lazy learning models for time series prediction. In: European Symposium on Artificial Neural Networks, ESANN 2005, pp. 509–514 (2005)Google Scholar
  52. 52.
    Takens, F.: Detecting strange attractors in fluid turbulence. In: Dynamical Systems and Turbulence. Springer, Berlin (1981)Google Scholar
  53. 53.
    Tiao, G.C., Tsay, R.S.: Some advances in non-linear and adaptive modelling in time-series. Journal of Forecasting 13(2), 109–131 (1994)CrossRefGoogle Scholar
  54. 54.
    Tong, H.: Threshold models in Nonlinear Time Series Analysis. Springer, Berlin (1983)CrossRefGoogle Scholar
  55. 55.
    Tong, H.: Non-linear Time Series: A Dynamical System Approach. Oxford University Press (1990)Google Scholar
  56. 56.
    Tong, H., Lim, K.S.: Thresold autoregression, limit cycles and cyclical data. JRSS_B 42, 245–292 (1980)Google Scholar
  57. 57.
    Tran, T.V., Yang, B.-S., Tan, A.C.C.: Multi-step ahead direct prediction for the machine condition prognosis using regression trees and neuro-fuzzy systems. Expert Syst. Appl. 36(5), 9378–9387 (2009)CrossRefGoogle Scholar
  58. 58.
    Weigend, A.S., Gershenfeld, N.A.: Time Series Prediction: forecasting the future and understanding the past. Addison Wesley, Harlow (1994)Google Scholar
  59. 59.
    Werbos, P.J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, Cambridge, MA (1974)Google Scholar
  60. 60.
    Werbos, P.J.: Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1(4), 339–356 (1988)CrossRefGoogle Scholar
  61. 61.
    Zhang, G., Eddy Patuwo, B., Hu, M.Y.: Forecasting with artificial neural networks: The state of the art. International Journal of Forecasting 14(1), 35–62 (1998)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Gianluca Bontempi
    • 1
  • Souhaib Ben Taieb
    • 1
  • Yann-Aël Le Borgne
    • 1
  1. 1.Machine Learning Group, Computer Science Department, Faculty of SciencesULB, Université Libre de BruxellesBrusselsBelgium

Personalised recommendations