Ensembles of Recurrent Neural Networks for Robust Time Series Forecasting

  • Sascha KrstanovicEmail author
  • Heiko PaulheimEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10630)


Time series forecasting is a problem that is strongly dependent on the underlying process which generates the data sequence. Hence, finding good model fits often involves complex and time consuming tasks such as extensive data preprocessing, designing hybrid models, or heavy parameter optimization. Long Short-Term Memory (LSTM), a variant of recurrent neural networks (RNNs), provide state of the art forecasting performance without prior assumptions about the data distribution. LSTMs are, however, highly sensitive to the chosen network architecture and parameter selection, which makes it difficult to come up with a one-size-fits-all solution without sophisticated optimization and parameter tuning. To overcome these limitations, we propose an ensemble architecture that combines forecasts of a number of differently parameterized LSTMs to a robust final estimate which, on average, performs better than the majority of the individual LSTM base learners, and provides stable results across different datasets. The approach is easily parallelizable and we demonstrate its effectiveness on several real-world data sets.


Time series Ensemble Meta-learning Stacking ARIMA RNN LSTM 


  1. 1.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  2. 2.
    Tsukamoto, K., Mitsuishi, Y., and Sassano, M.: Learning with multiple stacking for named entity recognition. In: Proceedings of the 6th Conference on Natural Language Learning, vol. 20, pp. 1–4. Association for Computational Linguistics (2002)Google Scholar
  3. 3.
    Lai, K.K., Yu, L., Wang, S., Wei, H.: A novel nonlinear neural network ensemble model for financial time series forecasting. In: Alexandrov, V.N., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2006. LNCS, vol. 3991, pp. 790–793. Springer, Heidelberg (2006). CrossRefGoogle Scholar
  4. 4.
    Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989). Elsevier, AmsterdamCrossRefGoogle Scholar
  5. 5.
    Zhang, G.P.: Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50, 159–175 (2003). Elsevier, AmsterdamCrossRefzbMATHGoogle Scholar
  6. 6.
    Adhikari, R., Agrawal, R.K.: A linear hybrid methodology for improving accuracy of time series forecasting. Neural Comput. Appl. 25(2), 269–281 (2014). Springer, London, UKCrossRefGoogle Scholar
  7. 7.
    Adhikari, R.: A neural network based linear ensemble framework for time series forecasting. Neurocomputing 157, 231–242 (2015). Elsevier, AmsterdamCrossRefGoogle Scholar
  8. 8.
    Armstrong, J.S.: Combining forecasts. In: Armstrong, J.S. (ed.) Principles of Forecasting. ISOR, pp. 417–439. Springer, Boston (2001). CrossRefGoogle Scholar
  9. 9.
    Babu, C.N., Reddy, B.E.: A moving-average filter based hybrid ARIMA-ANN model for forecasting time series data. Appl. Soft Comput. 23, 27–38 (2014). Elsevier, AmsterdamCrossRefGoogle Scholar
  10. 10.
    Wang, L., Zou, H., Su, J., Li, L., Chaudhry, S.: An ARIMA-ANN hybrid model for time series forecasting. Syst. Res. Behav. Sci. 30(3), 244–259 (2013)CrossRefGoogle Scholar
  11. 11.
    Aladag, C.H., Egrioglu, E., Kadilar, C.: Forecasting nonlinear time series with a hybrid methodology. Appl. Math. Lett. 22(9), 1467–1470 (2009)CrossRefzbMATHGoogle Scholar
  12. 12.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). zbMATHGoogle Scholar
  13. 13.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)CrossRefGoogle Scholar
  14. 14.
    Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings of the 23rd European Symposium on Artificial Neural Networks. Computational Intelligence and Machine Learning, pp. 89–94. Presses universitaires de Louvain (2015)Google Scholar
  15. 15.
    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, vol. 28, pp. 1310–1318 (2013)Google Scholar
  16. 16.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  17. 17.
    Assaad, M., Boné, R., Cardot, H.: A new boosting algorithm for improved time-series forecasting with recurrent neural networks. Inf. Fusion 9(1), 41–55 (2008)CrossRefGoogle Scholar
  18. 18.
    Durbin, J., Koopman, S.J.: Time Series Analysis by State Space Methods, vol. 38. Oxford University Press, Oxford (2012)CrossRefzbMATHGoogle Scholar
  19. 19.
    Hamilton, J.D.: Time Series Analysis, vol. 2. Princeton University Press, Princeton (1994)zbMATHGoogle Scholar
  20. 20.
    Shumway, R.H., Stoffer, D.S.: Time Series Analysis and Its Applications: with R Examples. Springer Science & Business Media, Heidelberg (2010)zbMATHGoogle Scholar
  21. 21.
    Brockwell, P.J., Davis, R.A.: Introduction to Time Series and Forecasting, 2nd edn. Springer, New York (2010)zbMATHGoogle Scholar
  22. 22.
    Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw. Mach. Learn. 4(2), 26–31 (2012)Google Scholar
  24. 24.
    Lichman, M.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2013).
  25. 25.
    Cortez, P., Rio, M., Rocha, M., Sousa, P.: Multi-scale Internet traffic forecasting using neural networks and time series methods. Expert Syst. 29(2), 143–155 (2012)Google Scholar
  26. 26.
    Hipel, K.W., McLeod, A.I.: Time Series Modelling of Water Resources and Environmental Systems, vol. 45. Elsevier, Amsterdam (1994)CrossRefGoogle Scholar
  27. 27.
    Chollet, F.: Keras (2015).

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Research Group Data and Web ScienceUniversity of MannheimMannheimGermany

Personalised recommendations