Skip to main content
SpringerLink
Log in

Statistical Evaluation of Deep Learning Models for Stock Return Forecasting

  • Published:
Computational Economics Aims and scope Submit manuscript

Abstract

Artificial intelligence applications, including algorithmic training, portfolio allocation, and stock return forecasting in the financial industry, are rapidly developing research fields. Particularly, the forecasting of stock return has received considerable attention. Selecting the most promising model to forecast stock return has always been the most prominent task in the literature. Several studies have reported the prediction ability of recurrent neural network (RNN) models, whereas only a few studies have rigorously evaluated the prediction performance of temporal convolution networks (TCNs) for stock return forecasting settings. Moreover, although most studies are focused on comparing the performance of deep learning models at a single horizon forecasting, the multi-horizon forecasting applications of stock return have been studied only limitedly. In this study, we aim to evaluate the forecasting performance of state-of-the-art deep learning models at multi-horizon forecasting paths in stock return forecasting tasks. Specifically, we focus on TCNs, RNNs, long short-term memory, and gated recurrent unit models in terms of forecasting performance and direction accuracy. Our evaluation framework based on the model confidence set shows which model(s) is (are) better, with statistical significance, for forecasting stock return at multi-horizon forecasting. The empirical results assert that the TCN model has the best out-of-sample forecasting performance at all forecasting horizons on three financial datasets compared to other models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Numerous studies on modelling stock price volatility have been conducted; however, this discussion is beyond the scope of this study.

References

  • Abadi, M., Barham, P., & Chen, J. et al (2016). TensorFlow: A System for Large-Scale Machine Learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. USENIX Association, USA, OSDI’16, pp 265–283

  • Abdullah, MHL., & Ganapathy, V. (2000). Neural network ensemble for financial trend prediction. In: IEEE Region 10 Annual International Conference, Proceedings/TENCON

  • Abhyankar, A., Copeland, L. S., & Wong, W. (1997). Uncovering nonlinear structure in real-time stock-market indexes: The S & P 500, the DAX, the Nikkei 225, and the FTSE-100. Journal of Business and Economic Statistics, 15(1), 1–14.

    Google Scholar 

  • Adebiyi, AA., Adewumi, AO., & Ayo, CK. (2014). Stock price prediction using the ARIMA model. Proceedings - UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, UKSim p 106–112

  • Andersen, T. G., Bollerslev, T., & Meddahi, N. (2011). Realized volatility forecasting and market microstructure noise. Journal of Econometrics, 160(1), 220–234.

    Google Scholar 

  • Bai, S., Kolter, JZ., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. ArXiv abs/1803.0

  • Başoğlu Kabran, F., & Demirberk Ünlü, K. (2020). A two-step machine learning approach to predict S &P 500 bubbles. Journal of Applied Statistics

  • Ben Taieb, S., Sorjamaa, A., & Bontempi, G. (2010). Multiple-output modeling for multi-step-ahead time series forecasting. Neurocomputing, 73(10–12), 1950–1957.

    Google Scholar 

  • Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning Long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.

    Google Scholar 

  • Bernardi, M., & Catania, L. (2018). The model confidence set package for R. International Journal of Computational Economics and Econometrics, 8(2), 144–158.

    Google Scholar 

  • Borovykh, A., Bohte, S., & Oosterlee, CW. (2018). Dilated convolutional neural networks for time series forecasting. Journal of Computational Finance

  • Brandt, M. W., & Jones, C. S. (2006). Volatility forecasting with range-based EGARCH models. Journal of Business and Economic Statistics, 24(4), 470–486.

    Google Scholar 

  • Chen, A. S., Leung, M. T., & Daouk, H. (2003). Application of neural networks to an emerging financial market: Forecasting and trading the Taiwan stock index. Computers and Operations Research, 30(6), 901–923.

    Google Scholar 

  • Chollet, F., et al (2015). Keras. https://keras.io

  • Christoffersen, P. F., & Diebold, F. X. (2006). Financial asset returns, direction-of-change forecasting, and volatility dynamics. Management Science, 52(8), 1273–1287.

    Google Scholar 

  • Chu, W., & Ghahramani, Z. (2005). Gaussian processes for ordinal regression zoubin ghahramani. Journal of Machine Learning Research, 6, 1019–1041.

    Google Scholar 

  • Chung, J., Gulcehre, C., & Cho, K. et al (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv https://arxiv.org/abs/arXiv:1412.3555

  • Committee NP (2013). Understanding Asset Prices. Nobel Prize in Economics documents 2013-1, Nobel Prize Committee

  • Dang Khoa, NL., Sakakibara, K., & Nishikawa, I. (2006). Stock price forecasting using back propagation neural networks with time and profit based adjusted weight factors. In: 2006 SICE-ICASE International Joint Conference, pp 5484–5488

  • Dauphin, YN., Fan, A., & Auli, M. et al (2016). Language modeling with gated convolutional networks. CoRR abs/1612.0. https://arxiv.org/abs/arXiv:1612.08083

  • Fama, E. F. (1965). The behavior of Stock-market prices. The Journal of Business, 38(1), 34–105.

    Google Scholar 

  • Fama, E. F. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2), 383–417.

    Google Scholar 

  • Fama, E. F. (1991). Efficient capital markets: II. The Journal of Finance, 46(5), 1575–1617.

    Google Scholar 

  • Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654–669.

    Google Scholar 

  • Gallagher, L. A., & Taylor, M. P. (2002). Permanent and temporary components of stock prices: Evidence from assessing macroeconomic shocks. Southern Economic Journal, 69(2), 345.

    Google Scholar 

  • Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2451–2471.

    Google Scholar 

  • Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In: Teh YW, Titterington M (eds) Proceedings of the thirteenth international conference on artificial intelligence and statistics, proceedings of machine learning research, vol 9. PMLR, Chia Laguna Resort, Sardinia, Italy, pp 249–256

  • Hansen, P. R. (2005). A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4), 365–380.

    Google Scholar 

  • Hansen, P. R., Lunde, A., & Nason, J. M. (2011). The model confidence set. Econometrica, 79(2), 453–497.

    Google Scholar 

  • Hanson, J., & Raginsky, M. (2019). Universal approximation of input-output maps by temporal convolutional nets. ArXiv abs/1906.0. https://arxiv.org/abs/arXiv:1906.09211

  • He, K., Zhang, X., & Ren, S. et al (2016). Deep residual learning for Image Recognition. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 770–778

  • Hiransha, M., Gopalakrishnan, E. A., Menon, V. K., et al. (2018). NSE stock market prediction using Deep-learning models. Procedia Computer Science, 132, 1351–1362.

    Google Scholar 

  • Hochreiter, S., & Schmidhuber, J. (1997). Long Short-term memory. Neural Computation, 9(8), 1735–1780.

    Google Scholar 

  • Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen [in German]. Diploma thesis, TU Münich

  • Hosaka, T. (2019). Bankruptcy prediction using imaged financial ratios and convolutional neural networks. Expert Systems with Applications, 117, 287–299.

    Google Scholar 

  • Jasper, S., Hugo, L., & Ryan, A. (2012). Practical Bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, p 2951–2959

  • Jayawardena, N. I., Todorova, N., Li, B., et al. (2018). (2020) Volatility forecasting using related markets’ information for the Tokyo stock exchange. Economic Modelling, 90, 143–158.

    Google Scholar 

  • Jiang, Q., Tang, C., & Chen, C. et al (2019). Stock price forecast based on LSTM neural network. In: Xu J, Cooke FL, Gen M, et al (eds) Proceedings of the twelfth International Conference on management science and engineering management. Springer International Publishing, Cham, pp 393–408

  • Kaastra, I., & Boyd, M. (1996). Designing a neural network for forecasting financial and economic time series. Neurocomputing, 10(3), 215–236.

    Google Scholar 

  • Kang, Y., Chen, Y., Chen, Y., et al. (2020). Probabilistic forecasting with temporal convolutional neural network. Neurocomputing, 399, 491–501.

    Google Scholar 

  • Kim, S., Ku, S., Chang, W., et al. (2020). Predicting the direction of US stock prices using effective transfer entropy and machine learning techniques. IEEE Access, 8, 111660–111682.

    Google Scholar 

  • Kumar, G., Singh, UP., & Jain, S. (2021). Swarm intelligence based hybrid neural network approach for stock price forecasting. Computational Economics

  • Lara-Benítez, P., Carranza-García, M., Luna-Romera, J. M., et al. (2020). Temporal convolutional networks applied to energy-related time series forecasting. Applied Sciences (Switzerland), 10(7), 1–17.

    Google Scholar 

  • Li, Y., Yu, R., & Shahabi, C. et al (2017). Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv pp 1–16. https://arxiv.org/abs/arXiv:1707.01926

  • Malkiel, B. G. (2003). The efficient market hypothesis and its critics. Journal of Economic Perspectives, 17(1), 59–82.

    Google Scholar 

  • Marcellino, M., Stock, J. H., & Watson, M. W. (2006). A comparison of direct and iterated multistep AR methods for forecasting macroeconomic time series. Journal of Econometrics, 135(1–2), 499–526.

    Google Scholar 

  • Masum, S., Liu, Y., & Chiverton, J. (2018). Multi-step time series forecasting of electric load using machine learning models. In: Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 10841 LNAI. Springer Verlag, pp 148–159

  • McMillan, D. G. (2001). Nonlinear predictability of stock market returns: Evidence from nonparametric and threshold models. International Review of Economics and Finance, 10(4), 353–368.

    Google Scholar 

  • Moosa, I., & Vaz, J. (2015). Why is it so difficult to outperform the random walk? An application of the Meese-rogoff puzzle to stock prices. Applied Economics, 47(4), 398–407.

    Google Scholar 

  • Oh, K. J., & Kim, K. J. (2002). Analyzing stock market tick data using piecewise nonlinear model. Expert Systems with Applications, 22(3), 249–255.

    Google Scholar 

  • Ozbayoglu, A. M., Gudelek, M. U., & Sezer, O. B. (2020). Deep learning for financial applications : A survey. Applied Soft Computing, 93(106), 384.

    Google Scholar 

  • O’Malley, T., Bursztein, E., & Long, J. et al (2019). Keras Tuner. https://github.com/keras-team/keras-tuner

  • Panas, E. (2001). Estimating fractal dimension using stable distributions and exploring long memory through ARFIMA models in athens stock exchange. Applied Financial Economics, 11(4), 395–402.

    Google Scholar 

  • Parray, I. R., Khurana, S. S., Kumar, M., et al. (2020). Time series data analysis of stock price movement using machine learning techniques. Soft Computing, 24(21), 16509–16517.

    Google Scholar 

  • Perez-Quiros, G., & Timmermann, A. (2000). Firm size and cyclical variations in stock returns. Journal of Finance, 55(3), 1229–1262.

    Google Scholar 

  • Pesaran, M. H., & Timmermann, A. (1992). A simple nonparametric test of predictive performance. Journal of Business & Economic Statistics, 10(4), 461.

    Google Scholar 

  • Pérez-Rodríguez, J. V., Torra, S., & Andrada-Félix, J. (2005). STAR and ANN models: Forecasting performance on the Spanish “Ibex-35’’ stock index. Journal of Empirical Finance, 12(3), 490–509.

    Google Scholar 

  • Quaedvlieg, R. (2021). Multi-horizon forecast comparison. Journal of Business and Economic Statistics, 39(1), 40–53.

    Google Scholar 

  • Sarantis, N. (2001). Nonlinearities, cyclical behaviour and predictability in stock markets: International evidence. International Journal of Forecasting, 17(3), 459–482.

    Google Scholar 

  • Schnaubelt, M. (2019). A comparison of machine learning model validation schemes for non-stationary time series data. FAU discussion papers in economics 11/2019, Nürnberg

  • Schäfer, A. M., & Zimmermann, H. G., et al. (2006). Recurrent neural networks are universal approximators. In S. D. Kollias, A. Stafylopatis, & W. Duch (Eds.), Artificial Neural Networks - ICANN 2006 (pp. 632–640). Berlin Heidelberg, Berlin, Heidelberg: Springer.

    Google Scholar 

  • Selvin, S., Vinayakumar, R., & Gopalakrishnan, EA. et al (2017). Stock price prediction using LSTM, RNN and CNN-sliding window model. 2017 International conference on advances in computing, communications and informatics, ICACCI 2017 2017-Janua:1643–1647

  • Sezer, O. B., Gudelek, M. U., & Ozbayoglu, A. M. (2020). Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied Soft Computing Journal, 90(106), 181.

    Google Scholar 

  • Shen, G., Tan, Q., Zhang, H., et al. (2018). Deep learning with gated recurrent unit networks for financial sequence predictions. Procedia Computer Science, 131, 895–903.

    Google Scholar 

  • Timmermann, A., & Granger, C. W. (2004). Efficient market hypothesis and forecasting. International Journal of Forecasting, 20(1), 15–27.

    Google Scholar 

  • Tsantekidis, A., Passalis, N., & Tefas, A. et al (2017). Forecasting stock prices from the limit order book using convolutional neural networks. In: 2017 IEEE 19th Conference on business informatics (CBI), pp 7–12

  • Vaswani, A., Shazeer, N., & Parmar, N. et al (2017). Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, et al (eds) Advances in Neural Information Processing Systems 30. Curran Associates, Inc., p 5998–6008

  • Wan, R., Mei, S., Wang, J., et al. (2019). Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting. Electronics (Switzerland), 8(8), 876.

    Google Scholar 

  • Wang, Y., Liu, Z., Hu, D., et al. (2019). Multivariate time series prediction based on optimized temporal convolutional networks with stacked auto-encoders. Proceedings of machine learning research, 101, 157–172.

  • West, K. D. (1996). Asymptotic inference about predictive ability. Econometrica, 64(5), 1067–84.

    Google Scholar 

  • Yin, W., Kann, K., & Yu, M. et al (2017). Comparative study of CNN and RNN for natural language processing. ArXiv abs/1702.0

  • Yoon, Y., & Swales, G. (1991). Predicting stock price performance: A neural network approach. Proceedings of the annual hawaii international conference on system sciences, 4, 156–162.

  • Yu, P., & Yan, X. (2020). Stock price prediction based on deep neural networks. Neural Computing and Applications, 32(6), 1609–1628.

    Google Scholar 

  • Yu, F., & Koltun, V. (2016). Multi-scale context aggregation by dilated convolutions. ArXiv abs/1511.0

  • Zhang, J., Li, L., & Chen, W. (2021). Predicting stock price using Two-stage machine learning techniques. Computational Economics, 57, 1237–1261.

    Google Scholar 

  • Zhang, K., Thé, J., Xie, G., et al. (2020). Multi-step ahead forecasting of regional air quality using spatial-temporal deep neural networks: A case study of huaihai economic zone. Journal of Cleaner Production, 277(123), 231.

    Google Scholar 

Download references

Funding

No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. Department of Statistics, Dokuz Eylul University, Buca, Izmir, Türkiye

    Firat Melih Yilmaz & Engin Yildiztepe

Authors
  1. Firat Melih Yilmaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Engin Yildiztepe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Engin Yildiztepe.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yilmaz, F.M., Yildiztepe, E. Statistical Evaluation of Deep Learning Models for Stock Return Forecasting. Comput Econ 63, 221–244 (2024). https://doi.org/10.1007/s10614-022-10338-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10614-022-10338-3

Keywords

Search

Navigation