Abstract
The accuracy of price forecasts is important for financial market trading strategies and portfolio management. Compared to traditional models such as ARIMA and other state-of-the-art deep learning techniques, temporal Transformers with similarity embedding perform better for multi-horizon forecasts in financial time series, as they account for the conditional heteroscedasticity inherent in financial data. Despite this, the methods employed in generating these forecasts must be optimized to achieve the highest possible level of precision. One approach that has been shown to improve the accuracy of machine learning models is ensemble techniques. To this end, we present an ensemble approach that efficiently utilizes the available data over an extended timeframe. Our ensemble combines multiple temporal Transformer models learned within sliding windows, thereby making optimal use of the data. As combination methods, along with an averaging approach, we also introduced a stacking meta-learner that leverages a quantile estimator to determine the optimal weights for combining the base models of smaller windows. By decomposing the constituent time series of an extended timeframe, we optimize the utilization of the series for financial deep learning. This simplifies the training process of a temporal Transformer model over an extended time series while achieving better performance, particularly when accounting for the non-constant variance of financial time series. Our experiments, conducted across volatile and non-volatile extrapolation periods, using 20 companies from the Dow Jones Industrial Average show more than 40% and 60% improvement in predictive performance compared to the baseline temporal Transformer.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
No datasets were generated or analysed during the current study.
Notes
simfin.com
investopedia.com/terms/v/vix.asp
References
Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM
Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In: 11th Conference on Innovative Data Systems Research. www.cidrdb.org
Borovkova, S., & Tsiamas, I. (2019). An ensemble of LSTM neural networks for high-frequency stock market classification. 38(6)
Challu, C., Olivares, K.G., Oreshkin, B.N., Garza, F., Mergenthaler-Canseco, M., & Dubrawski, A. (2022). N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv
Chan, E.P.: Quantitative Trading: How to Build Your Own Algorithmic Trading Business, 2 edition edn. Wiley
Chan, E. P. (2016). Machine Trading: Deploying Computer Algorithms to Conquer the Markets, 1st (edition). Wiley.
Chen, A., Chow, A., Davidson, A., DCunha, A., Ghodsi, A., Hong, S. A., Konwinski, A., Mewald, C., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., Singh, A., Xie, F., Zaharia, M., Zang, R., Zheng, J., & Zumar, C. (2020). Developments in MLflow: A system to accelerate the machine learning lifecycle. ACM
Chong, L. S., Lim, K. M., & Lee, C. P. (2020). Stock market prediction using ensemble of deep neural networks. In: 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET). IEEE
Chu, J., Cao, J., & Chen, Y. (2022). An ensemble deep learning model based on transformers for long sequence time-series forecasting. In: Zhang, H., Chen, Y., Chu, X., Zhang, Z., Hao, T., Wu, Z., Yang, Y. (eds.) Neural Computing for Advanced Applications vol. 1638. Springer
Corizzo, R., & Rosen, J. (2023). Stock market prediction with time series data and news headlines: a stacking ensemble approach
Databricks. (2023). What Is a Medallion Architecture? https://www.databricks.com/glossary/medallion-architecture
Dong, X., Yu, Z., Cao, W., Shi, Y., & Ma, Q. (200). A survey on ensemble learning. 14(2)
Fort, S., Hu, H., & Lakshminarayanan, B. (2020). Deep Ensembles: A Loss Landscape Perspective. arXiv
Franses, P. H. (2016). A note on the mean absolute scaled error. 32(1)
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022). Ensemble deep learning: A review. 115
Goerg, S. J., & Kaiser, J. (2009). Nonparametric testing of distributions – the Epps-Singleton two-sample test using the empirical characteristic function. The Stata Journal. 9(3)
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. Cambridge, MA: The MIT Press.
He, K., Yang, Q., Ji, L., Pan, J., Zou, Y. (2023). Financial time series forecasting with the deep learning ensemble model. Mathematics. 11(4). https://doi.org/10.3390/math11041054
Hollander, M., Wolfe, D. A., & Chicken, E. (2013). Nonparametric Statistical Methods (3rd ed.). Wiley.
Hu, X. (2021). Stock price prediction based on temporal fusion transformer. In: 2021 3rd International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI)
Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting. 22(4).
Hyndman, R., & Athanasopoulos, G. (2021). Forecasting: Principles and Practice (3rd ed.). OTexts: Melbourne, Australia.
Leskovec, J., Rajaraman, A., & Ullman, J. D. (2020). Mining of Massive Datasets (3rd ed.). Cambridge University Press.
Li, Y., & Pan, Y. (2022). A novel ensemble deep learning model for stock prediction based on stock prices and news. 13(2)
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019). Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc.
Lim, B., Arik, S. O., Loeff, N., & Pfister, T. (2021). Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting. 37(4),
Lim, B., Zohren, S.: Time-series forecasting with deep learning: a survey. 379(2194)
Majiid, M. R. N., Fredyan, R., & Kusuma, G.P. (2023). Application of ensemble transformer-RNNs on stock price prediction of bank central asia. 11(2)
Mendes-Moreira, J., Soares, C., Jorge, A. M., & Sousa, J. F. D. (2012). Ensemble approaches for regression: A survey. 45(1)
Mustapa, F. H., & Ismail, M. T. (2019). Modelling and forecasting S &P 500 stock prices using hybrid arima-garch model. Journal of Physics. 1366
Olorunnimbe, K., & Viktor, H. L. (2022a). Deep learning in the stock market - a systematic survey of practice, backtesting and applications. Artificial Intelligence Review.
Olorunnimbe, K., & Viktor, H. L. (2023). Towards efficient similarity embedded temporal Transformers via extended timeframe analysis. Submitted to Complex & Intelligent Systems.
Olorunnimbe, K., Viktor, H.L. (2022). Similarity embedded temporal transformers: Enhancing stock predictions with historically similar trends. In: 26th International Symposium on Methodologies for Intelligent Systems (ISMIS)
Ong, E.-J., & Bober, M. (2016). Improved hamming distance search using variable length hashing. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE
Paquet, E., & Soleymani, F. (2022). QuantumLeap: Hybrid quantum neural network for financial predictions. Expert Systems with Applications. 195
Pradeepkumar, D., & Ravi, V. (2017). Forecasting financial time series volatility using particle swarm optimization trained quantile regression neural network. 58
Prado, M.L.d. Advances in Financial Machine Learning. Wiley
Prado, M. (2013). What to Look for in a Backtest.https://doi.org/10.2139/ssrn.2308682
Raghubir, P., & Das, S. R. (2010). The long and short of it: Why are stocks with shorter runs preferred? 36(6)
Reporting Standards and Availability of Data, Materials, Code and Protocols | Nature. ISSN: 1476-4687. https://www.nature.com/nature/editorial-policies/reporting-standards
Research Data Policy | Springer Nature. https://www.springernature.com/gp/authors/research-data-policy
Russell, S., & Norvig, P. (2016). Artificial Intelligence: A Modern Approach, Global Edition, 4th edn. Pearson
Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020). DeepAR: Probabilistic forecasting with autoregressive recurrent networks. 36(3)
Santana Correia, A., & Colombini, E. L. (2022). Attention, please! a survey of neural attention models in deep learning. Artificial Intelligence Review.
Sezer, O. B., Gudelek, M. U., & Ozbayoglu, A. M. (2020). Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied Soft Computing. 90
Soleymani, F., & Paquet, E. (2020). Financial portfolio optimization with online deep reinforcement learning and restricted stacked autoencoder–DeepBreath. Expert Systems with Applications. 156
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023). Efficient transformers: A survey. 55(6)
Taylor, J.W. (2000). A quantile regression neural network approach to estimating the conditional density of multiperiod returns. 19(4)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In: Advances in Neural Information Processing Systems
Wen, M., Li, P., Zhang, L., & Chen, Y. (2019). Stock market trend prediction using high-order information of time series. IEEE Access. 7
Wen, R., Torkkola, K., Narayanaswamy, B., Madeka, D. (2017). A multi-horizon quantile recurrent forecaster. In: NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc.
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., Sun, L. (2022). Transformers in Time Series: A Survey. arXiv
Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., et al. (2016). The FAIR guiding principles for scientific data management and stewardship. 3(1)
Yang, B., Gong, Z.- J., & Yang, W. (2017). Stock market index prediction using deep neural network ensemble. In: 2017 36th Chinese Control Conference (CCC). IEEE
Acknowledgements
This research was enabled in part by compute resources provided by Compute Ontario (Graham) https://www.computeontario.ca and the Digital Research Alliance of Canada https://www.alliancecan.ca.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
The authors contributed equally to this work.
Corresponding author
Ethics declarations
Ethical approval
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Olorunnimbe, K., Viktor, H. Ensemble of temporal Transformers for financial time series. J Intell Inf Syst (2024). https://doi.org/10.1007/s10844-024-00851-2
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10844-024-00851-2