Abstract
In finance, a persistent issue in classification task has been the long dependency. For example, how does a model learn from the abnormal London Interbank Offer Rate movement months before the 2008 financial crisis and apply that knowledge in today’s market? For traditional time series modeling and for basic deep learning model such as long short-term memory, such knowledge is hard for a model to capture. To ameliorate this problem, we present a novel architecture, residual attention net (RAN), which merges a sequence architecture, universal transformer, and a computer vision architecture, residual net, with a high-way architecture for cross-domain sequence modeling. This approach helps the model “remember” time series events that happened long before the target event. The architecture aims at addressing the long dependency issue often faced by recurrent neural net-based structures. This chapter serves as a proof-of-concept for a new architecture, with RAN aiming at providing the model a higher level understanding of sequence patterns. To our best knowledge, we are the first to propose such an architecture. Out of the standard 85 UCR data sets, we have achieved 35 state-of-the-art results with 10 results matching current state-of-the-art results without further model fine-tuning. The results indicate that such architecture is promising in complex, long-sequence modeling and may have vast, cross-domain applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahmed K, Keskar NS, Socher R (2017) Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132
Bai S, Zico Kolter J, Koltun V (2018) An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271
Bin Y, Yang Y, Shen F, Xie N, Shen HT, Li X (2018) Describing video with attention-based bidirectional LSTM. IEEE Trans Cybern 49(7):2631–2641
Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł (2018) Universal transformers. arXiv preprint arXiv:1807.03819
Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P (2019) Deep neural network ensembles for time series classification. arXiv preprint arXiv:1903.06602
Guo Q, Qiu X, Liu P, Shao Y, Xue X, Zhang Z (2019) Star-transformer. arXiv preprint arXiv:1902.09113
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Karim F, Majumdar S, Darabi H (2019) Insights into LSTM fully convolutional networks for time series classification. IEEE Access
Karim F, Majumdar S, Darabi H, Chen S (2017) LSTM fully convolutional networks for time series classification. IEEE Access 6:1662–1669
Karim F, Majumdar S, Darabi H, Harford S (2019) Multivariate LSTM-FCNS for time series classification. Neural Networks 116:237–245
Kim S, Kang M (2019) Financial series prediction using attention LSTM. arXiv preprint arXiv:1902.10877
Liu J, Wang G, Hu P, Duan L-Y, Kot AC (2017) Global context-aware attention LSTM networks for 3D action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1647–1656
Martin C (2019) Neural network architecture using control logic determining convolution operation sequence, May 16 2019. US Patent App. 16/182,369
Riou M, Jabaian B, Huet S, Lefèvre F (2019) Reinforcement adaptation of an attention-based neural natural language generator for spoken dialogue systems. Dialogue & Discourse 10(1):1–19
Seo Y, Defferrard M, Vandergheynst P, Bresson X (2018) Structured sequence modeling with graph convolutional recurrent networks. In: International conference on neural information processing. Springer, Berlin, pp 362–373
Shifaz A, Pelletier C, Petitjean F, Webb GI (2019) TS-chief: A scalable and accurate forest algorithm for time series classification. arXiv preprint arXiv:1906.10329
Song DR, Thiagarajan JJ, Spanias A (2018) Attend and diagnose: Clinical time series analysis using attention models. In: Thirty-Second AAAI conference on artificial intelligence
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Zeng B, Zhao D, Singh C, Wang J, Chen C (2019) Holistic modeling framework of demand response considering multi-timescale uncertainties for capacity value estimation. Appl Energy 247:692–702
Zhang K, Hao M, Wang J, de Silva CW, Fu C (2019) Linked dynamic graph CNN: Learning on point cloud via linking hierarchical features. arXiv preprint arXiv:1904.10014
Zhang Y, Shen T, Ji X, Zhang Y, Xiong R, Dai Q (2018) Residual highway convolutional neural networks for in-loop filtering in HEVC. IEEE Trans Image Process 27(8):3827–3841
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Huang, S.H., Xu, L., Jiang, C. (2021). Artificial Intelligence and Advanced Time Series Classification: Residual Attention Net for Cross-Domain Modeling. In: Choi, P.M.S., Huang, S.H. (eds) Fintech with Artificial Intelligence, Big Data, and Blockchain. Blockchain Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-33-6137-9_5
Download citation
DOI: https://doi.org/10.1007/978-981-33-6137-9_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-33-6136-2
Online ISBN: 978-981-33-6137-9
eBook Packages: Computer ScienceComputer Science (R0)