Skip to main content
SpringerLink
Log in

A New MC-LSTM Network Structure Designed for Regression Prediction of Time Series

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Long short-term memory (LSTM) is regarded as one of the most popular methods for regression prediction of time series. In the memory unit of LSTM, since most values of gate structures are usually in the middle state (around 0.5), gate structures cannot effectively retain important information or discard trivial information. Furthermore, the information between two adjacent layers can not be sufficiently transmitted only through the hidden state. To address these issues, we propose a new LSTM structure based on memory cell (MC-LSTM for short) in this paper. First, a new gate stretching mechanism in memory unit is introduced to readjust the distributions of the gates values to push them away from the uncertainty of 0.5. Second, before the memory unit, we establish an interaction gate, in which the input information, the hidden state and the output memory cell of the previous layer interact with each other. By doing so, the information fusion between two adjacent layers can be enhanced and the long-term dependencies can be captured effectively as well. This new method can be used to process time series data, and its goal is to use historical data to predict the value of a future time period. Experimental results on one UCI dataset and eight Kaggle time series datasets validate that the proposed network structure is superior to the most advanced networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Gers FA, Eck D, Schmidhuber J (2002) Applying LSTM to time series predictable through time-window approaches. In: Artificial Neural Networks—ICANN 2001: International Conference Vienna, Austria, August 21–25, 2001 Proceedings, vol 11, pp 669–676

  2. Qu CX, Yang HF, Cai JH, Xun YL (2020) P-cygni profile analysis of the spectrum: Lamost j152238.11+333136.1. Spectrosc Spect Anal 40:1304

    Google Scholar 

  3. Tukymbekov D, Saymbetov A, Nurgaliyev M, Kuttybay N, Dosymbetova G, Svanbayev Y (2021) Intelligent autonomous street lighting system based on weather forecast using lstm. Energy 231:120902

    Article  Google Scholar 

  4. Xu Y, Lu X, Cetiner B, Taciroglu E (2020) Real time regional seismic damage assessment framework based on long short term memory neural network. Comput Aided Civ Infrast Eng 36(4):504–521

  5. Mikolov T, Karafiát M, Burget L, Cernocký J, Khudanpur S (2010) Recurrent neural network based language model. Interspeech 2(3):1045–1048

  6. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780

    Article  Google Scholar 

  7. Huang X, Ye Y, Wang C, Yang X, Xiong L (2021) A multi-mode traffic flow prediction method with clustering based attention convolution lstm. Appl Intell 1–14

  8. Niu H, Xu K, Wang W (2020) A hybrid stock price index forecasting model based on variational mode decomposition and lstm network. Appl Intell 50:4296–4309

    Article  Google Scholar 

  9. Chowdhury SA, Zamparelli R (2018) RNN simulations of grammaticality judgments on long-distance dependencies. In: Proceedings of the 27th international conference on computational linguistics, pp 133–144

  10. Wang Q, Downey C, Wan L, Mansfield PA, Moreno IL (2018) Speaker Diarization with LSTM. In: 2018 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 5239–5243

  11. Lou Y, Caruana R, Gehrke J (2012) Intelligible models for classification and regression. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 150–158

  12. Yu Y, Si X, Hu C, Zhang J (2019) A review of recurrent neural networks: Lstm cells and network architectures. Neural Comput 31:1235–1270

    Article  MathSciNet  MATH  Google Scholar 

  13. Li Z, He D, Tian F, Chen W, Qin T, Wang L, Liu T (2018) Towards binary-valued gates for robust lstm training, pp 2995–3004

  14. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  Google Scholar 

  15. Hinton GE, Osindero S, Teh YW (2014) A fast learning algorithm for deep belief nets. Neural Comput 18:1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  16. Dey R, Salem FM (2017) Gate-variants of Gated Recurrent Unit (GRU) neural networks. In: 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp 1597–1600

  17. Radojičić D, Kredatus S (2020) The impact of stock market price fourier transform analysis on the gated recurrent unit classifier model. Expert Syst Appl 159:113565

    Article  Google Scholar 

  18. Qing X, Niu Y (2018) Hourly day-ahead solar irradiance prediction using weather forecasts by lstm. Energy 148:461–468

    Article  Google Scholar 

  19. Hochreiter S (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. Internat J Uncertain Fuzziness Knowl-Based Syst 6:107–116

    Article  MATH  Google Scholar 

  20. Kawakami K (2008) Supervised sequence labelling with recurrent neural networks. Ph. D. thesis

  21. Sundermeyer M, Schlüter R, Ney H (2012) LSTM neural networks for language modeling. In: Thirteenth annual conference of the international speech communication association

  22. Poornima S, Pushpalatha M (2019) Prediction of rainfall using intensified lstm based recurrent neural network with weighted linear units. Atmosphere 10:668

    Article  Google Scholar 

  23. Thasleema TM, Kabeer V, Narayanan NK (2007) Malayalam vowel recognition based on linear predictive coding parameters and k-nn algorithm. In: International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007), vol 2, pp 361–365

  24. Yalamarthy KP, Dhall S, Khan MT, Shaik RA (2020) Low-complexity distributed-arithmetic-based pipelined architecture for an lstm network. IEEE Trans Very Large Scale Integr VLSI Syst 28:329–338

    Article  Google Scholar 

  25. Liu H, Mi X, Li Y (2018) Smart multi-step deep learning model for wind speed forecasting based on variational mode decomposition, singular spectrum analysis, lstm network and elm. Energy Convers Manage 159:54–64

    Article  Google Scholar 

  26. Chen J, Zeng GQ, Zhou W, Du W, Lu KD (2018) Wind speed forecasting using nonlinear-learning ensemble of deep learning time series prediction and extremal optimization - sciencedirect. Energy Convers Manage 165:681–695

    Article  Google Scholar 

  27. Choi JW, Ryu JHSJ (2019) andKim: Short-range radar based real-time hand gesture recognition using lstm encoder. IEEE Access 7:33610–33618

    Article  Google Scholar 

  28. Muzaffar S, Afshari A (2019) Short-term load forecasts using lstm networks. Energy Procedia 158:2922–2927

    Article  Google Scholar 

  29. Duan Y, Yisheng LV, Wang FY (2016) Travel time prediction with LSTM neural network. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC), pp 1053–1058

  30. Yan R, Liao J, Yang J, Sun W, Nong M, Li F (2020) Multi-hour and multi-site air quality index forecasting in beijing using cnn, lstm, cnn-lstm, and spatiotemporal clustering. Expert Syst Appl 169:114513

    Article  Google Scholar 

  31. Gers FA, Schmidhuber J, Cummins F (2000) Learning to forget: continual prediction with lstm. Neural Comput 12(10): 2451–2471

  32. Gers FA, Schmidhuber J (2000) Recurrent nets that time and count. In: Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, vol 3, pp 189–194

  33. Wu Z, King S (2016) Investigating gated recurrent networks for speech synthesis. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 5140–5144

  34. Zhou GB, Wu J, Zhang CL, Zhou ZH (2016) Minimal gated unit for recurrent neural networks. Int J Autom Comput 13:226–234

    Article  Google Scholar 

  35. Mirza AH, Kerpicci M, Kozat SS (2020) Efficient online learning with improved lstm neural networks. Digit Signal Process 102:102742

    Article  Google Scholar 

  36. Krause B, Lu L, Murray I, Renals S (2016) Multiplicative lstm for sequence modelling. arXiv preprint arXiv:1609.07959

  37. Melis G, Kočiský T, Blunsom P (2019) Mogrifier lstm. arXiv preprint arXiv:1909.01792

  38. Yu R, Gao J, Yu M, Lu W, Xu T, Zhao M, Zhang Z (2018) Lstm-efg for wind power forecasting based on sequential correlation features. Future Gener Comput Syst 93:33–42

  39. Levy O, Lee K, FitzGerald N, Zettlemoyer L (2018) Long short-term memory as a dynamically computed element-wise weighted sum. arXiv preprint arXiv:1805.03716

  40. Yao K, Cohn T, Vylomova K, Duh K, Dyer C (2015) Depth-gated lstm. arXiv preprint arXiv:1508.03790

  41. Bouktif S, Fiaz A, Ouni A, Serhani MA (2018) Optimal deep learning lstm model for electric load forecasting using feature selection and genetic algorithm: Comparison with machine learning approaches. Energies 11:1636

    Article  Google Scholar 

  42. Cai J, Yang Y, Yang H, Zhao X, Hao J (2022) Aris: a noise insensitive data pre-processing scheme for data reduction using influence space. ACM Trans Knowl Discov Data (TKDD) 6(16):1–39

  43. Tamura S (1989) An analysis of a noise reduction neural network. In: International Conference on Acoustics, Speech, and Signal Processing, pp 2001–2004

  44. Yang Y, Cai J, Yang H, Zhao X (2022) Density clustering with divergence distance and automatic center selection. Inf Sci 596:414–438

    Article  Google Scholar 

  45. Yang Y, Cai J, Yang H, Zhang J, Zhao X (2020) Tad: A trajectory clustering algorithm based on spatial-temporal density analysis. Expert Syst Appl 139:112846

    Article  Google Scholar 

  46. Hu J, Zheng W (2019) Transformation-gated LSTM: efficient capture of short-term mutation dependencies for multivariate time series prediction tasks. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp 1–8

  47. Bao ZQ, Zhao Y, Hu XT, Zhao YY, Huang QD (2020) Minimal peephole long short-term memory. Comput Eng Design 41:134–138

    Google Scholar 

  48. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  49. Chen Q, Zhang W, Zhu K, Zhou D, Dai H, Wu Q (2021) A novel trilinear deep residual network with self-adaptive dropout method for short-term load forecasting. Expert Syst Appl 182:115272

    Article  Google Scholar 

Download references

Acknowledgements

The work is supported by the National Natural Science Foundation of China (Grant No. U1931209), Key Research and Development Projects of Shanxi Province (Grant No. 201903D121116), the central government guides local science and technology development funds (Grant No. 20201070), and the Fundamental Research Program of Shanxi Province(Grant Nos. 20210302123223, 202103021224275).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Taiyuan University of Science and Technology, Waliu Rd, Taiyuan, 030024, Shanxi, China

    Haifeng Yang, Juanjuan Hu, Jianghui Cai, Yupeng Wang, Xin Chen, Xujun Zhao & Lili Wang

  2. School of Computer Science and Technology, North University of China, Shanglan Street, Taiyuan, 038507, Shanxi, China

    Jianghui Cai

  3. School of Computer Science and Technology, Dezhou University, university west Rd, Dezhou, 253023, Shandong, China

    Lili Wang

Authors
  1. Haifeng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juanjuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianghui Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yupeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xujun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianghui Cai.

Ethics declarations

conflicts of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, H., Hu, J., Cai, J. et al. A New MC-LSTM Network Structure Designed for Regression Prediction of Time Series. Neural Process Lett 55, 8957–8979 (2023). https://doi.org/10.1007/s11063-023-11187-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11187-3

Keywords

Search

Navigation