Abstract
In the current era of Internet of Things, typically data from multiple sources are captured through various sensors yielding Multivariate Time Series (MTS) data. Sensor MTS prediction has several real-life applications in various domains such as healthcare, manufacturing, and agriculture. In this paper, we propose a novel Recurrent Neural Network (RNN) architecture that leverages contextual information and attention mechanism for sensor MTS prediction. We adopt the notion of primary and contextual features to distinguish between the features that are independently useful for learning irrespective of other features, and the features that are not useful in isolation. The contextual information is represented through the contextual features and when used with primary features can potentially improve the performance of the model. The proposed architecture uses the contextual features in two ways. Firstly, to weight the primary input features depending on the context, and secondly to weight the hidden states in the alignment model. The latter is used to compute the dependencies between hidden states (representations) to derive the attention vector. Further, integration of the context and attention allows visualising temporally and spatially the relevant parts of the input sequence which are influencing the prediction. To evaluate the proposed architecture, we used two benchmark datasets as they provide contextual information. The first is NASA Turbofan Engine Degradation Simulation dataset for estimating Remaining Useful Life, and the second is appliances energy prediction dataset. We compared the proposed approach with the state-of-the-art methods and observed improved prediction results, particularly with respect to the first dataset.
This work is supported by CONEX-Plus programme funded by Universidad Carlos III de Madrid and the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 801538.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The code is available at https://github.com/rduttabaruah/CiRNN.
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate (2014). https://arxiv.org/abs/1409.0473
Candanedo, L.M., Feldheim, V., Deramaix, D.: Data driven prediction models of energy use of appliances in a low-energy house. Energy Build. 140, 81–97 (2017)
Cheng, Q., Chen, Y., Xiao, Y., Yin, H., Liu, W.: A dual-stage attention-based Bi-LSTM network for multivariate time series prediction. J. Supercomput. 78(14), 16214–16235 (2022)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734. Association for Computational Linguistics, Doha, Qatar (2014)
Cinar, Y., Mirisaee, H., Goswami, P., Gaussier, E., Aït-Bachir, A., Strijov, V.: Position-based content attention for time series forecasting with sequence-to-sequence RNNs. In: International Conference on Neural Information Processing, pp. 533–544 (2017). https://doi.org/10.1007/978-3-319-70139-4_54
da Costa, P., Akçay, A.E., Zhang, Y., Kaymak, U.: Attention and long short-term memory network for remaining useful lifetime predictions of turbofan engine degradation. IJPHM Special Issue PHM Appl. Deep Learn. Emerging Anal. 10(4), 1–12 (2019)
Du, S., Li, T., Yang, Y., Horng, S.J.: Multivariate time series forecasting via attention-based encoder-decoder framework. Neurocomputing 388, 269–279 (2020)
Dutta Baruah, R., Muñoz Organero, M.: Integrating explicit contexts with recurrent neural networks for improving prognostic models. In: IEEE Aerospace Conference (2023), accepted
Dutta Baruah, R., Organero, M.M.: Explicit context integrated recurrent neural network for sensor data applications (2023). https://arxiv.org/abs/2301.05031
Han, Z., Zhao, J., Leung, H., Ma, K.F., Wang, W.: A review of deep learning models for time series prediction. IEEE Sens. J. 21(6), 7833–7848 (2021)
Haruehansapong, K., Roungprom, W., Kliangkhlao, M., Yeranee, K., Sahoh, B.: Deep learning-driven automated fault detection and diagnostics based on a contextual environment: a case study of HVAC system. Buildings 13(1) (2023)
Heimes, F.O.: Recurrent neural networks for remaining useful life estimation. In: 2008 International Conference on Prognostics and Health Management, pp. 1–6 (2008)
Kinch, M.W., Melis, W.J., Keates, S.: The benefits of contextual information for speech recognition systems. In: 2018 10th Computer Science and Electronic Engineering (CEEC), pp. 225–230 (2018)
Li, H., Zhao, W., Zhang, Y., Zio, E.: Remaining useful life prediction using multiiscale deep convolutions neural network. Appl. Soft Comput. 89, 106113 (2020)
Li, X., Ding, Q., Sun, J.Q.: Remaining useful life estimation in prognostics using deep convolution neural networks. Reliabil. Eng. Syst. Safety 172, 1–11 (2018)
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., Zhang, H.: Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliabil. Eng. Syst. Safety 183, 240–251 (2019)
Liu, L., Song, X., Zhou, Z.: Aircraft engine remaining useful life estimation via a double attention-based data-driven architecture. Reliabil. Eng. Syst. Safety 221, 108330 (2022)
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421. Association for Computational Linguistics, Lisbon, Portugal (2015)
Munkhdalai, L., et al.: An end-to-end adaptive input selection with dynamic weights for forecasting multivariate time series. IEEE Access 7, 99099–99114 (2019). https://doi.org/10.1109/ACCESS.2019.2930069
Qin, Y., Song, D., Cheng, H., Cheng, W., Jiang, G., Cottrell, G.W.: A dual-stage attention-based recurrent neural network for time series prediction. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2627–2633. IJCAI’17, AAAI Press (2017)
Saxena, A., Goebel, K., Simon, D., Eklund, N.: Damage propagation modeling for aircraft engine run-to-failure simulation. In: 2008 International Conference on Prognostics and Health Management, pp. 1–9 (2008)
Shah, S.R.B., Chadha, G.S., Schwung, A., Ding, S.X.: A sequence-to-sequence approach for remaining useful lifetime estimation using attention-augmented bidirectional LSTM. Intell. Syst. Appl. 10, 200049 (2021)
Shih, S.Y., Sun, F.K., Lee, H.Y.: Temporal pattern attention for multivariate time series forecasting. Mach. Learn. 108(8), 1421–1441 (2019)
Song, Y., Gao, S., Li, Y., Jia, L., Li, Q., Pang, F.: Distributed attention-based temporal convolutional network for remaining useful life prediction. IEEE Internet Things J. 8(12), 9594–9602 (2020)
Sun, L., Zhong, Z., Zhang, C., Zhang, Y., Wu, D.: TESS: multivariate sensor time series prediction for building sustainable smart cities. ACM Trans. Sens. Netw. (2022), just Accepted
Turney, P.D.: The management of context-sensitive features: a review of strategies (2002). https://arxiv.org/abs/cs/0212037
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
Wang, X., Li, Y., Xu, Y., Liu, X., Zheng, T., Zheng, B.: Remaining useful life prediction for aero-engines using a time-enhanced multi-head self-attention model. Aerospace 10(1) (2023)
Wen, Q., et al.: Transformers in time series: a survey (2023)
Yang, Y., Jinfu, F., Zhongjie, W., Zheng, Z., Yukun, X.: A dynamic ensemble method for residential short-term load forecasting. Alex. Eng. J. 63, 75–88 (2023)
Zhang, T., Liao, L., Lai, H., Liu, J., Zou, F., Cai, Q.: Electrical energy prediction with regression-oriented models. In: Krömer, P., Zhang, H., Liang, Y., Pan, J.-S. (eds.) ECC 2018. AISC, vol. 891, pp. 146–154. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-03766-6_16
Zheng, S., Ristovski, K., Farahat, A., Gupta, C.: Long short-term memory network for remaining useful life estimation. In: 2017 IEEE International Conference on Prognostics and Health Management (ICPHM), pp. 88–95 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Ethical Statement
This research work does not involve any human subjects or personal information pertaining to them. It neither has potential policing or military use.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dutta Baruah, R., Muñoz-Organero, M. (2023). Exploiting Context and Attention Using Recurrent Neural Network for Sensor Time Series Prediction. In: Ifrim, G., et al. Advanced Analytics and Learning on Temporal Data. AALTD 2023. Lecture Notes in Computer Science(), vol 14343. Springer, Cham. https://doi.org/10.1007/978-3-031-49896-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-49896-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49895-4
Online ISBN: 978-3-031-49896-1
eBook Packages: Computer ScienceComputer Science (R0)
