Abstract
Remaining Useful Life (RUL) estimation is a fundamental task in the prognostic and health management (PHM) of industrial equipment and systems. To this end, we propose a novel approach for RUL estimation in this paper, based on deep neural architecture due to its great success in sequence learning. Specifically, we take the Transformer encoder as the backbone of our model to capture short- and long-term dependencies in a time sequence. Compared with convolutional neural network based methods, there is no limitation from the kernel size for a complete receptive field of all time steps. While compared with recurrent neural network based methods, we develop our model based on dot-product self-attention, enabling it to fully exploit parallel computation. Moreover, we further propose a gated convolutional unit to facilitate the model’s ability of incorporating local contexts at each time step, for the attention mechanism used in the Transformer encoder makes the output high-level features insensitive to local contexts. We conduct experiments on the C-MAPSS datasets and show that, the performance of our model is superior or comparable to those of other existing methods. We also carry out ablation studies and demonstrate the necessity and effectiveness of each component used in the proposed model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Babu, G. S., Zhao, P., & Li, X. L. (2016). Deep convolutional neural network based regression approach for estimation of remaining useful life. International conference on database systems for advanced applications (pp. 214–228). Cham: Springer.
Benkedjouh, T., Medjaher, K., Zerhouni, N., & Rechak, S. (2015). Health assessment and life prediction of cutting tools based on support vector regression. Journal of Intelligent Manufacturing, 26(2), 213–223.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., & Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860
Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Ellefsen, A. L., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering & System Safety, 183, 240–251.
Fahad, S.A., & Yahya, A.E. (2018). Inflectional review of deep learning on natural language processing. In: 2018 international conference on smart computing and electronic enterprise (ICSCEE), pp. 1–4. IEEE.
Guo, L., Li, N., Jia, F., Lei, Y., & Lin, J. (2017). A Recurrent Neural Network Based Health Indicator for Remaining Useful Life Prediction of Bearings. Neurocomputing, 240, 98–109.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
Huang, C.Z.A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A.M., Hoffman, M.D., Dinculescu, M., & Eck, D. (2018). Music transformer. arXiv preprint arXiv:1809.04281
Huang, R., Xi, L., Li, X., Liu, C. R., Qiu, H., & Lee, J. (2007). Residual life predictions for ball bearings based on self-organizing map and back propagation neural network methods. Mechanical Systems and Signal Processing, 21(1), 193–207.
Kacprzynski, G., Sarlashkar, A., Roemer, M., Hess, A., & Hardman, B. (2004). Predicting remaining life by fusing the physics of failure modeling with diagnostics. JOm, 56(3), 29–35.
Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.X., & Yan, X. (2019). Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. arXiv preprint arXiv:1907.00235
Li, X., Ding, Q., & Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering & System Safety, 172, 1–11.
Li, Y., Billington, S., Zhang, C., Kurfess, T., Danyluk, S., & Liang, S. (1999). Adaptive prognostics for rolling element bearing condition. Mechanical Systems and Signal Processing, 13(1), 103–113.
Malhi, A., Yan, R., & Gao, R. X. (2011). Prognosis of defect propagation based on recurrent neural networks. IEEE Transactions on Instrumentation and Measurement, 60(3), 703–711.
Mosallam, A., Medjaher, K., & Zerhouni, N. (2016). Data-driven prognostic method based on bayesian approaches for direct remaining useful life prediction. Journal of Intelligent Manufacturing, 27(5), 1037–1048.
Nieto, P. G., Garcia-Gonzalo, E., Lasheras, F. S., & de Cos Juez, F. J. (2015). Hybrid pso-svm-based method for forecasting of the remaining useful life for aircraft engines and evaluation of its reliability. Reliability Engineering & System Safety, 138, 219–231.
Oppenheimer, C. H., & Loparo, K. A. (2002). Physically based diagnosis and prognosis of cracked rotor shafts. Component and systems diagnostics, prognostics, and health management II (Vol. 4733, pp. 122–132). Washington: International Society for Optics and Photonics.
Peng, Y., Dong, M., & Zuo, M. J. (2010). Current status of machine prognostics in condition-based maintenance: a review. The International Journal of Advanced Manufacturing Technology, 50(1–4), 297–313.
Rush, A.M. (2018). The annotated transformer. In: Proceedings of workshop for NLP open source software (NLP-OSS), pp. 52–60.
San Kim, T., & Sohn, S.Y. (2020). Multitask learning for health condition identification and remaining useful life prediction: Deep convolutional neural network approach. Journal of Intelligent Manufacturing, 1–11.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine run-to-failure simulation. In: 2008 international conference on prognostics and health management, pp. 1–9. IEEE.
Tian, Z. (2012). An artificial neural network method for remaining useful life prediction of equipment subject to condition monitoring. Journal of Intelligent Manufacturing, 23(2), 227–237.
Tian, Z., Wong, L., & Safaei, N. (2010). A neural network approach for remaining useful life prediction utilizing both failure and suspension histories. Mechanical Systems and Signal Processing, 24(5), 1542–1555.
Tobon-Mejia, D.A., Medjaher, K., Zerhouni, N., & Tripot, G. (2011). Hidden markov models for failure diagnostic and prognostic. In: 2011 Prognostics and system health managment confernece, pp. 1–8. IEEE.
Tobon-Mejia, D. A., Medjaher, K., Zerhouni, N., & Tripot, G. (2012). A data-driven failure prognostics method based on mixture of gaussians hidden markov models. IEEE Transactions on Reliability, 61(2), 491–503.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017) Attention is all you need. arXiv preprint arXiv:1706.03762
Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018.
Wang, H., & Chen, J. (2014). Performance degradation assessment of rolling bearing based on bispectrum and support vector data description. Journal of Vibration and Control, 20(13), 2032–2041.
Wu, Q., Ding, K., & Huang, B. (2018). Approach for fault prognosis using recurrent neural network. Journal of Intelligent Manufacturing, 1–13.
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018). Remaining useful life estimation of engineered systems using vanilla lstm neural networks. Neurocomputing, 275, 167–179.
Xu, M., Dai, W., Liu, C., Gao, X., Lin, W., Qi, G.J., & Xiong, H. (2020). Spatial-temporal transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.02908
Xu, X., Wu, Q., Li, X., & Huang, B. (2020). Dilated convolution neural network for remaining useful life prediction. Journal of Computing and Information Science in Engineering, 20(2).
Yu, W., Kim, I. Y., & Mechefske, C. (2019). Remaining useful life estimation using a bidirectional recurrent neural network based autoencoder scheme. Mechanical Systems and Signal Processing, 129, 764–780.
Yu, W., Kim, I. Y., & Mechefske, C. (2020). An improved similarity-based prognostic algorithm for rul estimation using an rnn autoencoder scheme. Reliability Engineering & System Safety, 199, 106926.
Yuan, M., Wu, Y., & Lin, L. (2016). Fault diagnosis and remaining useful life estimation of aero engine using lstm neural network. In: 2016 IEEE international conference on aircraft utility systems (AUS), pp. 135–140. IEEE.
Zhang, C., Lim, P., Qin, A. K., & Tan, K. C. (2016). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2306–2318.
Zhang, X., Kang, J., & Jin, T. (2014). Degradation modeling and maintenance decisions based on bayesian belief networks. IEEE Transactions on Reliability, 63(2), 620–633.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long short-term memory network for remaining useful life estimation. In: 2017 IEEE international conference on prognostics and health management (ICPHM), pp. 88–95. IEEE.
Acknowledgements
This work was supported by the National Key R&D Program of China (No. 2018YFF0214705), and a grant from the Institute for Guo Qiang, Tsinghua University (No. 2019GQG0002).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Yu Mo and Qianhui Wu are contributed equally to this work.
Rights and permissions
About this article
Cite this article
Mo, Y., Wu, Q., Li, X. et al. Remaining useful life estimation via transformer encoder enhanced by a gated convolutional unit. J Intell Manuf 32, 1997–2006 (2021). https://doi.org/10.1007/s10845-021-01750-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-021-01750-x