Skip to main content
SpringerLink
Log in

Layer-wise relevance propagation for interpreting LSTM-RNN decisions in predictive maintenance

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Predictive maintenance (PdM) is an advanced technique to predict the time to failure (TTF) of a system. PdM collects sensor data on the health of a system, processes the information using data analytics, and then establishes data-driven models that can forecast system failure. Deep neural networks are increasingly being used as these data-driven models owing to their high predictive accuracy and efficiency. However, deep neural networks are often criticized as being “black boxes,” which owing to their multi-layered and non-linear structure provide little insight into the underlying physics of the system being monitored and that are nontransparent and untraceable in their predictions. In order to address this issue, the layer-wise relevance propagation (LRP) technique is applied to analyze a long short-term memory (LSTM) recurrent neural network (RNN) model. The proposed method is demonstrated and validated for a bearing health monitoring study based on vibration data. The obtained LRP results provide insights into how the model “learns” from the input data and demonstrate the distribution of contribution/relevance to the neural network classification in the input space. In addition, comparisons are made with gradient-based sensitivity analysis to show the power of LRP in interpreting RNN models. The LRP is proved to have promising potential in interpreting deep neural network models and improving model accuracy and efficiency for PdM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

The raw data used in case study was from NASA Ames Prognostics Data Repository (http://ti.arc.nasa.gov/project/prognostic-data-repository).

References

  1. Monostori L, Kádár B, Bauernhansl T, Monostori L, Kádár B, Bauernhansl T, Kondoh S, Kumara S, Reinhart G, Sauer O, Schuh G, Sihn W, Ueda K (2016) Cyber-physical systems in manufacturing. CIRP Ann 65:621–641. https://doi.org/10.1016/j.cirp.2016.06.005

    Article  Google Scholar 

  2. Menard S (2011) Applied logistic regression analysis. Appl Logist Regres Anal. https://doi.org/10.4135/9781412983433

  3. Wang G, Luo Z, Qin X, Leng Y, Wang T (2008) Fault identification and classification of rolling element bearing based on time-varying autoregressive spectrum. Mech Syst Signal Process 22:934–947. https://doi.org/10.1016/j.ymssp.2007.10.008

    Article  Google Scholar 

  4. Salem O, Guerassimov A, Mehaoua A (2014) Anomaly detection in medical wireless sensor networks using SVM and linear regression models. Int J E-Health Med Commun 5:20–45. https://doi.org/10.4018/ijehmc.2014010102

    Article  Google Scholar 

  5. Yan HC, Zhou JH, Pang CK (2015) Gamma process with recursive MLE for wear PDF prediction in precognitive maintenance under aperiodic monitoring. Mechatronics 31:68–77. https://doi.org/10.1016/j.mechatronics.2015.05.009

    Article  Google Scholar 

  6. Xu W, Wang W (2012) An adaptive gamma process based model for residual useful life prediction. Proc IEEE 2012 Progn Syst Heal Manag Conf PHM-2012 3–6. https://doi.org/10.1109/PHM.2012.6228785

  7. Wei Q, Xu D (2014) Remaining useful life estimation based on gamma process considered with measurement error. ICRMS 2014 - Proc 2014 10th Int Conf Reliab Maintainab Saf More Reliab Prod More Secur Life 645–649. https://doi.org/10.1109/ICRMS.2014.7107275

  8. Zhou D, Yu Z, Zhang H, Weng S (2016) A novel grey prognostic model based on Markov process and grey incidence analysis for energy conversion equipment degradation. Energy 109:420–429. https://doi.org/10.1016/j.energy.2016.05.008

    Article  Google Scholar 

  9. Ertunc HM, Loparo KA, Ocak H (2001) Tool wear condition monitoring in drilling operations using hidden Markov models (HMMs). Int J Mach Tools Manuf 41:1363–1384. https://doi.org/10.1016/S0890-6955(00)00112-7

    Article  Google Scholar 

  10. Ocak H, Loparo KA, Discenzo FM (2007) Online tracking of bearing wear using wavelet packet decomposition and probabilistic modeling: a method for bearing prognostics. J Sound Vib 302:951–961. https://doi.org/10.1016/j.jsv.2007.01.001

    Article  Google Scholar 

  11. Guo L (2017) A recurrent neural network based health indicator for remaining useful life prediction of bearings. Bol Tec Bull 55:585–590. https://doi.org/10.1016/j.neucom.2017.02.045

    Article  Google Scholar 

  12. Huang CG, Huang HZ, Li YF (2019) A bidirectional LSTM prognostics method under multiple operational conditions. IEEE Trans Ind Electron 66:8792–8802. https://doi.org/10.1109/TIE.2019.2891463

    Article  Google Scholar 

  13. Gugulothu N (2017) Predicting remaining useful life using time series embeddings based on recurrent neural networks. CEUR Workshop Proc 2657:1–9. https://doi.org/10.1145/nnnnnnn.nnnnnnn

    Article  Google Scholar 

  14. Montavon G, Samek W, Müller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Signal Process A Rev J 73:1–15. https://doi.org/10.1016/j.dsp.2017.10.011

    Article  MathSciNet  Google Scholar 

  15. Böhle M, Eitel F, Weygandt M, Ritter K (2019) Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer’s disease classification. Front Aging Neurosci 11:10. https://doi.org/10.3389/fnagi.2019.00194

    Article  Google Scholar 

  16. Arras L, Montavon G, Müller K-R, Samek W (2018) Explaining recurrent neural network predictions in sentiment analysis. 159–168. https://doi.org/10.18653/v1/w17-5221

  17. Hochreiter S. The vanishing gradient problem during learning recurrent neural nets and problem solutions

  18. Werbos PJ (1990) Backpropagation through time: what it does and how to do it. Proc IEEE 78:1550–1560. https://doi.org/10.1109/5.58337

    Article  Google Scholar 

  19. Hochreiter S (1997) Long short-term memory. 1780:1735–1780

  20. Wu D, Jiang Z, Xie X, Wei X (2020) LSTM learning with Bayesian and Gaussian processing for anomaly detection in industrial IoT. 16:5244–5253

  21. Wang Y, Perry M, Whitlock D, Sutherland JW (2020) Detecting anomalies in time series data from a manufacturing system using recurrent neural networks. J Manuf Syst. https://doi.org/10.1016/j.jmsy.2020.12.007

  22. Li Z, Li J, Wang Y, Wang K (2019) A deep learning approach for anomaly detection based on SAE and LSTM in mechanical equipment. Int J Adv Manuf Technol 103:499–510. https://doi.org/10.1007/s00170-019-03557-w

    Article  Google Scholar 

  23. Dudukcu HV, Taskiran M, Kahraman N (2020) LSTM and WaveNet implementation for predictive maintenance of turbofan engines. 20th IEEE Int Symp Comput Intell Informatics, CINTI 2020 - Proc 151–156. https://doi.org/10.1109/CINTI51262.2020.9305820

  24. Zhang W, Guo W, Liu X, Liu Y, Zhou J, Li B, Lu Q, Yang S (2018) LSTM-based analysis of industrial IoT equipment. IEEE Access 6:23551–23560. https://doi.org/10.1109/ACCESS.2018.2825538

    Article  Google Scholar 

  25. Zhao R, Wang J, Yan R, Mao K (2016) Machine health monitoring with LSTM networks. Proc Int Conf Sens Technol ICST:17–22. https://doi.org/10.1109/ICSensT.2016.7796266

  26. Binder A, Bach S, Montavon G et al (2016) Layer-wise relevance propagation for deep neural network architectures. Lect Notes Electr Eng 376:913–922. https://doi.org/10.1007/978-981-10-0557-2_87

    Article  Google Scholar 

  27. Arras L, Arjona-Medina J, Widrich M, et al (2019) Explaining and interpreting LSTMs. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 11700 LNCS:211–238. https://doi.org/10.1007/978-3-030-28954-6_11

  28. Wu H, Huang A, Sutherland JW (2020) Avoiding environmental consequences of equipment failure via an LSTM-based model for predictive maintenance. Procedia Manuf 43:666–673. https://doi.org/10.1016/j.promfg.2020.02.131

    Article  Google Scholar 

  29. University of Cincinnati (2006). “Bearing data set”, NASA Ames prognostics data repository (http://ti.arc.nasa.gov/project/prognostic-data-repository), NASA Ames Research Center, Moffett Field, CA

  30. Qiu H, Lee J, Lin J, Yu G (2006) Wavelet filter-based weak signature detection method and its application on rolling element bearing prognostics. J Sound Vib 289:1066–1090. https://doi.org/10.1016/j.jsv.2005.03.007

    Article  Google Scholar 

  31. Lee J, Wu F, Zhao W, Ghaffari M, Liao L, Siegel D (2014) Prognostics and health management design for rotary machinery systems - reviews, methodology and applications. Mech Syst Signal Process 42:314–334. https://doi.org/10.1016/j.ymssp.2013.06.004

    Article  Google Scholar 

  32. Zurada JM, Malinowski A, Cloete I (1994) Sensitivity analysis for minimization of input data dimension for feedforward neural network. Proc - IEEE Int Symp Circuits Syst 6:447–450. https://doi.org/10.1109/iscas.1994.409622

    Article  Google Scholar 

  33. Caesarendra W, Tjahjowidodo T (2017) A review of feature extraction methods in vibration-based condition monitoring and its application for degradation trend estimation of low-speed slew bearing. Machines 5:21. https://doi.org/10.3390/machines5040021

    Article  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge the support of the Wabash Heartland Innovation Network (WHIN) grant at Purdue University.

Funding

This research was supported by the Wabash Heartland Innovation Network (WHIN) grant at Purdue University.

Author information

Authors and Affiliations

  1. Environmental and Ecological Engineering, Purdue University, 500 Central Drive, West Lafayette, IN, 47907, USA

    Haiyue Wu, Aihua Huang & John W. Sutherland

Authors
  1. Haiyue Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aihua Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John W. Sutherland
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Haiyue Wu: Conceptualization, investigation, methodology, writing-original draft.

Aihua Huang: Writing-reviewing and editing.

John W. Sutherland: Supervision, writing-reviewing and editing.

Corresponding author

Correspondence to Haiyue Wu.

Ethics declarations

Ethics approval

The authors would like to declare that the work described was original which has not been published previously or not under consideration for publication elsewhere.

Consent to participate

Not applicable

Consent for publication

All of the authors have reviewed the final version of the manuscript and approved it for publication.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 3 Features used in the LSTM-RNN model (feature details could be found in [11, 30])
Full size table
Table 4 List of notations
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Huang, A. & Sutherland, J.W. Layer-wise relevance propagation for interpreting LSTM-RNN decisions in predictive maintenance. Int J Adv Manuf Technol 118, 963–978 (2022). https://doi.org/10.1007/s00170-021-07911-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-021-07911-9

Keywords

Search

Navigation