Equipment Health Indicator Learning Using Deep Reinforcement Learning

  • Chi ZhangEmail author
  • Chetan Gupta
  • Ahmed Farahat
  • Kosta Ristovski
  • Dipanjan Ghosh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11053)


Predictive Maintenance (PdM) is gaining popularity in industrial operations as it leverages the power of Machine Learning and Internet of Things (IoT) to predict the future health status of equipment. Health Indicator Learning (HIL) plays an important role in PdM as it learns a health curve representing the health conditions of equipment over time, so that health degradation is visually monitored and optimal planning can be performed accordingly to minimize the equipment downtime. However, HIL is a hard problem due to the fact that there is usually no way to access the actual health of the equipment during most of its operation. Traditionally, HIL is addressed by hand-crafting domain-specific performance indicators or through physical modeling, which is expensive and inapplicable for some industries. In this paper, we propose a purely data-driven approach for solving the HIL problem based on Deep Reinforcement Learning (DRL). Our key insight is that the HIL problem can be mapped to a credit assignment problem. Then DRL learns from failures by naturally backpropagating the credit of failures into intermediate states. In particular, given the observed time series of sensor, operating and event (failure) data, we learn a sequence of health indicators that represent the underlying health conditions of physical equipment. We demonstrate that the proposed methods significantly outperform the state-of-the-art methods for HIL and provide explainable insights about the equipment health. In addition, we propose the use of the learned health indicators to predict when the equipment is going to reach its end-of-life, and demonstrate how an explainable health curve is way more useful for a decision maker than a single-number prediction by a black-box model. The proposed approach has a great potential in a broader range of systems (e.g., economical and biological) as a general framework for the automatic learning of the underlying performance of complex systems.


Health indicator learning Deep Reinforcement Learning Predictive Maintenance 

Supplementary material

473908_1_En_30_MOESM1_ESM.pdf (260 kb)
Supplementary material 1 (pdf 259 KB)


  1. 1.
    Ahmad, R., Kamaruddin, S.: An overview of time-based and condition-based maintenance in industrial application. CAIE 63(1), 135–149 (2012)Google Scholar
  2. 2.
    Susto, G.A., Schirru, A., Pampuri, S., McLoone, S., Beghi, A.: Machine learning for predictive maintenance: a multiple classifier approach. IINF 11(3), 812–820 (2015)Google Scholar
  3. 3.
    Zheng, S., Ristovski, K., Farahat, A., Gupta, C.: Long short-term memory network for remaining useful life estimation. In: IEEE PHM (2017)Google Scholar
  4. 4.
    Liu, K., Gebraeel, N.Z., Shi, J.: A data-level fusion model for developing composite health indices for degradation modeling and prognostic analysis. ASE 10, 652–664 (2013)Google Scholar
  5. 5.
    Saxena, A., et al.: Metrics for evaluating performance of prognostic techniques. In: IEEE PHM (2008)Google Scholar
  6. 6.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  7. 7.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)CrossRefGoogle Scholar
  8. 8.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS (2014)Google Scholar
  9. 9.
    Saxena, A., Goebel, K., Simon, D., Eklund, N.: Damage propagation modeling for aircraft engine run-to-failure simulation. In: IEEE PHM (2008)Google Scholar
  10. 10.
    Li, X., Qian, J., Wang, G.G.: Fault prognostic based on hybrid method of state judgment and regression. AIME 5, 149562 (2013)Google Scholar
  11. 11.
    Ramasso, E.: Investigating computational geometry for failure prognostics in presence of imprecise health indicator: results and comparisons on C-MAPSS datasets. PHM 5(4), 005 (2014)Google Scholar
  12. 12.
    Sateesh Babu, G., Zhao, P., Li, X.-L.: Deep convolutional neural network based regression approach for estimation of remaining useful life. In: Navathe, S.B., Wu, W., Shekhar, S., Du, X., Wang, X.S., Xiong, H. (eds.) DASFAA 2016. LNCS, vol. 9642, pp. 214–228. Springer, Cham (2016). Scholar
  13. 13.
    Liao, Y., Zhang, L., Liu, C.: Uncertainty prediction of remaining useful life using long short-term memory network based on bootstrap method. In: IEEE PHM (2018)Google Scholar
  14. 14.
    Ramasso, E.: Contribution of belief functions to Hidden Markov models with an application to fault diagnosis. In: MLSP (2009)Google Scholar
  15. 15.
    Ramasso, E., Denoeux, T.: Making use of partial knowledge about hidden states in HMMs: an approach based on belief functions. FUZZ 22(2), 395–405 (2014)CrossRefGoogle Scholar
  16. 16.
    El-Koujok, M., Gouriveau, R., Zerhouni, N.: Reducing arbitrary choices in model building for prognostics: an approach by applying parsimony principle on an evolving neuro-fuzzy system. Microelectron. Reliab. 51(2), 310–320 (2011)CrossRefGoogle Scholar
  17. 17.
    Le Son, K., Fouladirad, M., Barros, A.: Remaining useful life estimation on the non-homogenous gamma with noise deterioration based on gibbs filtering: a case study. In: IEEE PHM (2012)Google Scholar
  18. 18.
    Ramasso, E., Saxena, A.: Performance benchmarking and analysis of prognostic methods for CMAPSS datasets. PHM 5(2), 1–15 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Chi Zhang
    • 1
    Email author
  • Chetan Gupta
    • 1
  • Ahmed Farahat
    • 1
  • Kosta Ristovski
    • 1
  • Dipanjan Ghosh
    • 1
  1. 1.Industrial AI LaboratoryHitachi America Ltd.Santa ClaraUSA

Personalised recommendations