Predicting maintenance through an attention long short-term memory projected model

Abstract

Long sequence information remains a challenging problem in deep learning nowadays for predicting remaining useful life (RUL). In this work, we propose a novel deep learning module called attention long short-term memory projected (ALSTMP) for RUL estimation to mitigate the inefficient information of long-term dependencies. The ALSTMP is designed to utilize attention mechanisms in traditional long short-term memory (LSTM) for effectively collecting key features of the dataset. Moreover, the time-window length method is implemented to generate a better feature extraction. The proposed model not only outperforms the traditional LSTM and its extension but also the latest existing approaches with a smaller quantity of parameters compared with recent deep learning approaches.

Appendices

Appendices

Appendix A: Ablation study

In this work, we conducted experiments on an alternative structure, where the attention mechanism is applied to forget gate ${{f}_{t}}$ of the traditional LSTM, referred to as ALSTMP(1), to strengthen our scientific contribution. We did not investigate the integration for output gate ${{o}_{t}}$ because it has no contribution to the cell gate memory ${{C}_{t}}$. The RUL prediction is made on the two most difficult sub-datasets of the C-MAPSS dataset, which are FD002 and FD004. We investigated two structures with different time window lengths. The results are presented in Tables 6 and 7.

Table 8 Description of each engine in DS02 sub-dataset of N-CMAPSS

The results of ALSTMP are generally better than those of ALSTMP(1), as described in the two figures. Furthermore, the RMSE and Score values of ALSTMP(1) tend to increase with a larger time window length (the smaller the RMSE and Score, the better the model). In conclusion, the results show that our proposed ALSTMP is the best possible design and well-suited for the time window length method. The results on RMSE and Score values are depicted in Fig. 10.

Appendix B: N-CMAPSS dataset description

In 2021, Arias Chao et al. (2021) further improved the original CMAPSS dataset to generate the new dataset, referred to as N-CMAPSS. In this dataset, realistic flight conditions, such as those recorded on board a commercial jet, were utilized as inputs to the original CMAPSS model (Saxena et al., 2008). The major difference with respect to the original CMAPSS dataset is that the data in N-CMAPSS were significantly enlarged to million (M) samples. Meanwhile, eight aircraft fleets and seven different failure modes exist in the N-CMAPSS dataset, which contains different run-to-fail data, including operating conditions, monitoring sensors, degradation information, and auxiliary information. In the operative conditions, four variables, including altitude, flight Mach number, throttle-resolver angle, and total temperature at the fan inlet, are provided. The monitoring data consist of 14 sensor data and 14 virtual sensor data, while other variables, such as flight classes, RUL label, and binary health state, are provided. Finally, the degradation information described multiple fault latent factors in the dynamic model.

Table 9 RUL prediction of the ALSTMP model on different time window lengths using DS02 test set

Table 10 The performance comparison results on DS02 test set between the proposed ALSTMP model and recent benchmarks

We validated the performances of our proposed model by using the N-CMAPSS dataset. In particular, we used the DS02 sub-dataset of N-CMAPSS, which was developed for data-driven problems (Saxena et al., 2008). The DS02 sub-dataset includes run-to-failure degradation trajectories from nine different turbofan engines with unknown initial healthy conditions. On the basis of the work in Chao et al. (2022), we used six engines (${{u}_{2}}$, ${{u}_{5}}$, ${{u}_{10}}$, ${{u}_{16}}$, ${{u}_{18}}$, and ${{u}_{20}}$) for the training set (${{D}_{train}}$) and three engines (${{u}_{11}}$, ${{u}_{14}}$, and ${{u}_{15}}$) for the test set (${{D}_{test}}$). Specifically, the evaluation results on the test set (${{D}_{test}}$) can implicitly reflect the comprehensive effectiveness of the RUL models because the two units (${{u}_{11}}$ and ${{u}_{14}}$) contain shorter and lower altitude flights compared with those units in the training set (${{D}_{train}}$).

The details of the DS02 sub-dataset are described in Table 8. The total number of samples in the training and test sets are 5.26 and 1.25M, respectively. The end-of-life times (${{t}_{EOL}}$) relate to the ending life cycle of the engine’s lifespan, which is also considered as the initial labeled RUL corresponding to that engine. Meanwhile, the two different failure modes, the abnormal high-pressure turbine (HPT) and low-pressure turbine (LPT), are also presented in the table. It is worth noticing that the unit with combined failure modes is subject to a more complicated failure mode than a single one.

Appendix C: Experimental results on the DS02 sub-dataset of N510 experimental results on the DS02 sub-dataset of N-CMAPSS

We utilized the same setup as the work in Chao et al. (2022). In particular, the sample rate of the data is 0.1 Hz, where the total samples for the training and test sets are 0.526 and 0.125M, respectively. Moreover, we chose the same 20 signals from the multivariate time series for our selected features ${{N}_{f}}$. We also applied the time window length method to the DS02 sub-dataset to investigate the effectiveness of our proposed model. No deletion was carried out on the test set since all the recorded data cycles are large enough for the time window length method. We then trained our proposed model with a set of six different window lengths to figure out the most effective window for the test set of DS02. Thereafter, we made the RUL prediction on each unit of the test set and the whole data (Average) to examine how the predicted data are affected by the different lengths of time window. The evaluation results are described in Table 9.

The table illustrates that the RMSE and Score values of the unit reduce when the time window length is increased, indicating that the larger the time window size, the better the RUL prediction. Meanwhile, the RUL estimation for the other units (${{u}_{14}}$ and ${{u}_{15}}$) is better with a smaller time window length, where the best results are obtained at the length of 30. Unit ${{u}_{11}}$ has the longest life cycles among all the three units. Therefore, it can be reflected that a bigger time window length will be more suitable for those units with longer life cycles. In terms of RUL prediction on the whole test set (Average), the table demonstrates that the best results commonly line up on the time window length of 40, 50, 60, and 70, where the best result is achieved at the length of 40. Finally, we used the best result in Table 9 for comparison with the benchmarks.

Fig. 11

RUL prediction of the different models and actual values on unit 14

To set up, we selected the time window length ${{L}_{tw}}=40$ for all the comparing models at the pre-processing stage. The architectures of the comparison include a simple multilayer perceptron (MLP) structure in Mo and Iacca (2022) and a proposed CNN structure in Chao et al. (2022). In particular, the MLP network consists of four hidden layers, in which the first three of them have 200 neurons, and the fourth layer has 50 neurons. The ReLU activation function is applied for all hidden nodes. Meanwhile, the CNN structure is constructed of three convolution layers followed by a fully connected layer (FC). The selected kernel size is 10 for all CNN layers. The output size of the first two layers is 10, while it is reduced to one for the last CNN layer. The output size for the FC layer is 50, and the ReLU activation function is applied for all layers. The results are presented in Table 10.

Fig. 12

RUL prediction of the different models and actual values on unit 15

Table 11 The results of the Wilcoxon signed-rank test on test set (${{D}_{test}}$)

It can be observed that our proposed ALSTM model outperformed all the best-presented benchmarks on RUL prediction of units ${{u}_{11}}$, ${{u}_{15}}$ and the whole test set ${{D}_{test}}$ (Average). Comparing with the MLP method, the results show that our proposed model obtains 20.55%, 10.51%, 22.06%, and 17.07% reductions in the RMSE values, respectively. Meanwhile, the Score values reduce by 26.43%, 18.24%, 26.59%, and 22.15% in units ${{u}_{11}}$, ${{u}_{14}}$, ${{u}_{15}}$, and the whole test set ${{D}_{test}}$, respectively. The superior results of our proposed method have shown the benefit of incorporating the attention mechanism inside the LSTM and the effectiveness of the time window length method. In summary, the results on unit ${{u}_{14}}$ from our proposed model are inferior to the best approaches (CNN); however, the comprehensive performances on the other sub-dataset are still remarkable. Lastly, we evaluated each specific unit for all models in Figs. 10, 11, and 12.

Appendix D: Statistical discussion

In addition to using the RMSE and Score values to validate the performance of our proposed model, we investigate whether the obtained results are statistically significant. In this study, we used the Wilcoxon signed-rank test to compare the errors in pair of the RUL predictive models. We used the same setup as motivated by the work in Nguyen et al. (2020), where the formulas of the absolute error for each RUL estimation are defined as follows:

$$\begin{aligned} {{A}_{1}}= & {} \left| {{Y}_{t}}- \right. \left. {{Y}_{p1}} \right| , \end{aligned}$$

(24)

$$\begin{aligned} {{A}_{2}}= & {} \left| {{Y}_{t}}- \right. \left. {{Y}_{p2}} \right| , \end{aligned}$$

(25)

where ${{Y}_{t}}$ is true labeled RUL, ${{Y}_{p1}}$ is the predicted RUL of model 1 and ${{Y}_{p2}}$ is the predicted RUL of model 2. We applied a one-tailed hypothesis test to determine whether the prediction of model 1 is better than model 2. The null hypothesis (${{H}_{o}}$) is that “two models have the same predictive error” (${{A}_{1}}={{A}_{2}}$). The alternative hypothesis (${{H}_{a}}$) is that “the first model has a smaller error than the second model” (${{A}_{1}}<{{A}_{2}}$). In this case, the Wilcoxon signed-rank test is run using the SciPy library in Python with a significance level of 0.05. The output of Wilcoxon signed-rank test is a ${p{\text {-}}value}$, which is used to judge whether the null hypothesis (${{H}_{o}}$) will fail to reject $(p{\text {-}}value>0.05)$ and vice versa, the null hypothesis (${{H}_{a}}$) will be rejected $(p{\text {-}}value<0.05)$. The results of the Wilcoxon signed-rank test on the test set (${{D}_{test}}$) for each combination of the three models are summarized in Table 11.

The table represents the results from running the Wilcoxon signed-rank test, where errors between different models are compared. In particular, the absolute error of each model is calculated and denoted as ${{A}_{MLP}}$, ${{A}_{CNN}}$, and ${{A}_{ALSTMP}}$ corresponding to the compared models in Table 10. The ${p{\text {-}}value}$ between MLP and CNN on units ${{u}_{14}}$ and ${{u}_{15}}$ are smaller than the threshold value (0.05), while the results for the other two sets (unit ${{u}_{11}}$ and Average) are not small enough to reject the null hypothesis. Meanwhile, the ${p{\text {-}}value}$ between our proposed model and each of the compared models (ALSTMP–MLP and ALSTMP–CNN) are small enough to reject the null hypothesis, except the ${p{\text {-}}value}$ of ALSTMP–MLP on unit ${{u}_{11}}$. Based on the Wilcoxon signed-rank test on the full test set (Average), it can be concluded that the error from our proposed model was a statistically better model than the MLP and CNN.