Multi-layer Contribution Propagation Analysis for Fault Diagnosis

The recent development of feature extraction algorithms with multiple layers in machine learning and pattern recognition has inspired many applications in multivariate statistical process monitoring. In this work, two existing multilayer linear approaches in fault detection are reviewed and a new one with extra layer is proposed in analogy. To provide a general framework for fault diagnosis in succession, this work also proposes the contribution propagation analysis which extends the original definition of contribution of variables in multivariate statistical process monitoring. In fault diagnosis stage, the proposed contribution propagation analysis for multilayer linear feature extraction algorithms is compared with the fault diagnosis results of original contribution plots associated with single layer feature extraction approach. Plots of variable contributions obtained by the aforementioned approaches on the data sets collected from a simulated benchmark case study (Tennessee Eastman process) as well as an industrial scale multiphase flow facility are presented as a demonstration of the usage and performance of the contribution propagation analysis on multilayer linear algorithms.


Introduction
The fast-developing feature extraction approaches in machine learning and pattern recognition, such as neural network and Bayesian network, are playing a more and more important role in data-driven fault detection and diagnosis [1,2] , many of which have multiple layers of data analysis so that the algorithm performance can be enhanced.In the meantime, multivariate statistical process monitoring approaches, such as principle component analysis [3] , canonical variate analysis [4] , Fisher discriminant analysis [5] , independent component analysis [6] and their kin [7][8][9][10][11] , may also take the advantage of multiple layers of learning in order to handle the complexities in process data analytics and data-driven process monitoring.Existing complexities, such as nonlinearity, temporal correlation, multimodality, and non-Gaussianity, may be reflected by process data especially when abundant measurements are available for process data analytics while each basic multivariate data analytic method may cope with only some of them individually.Therefore, multi-layer feature extraction structure is of interest for multivariate statistical process monitoring and attention has been paid to this establishment recently.For instance, Cao and Samuel [12] have established a principal component analys-is PCA-enhanced canonical variate analysis (CVA) method for fault detection and applied it to the Tennessee Eastman challenge process.The state-space independent component analysis (ICA) [13] was developed by combining CVA and ICA in order to handle both temporal correlation and non-Gaussianity in process data.More examples of this establishment include CVA-fisher discriminant analysis (FDA) [9] and ICA-PCA [14] based process monitoring.Similar multi-layer approach has been proposed for nonlinear process monitoring [15,16] and heterogeneous data fusion [16] .These findings inspire the investigation of improving process monitoring algorithms with extra layers of feature extraction, and the PCA-enhanced state-space ICA approach proposed in this work is an example.
An existing issue with these multi-layer algorithms is their interpretability.The features obtained may be more representative and sensitive to fault occurrence; the transparency of algorithm, however, will be sacrificed due to multiple layers of projection.As an example of datadriven fault diagnosis methods, contribution plots [17] have been extensively applied to identifying variables associating to a certain fault and locating the fault.The contribution plots root in the idea that the process variable that has a significant impact on the fault is supposed to have larger contributions to the monitoring statistics.Therefore, studies and applications of this idea are persistent and profound [18][19][20] .The mathematical formulation of contribution also provides flexibility for extensions in different scenarios, such as generalized contribution plots [21] , reconstruction-based contributions [22] and contri-bution rate [23] .Nevertheless, how to extract contributions from multi-layer algorithms is still not well understood.This work exploits the usage of contribution plots in faulty variable identification when multi-layer linear feature extraction algorithms are adopted in multivariate statistical process monitoring.A general structure of contribution propagation analysis is proposed accordingly.This analysis takes a step back to the original definition of contribution and investigates the way it propagates between layers with respect to the mathematical formulation of different multi-layer linear feature extraction algorithms so as to quantify the influence of original variables on the final monitoring statistics.In the fault diagnosis stage, once the contributions propagating through the intermediate layers have been quantitatively analyzed, it is reasonable to infer that contribution of influential variables will be properly emphasized and these variables can be identified.Applying the proposed analysis, the contribution propagation solution is derived for the following linear multi-layer feature extraction algorithms: PCA-enhanced CVA, state-space ICA, and PCA-enhanced state-space ICA.
As an extension to a preliminary study on this topic [24] , in which a two-layer contribution analysis for PCA-enhanced CVA was presented, this work has the following contributions.Firstly, a novel three-layer feature extraction algorithm, PCA-enhanced state-space ICA, is proposed as a more sophisticated and advanced feature extraction approach; its application in multivariate statistical process monitoring is presented along with PCAenhanced CVA and state-space ICA.Secondly, the contribution propagation framework is proposed for faulty variable identification by extending the contribution analysis adopted for PCA-enhanced CVA based process monitoring.By applying this framework, the applicability of contribution-based fault diagnosis methods will be enhanced and the interpretability of various multi-layer feature extraction algorithms in process monitoring will be improved significantly.Improved interpretability and faulty variable identification results of proposed PCA-enhanced state-space ICA and contribution propagation analysis can be demonstrated via well-acknowledged benchmark data set from Tennessee Eastman (TE) process; furthermore, these approaches are also feasible for real-life industrial plant monitoring.
The organization of this paper is as follows.The general structure of fault detection algorithms based on multilayer linear feature extraction is revisited in Section 2.1 along with the formulation of PCA-enhanced CVA, statespace ICA, and PCA-enhanced state-space ICA.For fault diagnosis, contribution propagation analysis is proposed for contribution plots calculation in these multi-layer linear feature extraction based monitoring techniques and the solutions to these contribution plots in Section 3. Section 4 presents two case studies: 1) The simulated case study of Tennessee Eastman process; 2) A case study on the benchmark data set collected from the multiphase flow facility.The performance of identifying influential variables based on their contributions using the following linear algorithms are compared in succession: CVA with ordinary contribution plots (single-layer) [18] ; PCA-enhanced CVA, state-space ICA with contribution propagation (2-layer); and PCA-enhanced state-space ICA with contribution propagation (3-layer).Section 5 summarizes the findings in this work and illustrates potential directions of extension in future study.

Multiple layer linear algorithms for process monitoring
Fig. 1 represents the general formulation of multi-layer linear feature extraction based fault detection algorithm.
is the original data, are the intermediate features obtained by each feature extraction layer, and are the representative features and residuals, respectively.The monitoring statistic is calculated using and .
PCA-enhanced CVA, state-space ICA and the new PCA-enhanced state-space ICA will be introduced next as three examples of multi-layer linear algorithms shown in Fig. 1.

PCA-enhanced CVA v r
According to [12], CVA fault detection algorithm is enhanced by using latent variables extracted by PCA as its input instead of the original measured variables.Firstly, PCA projects the original measurement data indimensional variable space to a reduced -dimensional principal component space with maximum explanation of variations in original variables.The model structure of PCA is illustrated as follows: where with zero mean and unit variance is the standardized original data set in the Euclidean space , is the extracted principal components and is the projection matrix.The projection matrix is obtained by eigenvalue decomposition of sample covariance matrix .Hence, the principal component vector is linear projection of original variable vector .

X Y
CVA is a linear dynamic feature extraction method from which the canonical variates with maximum correlation between past and future vectors can be acquired.Instead of using original data matrix , PCA-enhanced CVA algorithm adopts the principal components extracted by PCA as the input to CVA and get canonical where such that is the mean of principal components over time.
Furthermore, past and future Henkel matrices and comprise of past/future vector pairs, making the time lagged data matrices for feature extraction starting at time stamp (the minimal initial time point for constructing past vector): The time lags, and , can be estimated by calculating the auto-correlation of time-lagged variable vectors.Analogically to PCA, the quasi-covariance matrix is defined by covariance and cross-covariance matrices of and : J L H Consequentially, the projection matrices and are the normalized results of singular value decomposition result of : ) The canonical variate vector and residual vector are both linear projections of past vector at time :

State-space ICA
Another example of multi-layer linear feature extraction algorithms is the state-space ICA [13] .Following the algorithm structure in Fig. 1, CVA comes as the first layer for handling process dynamics and ICA is conducted in succession to both canonical variate space and residual space to account for non-Gaussianity.These independent components are used for monitoring statistics calculation.
In previous section, (8) have been derived for calculating canonical variates and residuals in CVA approach.Further linear projection of canonical variate space and residual space in order to maximize the independence between features are defined in (9) in state-space ICA:

Wz We
The explicit solution to the projection matrices and , such that the independence of individual components is maximized, is not available; instead, iterative optimization approaches like FastICA [25] may be used.

PCA-enhanced state-space ICA
The development of PCA-enhanced CVA and statespace ICA indicates that applying multiple feature extraction layers successively may improve the performance of monitoring algorithms.Therefore, PCA-enhanced state-space ICA with 3 layers of feature extraction is proposed as a further extension: 1) Apply PCA to the original data set for dimensionreduced principal components.
2) Apply CVA to the Henkel matrix constructed by principal components in order to handle process dynamics.
3) Apply ICA to the canonical variates and residuals for extraction of representative features.
The structure of PCA-enhanced state-space ICA algorithm can also be summarized using Fig. 1.

Fault detection stage
After features have been extracted from original data, monitoring statistics are to be calculated using these features and compared with their control limits for fault detection.Qin [26] has studied a variety of monitoring metrics in data-driven process monitoring.The most widely used ones among all are the statistics for detecting systematic variation using representative features (e.g., canonical variates in CVA and independent components in ICA) and statistics for random error using residuals, shown in (10).
Based on normal data, upper control limits with confidence level , i.e., and , of statistics in (10) are defined as Due to the potential non-Gaussianity of the process variables, the distribution functions in (11) and corresponding control limits are estimated via kernel density estimation [27] .In online fault detection, monitoring statistics calculated with the real-time measurements are compared with these control limits to determine the fault occurrence based on the condition shown in (12).

Contribution propagation analysis
This section discusses the general formulation of contribution plots under multi-layer feature extraction framework and derives the solution to PCA-enhanced, state-space ICA, and PCA-enhanced state-space ICA methods.

Contribution propagation
Equation (13) formulates the structure of a general 2-layer linear feature extraction based fault detection method. where x: original measured variable vector; P : first layer projection matrix; y: intermediate features from first layer; J : second layer projection matrix for feature variables representing systematic error; L : second layer projection matrix for residuals representing random error; z, e: features from second layer representing systematic and random error; , Q: monitoring statistics for systematic and random error.

), e(2), • • •, e(n)] T z e
In this formulation, the original data set is initially processed by the first feature extraction method to obtain the intermediate feature data .is further processed by the second layer of feature extraction method to attain the feature variables and residual variables .The final monitoring statistics are based on and .It is obvious that aforementioned PCA-enhanced CVA and Statespace ICA fall into this category.
The objective of contribution plots-based fault diagnosis is to gain the contribution of original process variables to the final monitoring statistics such as and .In order to do so, Fig. 2 illustrates the propagation of variable contributions under this multi-layer feature extraction framework.
When the number of intermediate layers increases, the propagation procedure of contributions will remain the same.Hence, this framework also applies to PCA-enhance State-space ICA algorithm which has three layers of feature extraction.
The following equations hold for individual variables , and : ; ; . Analogy can be made for the contribution plots to statistics.The general philosophy behind is to calculate the weighted com- In general, the individual contributions of single variables are displayed in (14), where and are intermediate coefficients with respect to and , respectively.
PCA, CVA and ICA are all linear layers; therefore, a close form solution to the complete contribution plots of original variables to monitoring statistics can be derived explicitly.

Contribution calculation for PCA-enhanced CVA-based monitoring
Following the general formulation proposed previously, the contribution plots of this PCA-enhanced CVA algorithm can be derived in 4 steps.
x y 1) Contribution of the original process variable vector to the principal components .The general process model of PCA is where and is the i-th process variable, whilst is the i-th row vector of .Therefore, for each variable , its contribution to the entire principal components is calculated by (16): y 2) Contribution of principal components to canonic- where is the coefficient of the -th principal component with time delay with respect to the -th canonical variate .and are the projection matrices obtained by CVA.z e 3) Contribution of canonical variates and residuals to monitoring statistics: According to (10), the monitoring statistics can be decomposed into summations of squared terms of individual features, as shown in (18).
4) Contribution of the -th process variables to monitoring statistics: Noticing that based on ( 16), the contribution of to at time is , the contribution of variable to and at time can be obtained by (19) and (20).
where the is the t-th column vector of past Hankel matrix of which is constructed the same way as .

Contribution calculation for statespace ICA based monitoring
Similarly, the contribution propagation for state-space ICA can be obtained: Contribution of the original process variable vector to the canonical variates and residuals : Similarly to the second step in PCA-enhanced CVA, (21) defines the contribution of the -th process variable to the -th canonical variate and the -th residual : 2) Contribution of and to independent components and : Since both CVA and ICA are linear projections of original features, the contribution calculation resembles the procedure in (16).
4) Contribution of original variables to monitoring statistics: The contribution of variable to and at time can be obtained by ( 24) and (25).

wz;α w e;β
Wz We where and are row vectors of and .Note that due to the mathematical formulation of CVA and ICA, the covariance matrix of extracted features is diagonal.Therefore, the cross terms of canonical variates/independent components do not exist in monitoring statistics and these statistics can be easily decomposed into summation of contributions of individual features.In other linear algorithms of which the covariance matrix is not diagonal, the contribution propagation framework will result in an approximated solution.

Contribution calculation for PCA-enhanced state-space ICA based monitoring
Similarly to PCA-enhanced CVA and state-space ICA, contribution propagation analysis calculates the contributions propagated in each layer and estimates the contribution of original variables to final monitoring statistics by synthesizing the propagated contributions when PCAenhanced state-space ICA is adopted.To be brief, the procedure is summarized as follows: 1) Contribution of original process variables to prin-cipal components; 2) Contribution of principal components to canonical variates and residuals; 3) Contribution of canonical variates and residuals to corresponding independent components; 4) Contribution of independent components to monitoring statistics; 5) Contribution of original process variables to monitoring statistics.

Algorithm summary
To summarize, the online fault detection and diagnosis procedure based on multi-layer linear feature extraction techniques is follows: x z * e * 3) Calculate the monitoring statistics using and ; x * i 4) If the monitoring statistics exceed their control limits, calculate the contribution of all to the monitoring statistics propagated through multiple layers so as to identify the influential variables.

Case studies
In this section, the proposed contribution propagation framework will be validated using two case studies: a simulated benchmark case study of the Tennessee Eastman process and a benchmark data set collected by experiments on a multiphase flow facility.

Process description
As a simulated case study, the Tennessee Eastman process (TEP) has been widely acknowledged as a bench-mark for development and validation of process monitoring and control algorithms.The process configuration of TEP used in this work is inherited from [28].Simulated data set consisting of 22 measured variables is used for training and validation; the tag names and descriptions of these variables are presented in Table 1.
To validate, a training set of 1 000 samples is collected in normal operating condition.For validation, two extra data sets of 1 000 samples each are generated under in presence of two fault scenarios (IDV (11) and IDV( 13) in [28]), respectively.The fault descriptions are provided Table 2.The faulty period starts at Sample 200 and ends at 1 000.

Fault diagnosis results
It has been demonstrated in previous works that PCA-enhanced CVA and state-space ICA can improve fault detection performance on the TEP data set by comparing with CVA and dynamic extensions of PCA and ICA [12,13] .Therefore, this section focuses on the fault diagnosis performance obtained by contribution propagation framework used in multi-layer linear feature extractions and the ordinary contribution plots applied to CVAbased monitoring.
The contribution plots of all 4 monitoring algorithms for both faults are presented in Figs. 3 and 4 in the order of the number of layers increases.
Table 3 further summarizes the influential variables identified by their contributions in CVA, PCA-enhanced CVA, state-space ICA, and PCA-enhanced state-space ICA.The union set of variables that rank the top 2 in their contributions to and statistics is identified as influential variables.A more systematic way of identifying influential variables using the control limits of variable contributions [21] can be considered as a potential extension.
For IDV (11), it has been agreed by the contribution plots of all 4 algorithms that XMEAS(9), the reactor temperature, is the most directly influenced variable when the reactor cooling water inlet has large random Table 1 Process variables dominant in the separator or stripper, the pressure variation in the stripper will be less significant.Instead, the stripper level (XMEAS(15)) will be influenced owing to the insufficient reaction.Variations in temperature also propagates from reactor (XMEAS( 11)) to the stripper (XMEAS( 18)) due to the exothermic nature of this reaction.These variables indicate that this fault impacts the condition in reactor and other components downstream and therefore restrain the area of fault existence.Fig. 5 visualizes the trend plot influential variables identified by state-space ICA (XMEAS (7) and XMEAS (11)) and a clear trend of variation can be observed after the fault was seeded.
It can be concluded that, in this simulated case study, the proposed contribution propagation framework provides a fault isolation approach for PCA-enhanced CVA, state-space ICA, and PCA-enhanced state-space ICA and identifies influential variables.While the fault detection performance is improved by multiple layers of feature extraction, the contribution of original measured variables can still be estimated via contribution propagation and hence facilitates fault diagnosis afterwards.

Multiphase flow benchmark 4.2.1 Process description
The multiphase flow facility in the Process System Engineering Lab of Cranfield University is a unique industrial-scale rig for researches and experiments on measuring, monitoring and control of multiphase flows.Water, oil and air are supplied from individual pipelines; by converging and intersection of pipelines, 3-phase flows are mixed, making a multiphase flow with liquid and gas.The multiphase flow is transported, measured, separated and recycled successively afterwards.Being fully automated, this facility can operate in multiple normal operating conditions as well as simulate various faulty scenarios with manually seeded faults.It is also well equipped with measurement instrumentations which contain both regular process variables such as pressure and temperature, and mechanical condition variable such as pump current.All measurement data are collected in real-time and recorded by DeltaV system for further analysis.A more detailed description of this benchmark case study and previous work on statistical monitoring of it can be found in [4] and [30].
The schematic with the layout of measurement instrumentations of this facility is shown in Fig. 6.A total of 23 process variables are measured and recorded in the benchmark data set and variable descriptions are provided by Table 4.

Fault diagnosis results
In the preliminary fault detection step, CVA, PCA-enhanced CVA, state-space ICA, and PCA-enhanced statespace ICA algorithms are used for calculating the monitoring statistics separately; the propagated contributions of original variables to final monitoring statistics in multilayer algorithms are calculated with respect to Section 3. The results obtained from proposed framework are compared with those obtained directly by CVA-based monitoring using original contribution plots [4] .Fig. 7 compares the contribution plots of all variables obtained by CVA, PCA-enhanced CVA, state-space ICA, and PCA-enhanced state-space ICA in Fault 1. Their results in Fault 2 are compared in Fig. 8.To summarize the fault diagnosis results suggested by contributions, Table 5 shows the influential variables identified by contributions of variables using different monitoring algorithms.
For Fault 1, pipeline blockage is mimicked by turning the control valve on the input pipeline to top riser (VC404) gradually.According to the contribution plots, PCA-enhanced CVA and PCA-enhanced state-space ICA suggest the differential pressure over VC404 (V7) is the most influential variable while sate-space ICA and CVA both identify the riser top pressure (V3) as an influential variable too.Noticing that the valve opening of VC404 is not involved as process variable, the pressure drop over this valve and pressure measurements in adjacent to it will be the proper indicator of pipeline blockage.Therefore, the influential variables, which the deviation and original contribution plots both agree upon, are the most relevant ones among all with respect to Fault 1 and the results of contribution-based fault diagnosis are convincing.For Fault 2, due to the insufficient air/water supply, liquid accumulates at the bottom of the riser and blocks the gas transportation, causing the bottom pressure to increase.The liquid level will continue rising until blocked gas penetrates and results in liquid blow out at the riser top.Remaining liquid will fall back to the riser bottom and accumulate again.From process data perspective, the slugging fault will cause large fluctuations in process variables, such as pressure, flow rate and density, at both riser top and riser bottom.However, unlike single variable faulty scenario, the identified influential variables from different algorithms in slugging condition are no longer unified.For instance, the differential pressure between riser bottom and top (V6), riser bottom pressure (V2), and riser top density (V13) are identified commonly while variables in vicinity, such as riser top flow rate (V10) and riser top pressure (V3), are also suggested by different algorithms.Roughly speaking, all vari-ables with large contributions are in line with the fact that variables at riser top and bottom will be mostly influenced when slugging occurs; while the contribution of specific measurements will be determined by the corresponding feature extraction algorithm applied.
It can be concluded that the proposed contribution propagation framework is capable of estimating the contribution of original variables in monitoring statistics when multi-layer linear feature extraction method is employed in monitoring statistics calculation; therefore it is suitable for fault diagnosis and isolation of relevant variables in these situations.The fault diagnosis performance can be maintained as the fault detection performance is improved by substituting simple feature extraction methods with multi-layer feature ones.In the meantime, variations may exist in diagnosis results due to different feature extraction methods applied, especially in presence of complicated fault that influences multiple process vari-

Discussions
The case study further demonstrates the fault isola-tion ability of proposed contribution propagation approach for PCA-enhanced CVA and state-space ICA in data set collected from real process.Moreover, the solution can be visualized and interpreted the same way as the ordinary contribution plots in basic feature extraction approaches.On the other hand, there still exists potential of improvement for detection and diagnosis of the slugging fault.
analysis, which is a novel fault diagnosis approach based on the original contribution concept, for fault diagnosis and influential process variable identification that applies when multi-layer linear feature extraction algorithms are used for fault detection.As a more advanced example of multi-layer linear algorithms, PCA-enhanced state-space ICA has also been proposed in addition to existing PCAenhanced CVA and state-space ICA.By the validation of the TE data and the multiphase flow facility data, it has been demonstrated that this contribution propagation analysis is compatible with various types of multi-layer feature extraction algorithms and the interpretability of the monitoring results obtained by these existing algorithms has been improved significantly.
The gap of contribution plots-based fault diagnosis still exists in this multi-layer contribution propagation framework for monitoring of processes and faults with extra complexity due to the mathematical complexity of nonlinear approaches adopted, such as kernel transformation.Therefore, it is worthwhile considering the propagation of contribution plots in context of "kernelized" and other advanced feature extraction methods so as to provide a general solution to the contribution plots of variables for different monitoring techniques in the future.

Fig. 5
Fig. 5 Trend plot of influential variables

Fig. 6
Fig. 6 Schematic of the multiphase flow facility

Table 3
Fault isolation

Table 4
Measured variables in multiphase flow facility