Identification of the key manufacturing parameters impacting the prediction accuracy of support vector machine (SVM) model for quality assessment

Abstract

In the context of manufacturing, support vector machines (SVM) are commonly used to predict quality, i.e., predict the characteristics of a product according to the manufacturing parameters. The prediction accuracy of a SVM model is affected by a number of factors: training set size, data set quality, etc. Manufacturing datasets are usually prone to measurement uncertainties. Such uncertainties affect the observed values of the manufacturing parameters, thereby affecting the predictive performance of the SVM. To address this issue, several works in the literature have been proposed to improve the robustness of SVM to measurement uncertainties. These works, however, do not evaluate the contribution of the uncertainties of each parameter to the overall impact. For this reason, this paper focuses on quantifying the impact of the uncertainties of each parameter on the accuracy of the SVM prediction. Three approaches are proposed to do so. The first two approaches are based on Monte-Carlo simulation and allow providing quantitative measures that represent the impact of the uncertainties of each manufacturing parameter on the accuracy of the SVM. On the other hand, the third approach relies on simple statistical tools in order to estimate the impact of the uncertainties of each parameter. The proposed approaches would eventually make it possible to identify the uncertainties of the parameters that mostly affect the SVM. Such parameters are referred to as key measurement uncertainties. Identifying the key measurement uncertainties would provide a better understanding of how the SVM is affected by uncertainties, as it would provide a strong basis for improving the robustness of SVM in future works. The proposed approaches are applied to four datasets, and their performances are discussed and compared.

Appendices

Appendix A: Support vector machine

SVM was first introduced for the classification of linearly separable binary problems. Given a set of N observations $D = \left\{ {\left( {x_{1} ,y_{1} ), \ldots ,(x_{N} ,y_{N} } \right)} \right\}$, where $x_{i}$ represents the ith training data and $y_{i}$ is the class label of $x_{i}$, the SVM tries to find a hyperplane that separates the set space into two areas, each of which includes individuals from one of the two classes. The separating hyperplane is given as $w^{T} x + b = 0$, where $b$ is the bias of the hyperplane, $x$ are the data located within the hyperplane, and $w{ }$ are the weights that determine the hyperplane’s orientation. SVM seeks to maximize the margin by establishing formulation (Eq. 4) that allows defining the parameters of the optimal hyperplane [39]

$$ \min \quad \frac{1}{2}w^{2} $$

$$ s.t\quad y_{i} \left( {w^{T} \times x_{i} + b} \right) \ge 1\quad \forall i = 1, \ldots ,N $$

$$ y_{i} \in \left\{ { - 1,1} \right\} $$

$$ w \in {\mathbb{R}}^{d} ;b \in {\mathbb{R}} $$

(4)

Additionally, the linear SVM formulation might be useful to deal with the classification of some non-linearly separable problems. This can be done by allowing the constraint (of formulation 3) to be violated by some datapoints. To this end, the formulation of the SVM is modified so that it can accept that the margin is crossed, and that certain datapoints are on the wrong side of the hyperplane. A new variable “${\upxi }_{i}$” associated to each datapoint is therefore introduced to inform how far this point violates the constraint. This new variable, called the slack variable, can be either:

• ξ = 0: if the data point complies with the constraint.

• ξ > 1: if the data point is misclassified.

• 0 < ξ < 1: if the data point has crossed the margin.

Consequently, the new formulation (Eq. 5) aims to minimize the sum of errors $\left( {\sum\nolimits_{i = 1}^{n} {{\upxi }_{i} } } \right)$ in addition to $w^{2}$, while it introduces a new regularization parameter “C” that manages the compromise between the margin width and the relaxation of the constraint.

$$ \min \quad \frac{1}{2}w^{2} + C\mathop \sum \limits_{i = 1}^{n} {\upxi }_{i} $$

$$ s.t\quad y_{i} \left( {w^{T} \times x_{i} + b} \right) \ge 1\quad \forall i = 1, \ldots ,N $$

$$ {\upxi }_{i} \ge 0 $$

(5)

$$ y_{i} \in \left\{ { - 1,1} \right\} $$

$$ w \in {\mathbb{R}}^{d} ;b,{\upxi }_{i} \in {\mathbb{R}} $$

However, to address most of the non-linearly separable problems, the kernel trick method is adopted to create non-linear SVM models. The kernel trick consists on mapping the datapoints to a larger dimensional space called the feature space $\phi :R^{d} \to R^{p}$ which is equivalent to replacing each datapoint “$x_{i}$” by its image “$z_{i} = \phi (x_{i} )$” in the new space. The motivation of applying the kernel trick comes from the fact that data which is not separable in the input space can always be separated in a space of high enough dimensionality. This data transformation refers us to the new formulation (Eq. 6).

$$ \min \quad \frac{1}{2}w^{2} + C\mathop \sum \limits_{i = 1}^{n} {\upxi }_{i} $$

$$ s.t\quad y_{i} \left( {w^{T} \times \phi \left( {x_{i} } \right) + b} \right) \ge 1\quad \forall i = 1, \ldots ,N $$

$$ {\upxi }_{i} \ge 0 $$

$$ y_{i} { } \in \left\{ { - 1,1} \right\} $$

(6)

To compute the SVM formulations, the dual formulation is used. The formulations seen above are constrained quadratic optimization problems that are written as follows (Eq. 7), and can be reformulated as the equation (Eq. 8) using Lagrange multipliers “αi”:

$$ \min \quad \frac{1}{2}u^{T} Qu + p^{T} u $$

(7)

$$ s.t\quad a^{T} u \ge c $$

$$ \min \quad \frac{1}{2}u^{T} Qu + p^{T} u + \mathop \sum \limits_{i} {\upalpha }_{i} (c - a_{i}^{T} u) $$

(8)

Similarly, the formulation of the SVM can be written:

$$ L\left( {w,b} \right) = \frac{1}{2}w^{T} w +\sum \limits_{i} {{ \alpha }}_{i} (1 - y_{i} (w^{T} x_{i} + b)) $$

(9)

At the minimum, we can take the derivatives with respect to the primal variables “w” and “b” and set these to zero:

$$ \frac{\partial L}{{\partial b}} = - \mathop \sum \limits_{i = 1}^{n} \alpha_{i} y_{i} = 0 $$

(10)

$$ \frac{\partial L}{{\partial w}} = w - \mathop \sum \limits_{i = 1}^{n} \alpha_{i} y_{i} x_{i} = 0 $$

(11)

By replacing the two previous formulas in the formulation (8):

$$ \min \quad \frac{1}{2}\mathop \sum \limits_{i = 1}^{n} \mathop \sum \limits_{j = 1}^{n} y_{i} y_{j} \alpha_{i} \alpha_{j} x_{i}^{T} x_{j} - \mathop \sum \limits_{i = 1}^{n} \alpha_{i} $$

$$ s.t\quad \mathop \sum \limits_{i = 1}^{n} y_{i} \alpha_{i} $$

(12)

$$ \alpha_{i} \ge 0 $$

This formulation has a major advantage when using the kernel trick because the functional form of the mapping $\phi ({\text{x}}_{i} )$ does not need to be known since it is implicitly defined in the feature space by the choice of the kernel function $K\left( {x_{i} \times x_{j} } \right) = \phi (x_{i} ) \times \phi (x_{j} ).$

Appendix B: Sobol sensitivity analysis

Sobol’s analysis is one the different methods to analyse the global sensitivity of a model. It is based on variance decomposition techniques to provide a quantitative measure of the contributions of the input to the output variance. The decomposition of the output variance in a Sobol sensitivity analysis employs the same principal as the classical analysis of variance in a factorial design, which allows representing the output’s variance as follow:

$$ V = \mathop \sum \limits_{i = 1}^{p} V_{i} + \mathop \sum \limits_{1 \le i \le j \le p}^{p} V_{ij} + \cdots + V_{1 \ldots p} $$

(13)

where: V: Total variance of the model output. ${\text{V}}_{{\text{i}}}$: The first order contribution of the ith model parameter. ${\text{V}}_{{{\text{ij}}}}$: The contribution of the interaction of the ith and jth parameters.

Based on this decomposition, Sobol defines first order sensitivity indices, Eq. 14, as well as higher-order sensitivity indices Eq. 15, such as:

$$ S_{i} = \frac{{V_{i} }}{V} $$

(14)

$$ S_{ij} = \frac{{V_{ij} }}{V};\quad S_{ijk} = \frac{{V_{ijk} }}{V} $$

(15)

Also, to measure the total sensitivity of the variance Y due to a variable Xi, Homma and Saltelli [40] introduced Total-effect indices Eq. 16 defined as the sum of the contribution caused by Xi and by its interactions -of any order- with any other input variables.

$$ S_{{T_{i} }} = \mathop \sum \limits_{k\# i} S_{k} $$

(16)

where #i represents all sets containing the index i.

One of the ways to estimate Sobol indices is by using the Monte-Carlo simulation. The Monte-Carlo estimation consists on estimating: the output expected value (E[Y]), the output variance (V[Y]), and both quantities ($\^U_{i}$) and ($\^U_{\sim i}$). The estimation of these four quantities requires two -samples $\tilde{X}_{\left( N \right)}^{\left( 1 \right)} = (x_{k1}^{\left( 1 \right)} , \ldots ,x_{kp}^{\left( 1 \right)} )_{k = 1..N}$ and ${ }\tilde{X}_{\left( N \right)}^{\left( 2 \right)} = (x_{k1}^{\left( 2 \right)} , \ldots ,x_{kp}^{\left( 2 \right)} )_{k = 1..N}$, using the following formulas:

$$ \^E = E\left[ Y \right] = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} f(x_{k1} , \ldots ,x_{kp} ) $$

(17)

$$ \hat{V} = V\left[ Y \right] = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} f^{2} (x_{k1} , \ldots ,x_{kp} ) - \^E^{2} $$

(18)

$$ \^U_{i} = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} f(x_{k1}^{\left( 1 \right)} , \ldots ,x_{{k\left( {i - 1} \right)}}^{\left( 1 \right)} ,x_{ki}^{\left( 1 \right)} ,x_{{k\left( {i + 1} \right)}}^{\left( 1 \right)} , \ldots ,x_{kp}^{\left( 1 \right)} ).f\left( {x_{k1}^{\left( 2 \right)} , \ldots ,x_{{k\left( {i - 1} \right)}}^{\left( 2 \right)} ,x_{ki}^{\left( 1 \right)} ,x_{{k\left( {i + 1} \right)}}^{\left( 2 \right)} , \ldots ,x_{kp}^{\left( 2 \right)} } \right) $$

(19)

$$ \^U_{\sim i} = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} f(x_{k1}^{\left( 1 \right)} , \ldots ,x_{{k\left( {i - 1} \right)}}^{\left( 1 \right)} ,x_{ki}^{\left( 1 \right)} ,x_{{k\left( {i + 1} \right)}}^{\left( 1 \right)} , \ldots ,x_{kp}^{\left( 1 \right)} ).f\left( {x_{k1}^{\left( 1 \right)} , \ldots ,x_{{k\left( {i - 1} \right)}}^{\left( 1 \right)} ,x_{ki}^{\left( 2 \right)} ,x_{{k\left( {i + 1} \right)}}^{\left( 1 \right)} , \ldots ,x_{kp}^{\left( 1 \right)} } \right) $$

(20)

where “f” is the model that links the inputs to the output.

Based on the previous quantities, the Sobol indices can be estimated in the following manner:

• First order sensitivity indices: $\hat{S}_{i} = \frac{{\^U_{i} - \^E^{2} }}{{\hat{V}}}$

• Total-effect indices: $\hat{S}_{Ti} = 1 - \frac{{\^U_{\sim i} - \^E^{2} }}{{\hat{V}}}$