Statistical learning and optimization of the helical milling of the biocompatible titanium Ti-6Al-7Nb alloy

Abstract

Helical milling has been applied for hole-making in titanium alloys, especially in the Ti-6Al-4V alloy, considering the aims of the aeronautic, automobile, and other sectors. When considering hole-making in Ti-alloys for biomedical applications, few studies have been carried out. Besides, intelligent approaches for modeling and optimization of this process in these special alloys are demanded to achieve the best results in terms of hole surface quality, and productivity. This work presents an approach for modeling and optimization of helical milling for hole-making of Ti-6Al-7Nb biocompatible titanium alloy. The surface roughness of the holes was measured to quantify the hole quality. Principal component analysis was performed for dimensionality reduction of the roughness outputs. For modeling, a learning procedure was proposed considering polynomial response surface regression, tree-based methods, and support vector regression. Cross-validation is used for learning and model selection. The results pointed out that the support vector regression model was the best one. Multi-objective evolutionary optimization was performed considering the support vector regression model and the deterministic model of the material removal rate. The Pareto set and the Pareto frontier were plotted and discussed concerning practical aspects of the helical milling process. The proposed learning and optimization approach enabled the achievement of the best results of the helical milling in the biocompatible Ti-6Al-7Nb alloy and can be applied to other intelligent manufacturing applications.

Appendix A: Statistical learning for regression modeling

Response surface methodology (RSM) has been widely applied for designed process modeling to enable optimization. Since RSM models are second-order multiple regression, it is based on the parametric classical estimation. Learning methods focus on non-parametric model estimation with good generalization capability as opposed to model estimation methodology under statistical assumptions. The theory of such approaches is called Statistical Learning Theory or Vapnik–Chervonenkis (VC) theory [54]. Statistical learning theory aims to describe the properties of learning machines to understand how they can generalize well in unseen data [55]. In this section, some statistical learning principles and modeling approaches will be summarized as alternatives for modeling (learning) and explainability.

1.1 A.1: Some learning principles

The statistical learning theory was developed from 1960 to 1990, but in the middle 1990s the theory was “confirmed” with the proposition of support vector machines, based on the developed theory [56, 57]. Considering a function estimation problem composed of the following: (i) a set of independently drawn random vectors x from unknown distribution P(x); (ii) a supervisor that returns an output vector y for each input vector x, according to a conditional fixed and unknown distribution P(y∣x); and (iii) a learning machine that is capable of implementing a set of functions f(x,α), α ∈Λ, where Λ is a set of parameters. The learning problem consists of selecting the best function, f(x,α), α ∈Λ, to approximate the supervisor’s response. This process is conducted based on a set of a training set of N iid observations of the studied phenomena, (x₁,y₁),..., (x_N,y_N), drawn according to P(x,y) = P(x)P(y∣x) [56, 58]. Let the supervisor’s answer, y, be a real value. The regression problem may be solved by minimizing the quadratic loss function of the Eq. 21 [56, 58].

$$ L(y, f(\mathbf{x,\alpha})) = (y - f(\mathbf{x,\alpha}))^{2} $$

(21)

The empirical risk minimization (ERM) principle of statistical learning is related to the minimization of such function considering empirical data. The least squares method is a classical approach of ERM for regression problems. Considering N observations of the pair of regressors’ vector and output, (x₁,y₁),..., (x_N,y_N), the least squares method seeks to minimize Eq. 22 [56, 58].

$$ R_{emp}(\mathbf{\alpha}) = \frac{1}{N}{\sum}_{i=1}^{N}(y_{i} - f(\mathbf{x}_{i},\mathbf{\alpha}))^{2} $$

(22)

The empirical risk function of Eq. 22 considers only the training set. It is, however, necessary to minimize the expected risk, $R(\mathbf {\alpha }) = {\int \limits } L(y, f(\mathbf {x,\alpha }))dP(\mathbf {x},y)$, which is impossible in practical problems, since it would only be achievable, if population data of the joint distribution is considered P(x,y). Therefore, it is important to know if the empirical risk is a good estimator of the expected risk. This will define how well the approximated function generalizes, i.e., which is the quality of the predictions of unseen or future data [56, 59].

Through these thoughts the learning process can be summarized in two important questions [60]:

• Is it possible to now if R_emp(x,α) is close enough of R(x,α)?

• Is it possible to make R(x,α) small enough?

The most simple way to answer these questions is through cross-validation. The next sections present some important statistical learning approaches for regression modeling.

1.2 A.2: Support vector regression

Support vector regression was proposed by Drucker et al. [57] as an approach for regression, based on the concept of support vectors. Through support vector regression, learning is achieved considering a small subset of the training data, called support vectors, and the optimal modeling of the support vectors generally guarantees optimal modeling of the available data. Let the training data be (x₁,y₁), (x₂,y₂),..., (x_N,y_N), i = 1,...,N, with x_i = [x_i1,x_i2,...,x_ik]. Support vector regression seeks to approximate a function with maximum deviation ε to the training data. In this fashion, there is no problem to have an error, since it is less than ε.

Let the estimation of a linear function, according to the Eq. 23.

$$ \hat{y}_{i} = \mathbf{x}_{i}^{T}\mathbf{w} + b, i = 1, ..., N $$

(23)

To get such approximation, a loss function, L, added by a regularization (shrinkage) term must be minimized, according to Eq. 24, where C is a regularization constant [58]. This formulation is similar to the rigid regression approach [61], if the loss function is a quadratic function of the error, ${\sum }_{i=1}^{N} L(y_{i}, \hat {y}_{i}) = {\sum }_{i=1}^{N} (y_{i} - \hat {y}_{i})^{2}$, generally referred as the residuals sum of squares (RSS). However, in the rigid regression original formulation the regularization constant multiplies the term w^Tw.

$$ C\sum\limits_{i=1}^{N} L(y_{i}, \hat{y}_{i}) + \mathbf{w}^{T}\mathbf{w} $$

(24)

In the case of the support vector regression, it was proposed an ε-insensitive loss function [57], presented in Eq. 25, with $\xi _{i}= \lvert y_{i} - \hat {y}_{i} \rvert - \varepsilon$. Figure 18 shows the loss ε-insensitive function.

$$ L = \begin{cases} 0 \text{, if } \lvert y_{i} - \hat{y}_{i} \rvert < \varepsilon\\ \lvert y_{i} - \hat{y}_{i} \rvert - \varepsilon \text{, otherwise} \end{cases} $$

(25)

Fig. 18

ε-insensitive loss (a); SVM regression model (b)

Therefore, the primal support vector regression estimation problem can be written as in the formulation 26 [58].

$$ \text{Min}\ \begin{Bmatrix} \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C{\sum}_{i=1}^{N}(\xi_{i} + \xi_{i}^{*}) \end{Bmatrix} \\ $$

$$ \textrm{s.t.: } \begin{cases} y_{i} - \left[\mathbf{x}_{i}^{T}\mathbf{w} + b\right] \leq \varepsilon + \xi_{i}\\ \left[\mathbf{x}_{i}^{T}\mathbf{w} + b\right] - y_{i} \leq \varepsilon + \xi_{i}^{*} \\ \xi_{i}, \xi_{i}^{*} \geq 0 \end{cases} $$

(26)

To solve this estimation problem more easily, the dual formulation can be considered. Besides, the dual formulation will enable the extent of support vector regression to non-linear problems. However, before, it is suitable to present the Lagrangian formulation to the primal support vector regression problem, as follows in Eq. 27, where $\alpha _{i}, \alpha _{i}^{*}, \eta _{i}$, and $\eta _{i}^{*}$ are the Lagrange multipliers, which must be non-negative [57, 58].

$$ \begin{array}{@{}rcl@{}} L &=& \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C\sum\limits_{i=1}^{N}(\xi_{i} + \xi_{i}^{*})\\ && - \sum\limits_{i=1}^{N} \alpha_{i}(\varepsilon + \xi_{i} - y_{i} + \mathbf{x}_{i}^{T}\mathbf{w} + b) \\ &&- \sum\limits_{i=1}^{N} \alpha_{i}^{*}(\varepsilon + \xi_{i}^{*} - \mathbf{x}_{i}^{T}\mathbf{w} - b + y_{i}) \\ &&-\sum\limits_{i=1}^{N}(\eta_{i}\xi_{i} + \eta_{i}^{*}\xi_{i}^{*}) \end{array} $$

(27)

By deriving the Lagrangian function with respect to the primal variables problem, w, b, ξ_i, and $\xi _{i}^{*}$, and setting it equal to zero, first-order optimality conditions are achieved as follows [55].

$$ \begin{array}{@{}rcl@{}} \frac{\partial L}{\partial b} &=& \sum\limits_{i=1}^{N}(\alpha_{i} + \alpha_{i}^{*}) = 0\\ \frac{\partial L}{\partial \mathbf{w}} &=& \mathbf{w} - \sum\limits_{i=1}^{N}(\alpha_{i} + \alpha_{i}^{*})\mathbf{x}_{i} = 0\\ \frac{\partial L}{\partial \xi_{i}} &=& C - \alpha_{i} - \eta_{i} = 0\\ \frac{\partial L}{\partial \xi_{i}^{*}} &=& C - \alpha_{i}^{*} - \eta_{i}^{*} = 0 \end{array} $$

(28)

Replacing the results of Eq. 28 in Eq. 26, it is achieved the dual formulation of the support vector regression problem, as follows [57, 58]

$$ \text{Max}\ \begin{Bmatrix} -\frac{1}{2}{\sum}_{i,j}^{N}(\alpha_{i} - \alpha_{i}^{*})(\alpha_{j} - \alpha_{j}^{*})\mathbf{x}_{i}\mathbf{x}_{j} - \varepsilon{\sum}_{i=1}^{N}(\alpha_{i} - \alpha_{i}^{*}) \\ + {\sum}_{i=1}^{N} y_{i}(\alpha_{i} - \alpha_{i}^{*}) \end{Bmatrix} \\ $$

$$ \textrm{s.t.: } \begin{cases} {\sum}_{i=1}^{N}(\alpha_{i} - \alpha_{i}^{*}) = 0\\ \alpha_{i}, \alpha_{i}^{*} \in [0,C] \end{cases} $$

(29)

In this dual formulation, w is rewritten as a linear combination of the training observations patterns, x_ix_j. In the dual formulation, η_i and $\eta _{i}^{*}$ were described considering C, α_i, and $\alpha _{i}^{*}$, and, therefore, were eliminated. Besides, $\mathbf {w} = {\sum }_{i=1}^{N}(\alpha _{i} + \alpha _{i}^{*})\mathbf {x}_{i}$. In this sense, in the support vector regression, the initial model presented in Eq. 23 can be rewritten as follows.

$$ \hat{y} = \sum\limits_{i=1}^{N}(\alpha_{i} - \alpha_{i}^{*})\mathbf{x}_{i}\mathbf{x} + b $$

(30)

This support-vector expansion means that the initial model terms w are rewritten as linear combinations of the training data, but the model includes only the support vectors, i.e. x_i such that α_i > 0 or $\alpha _{i}^{*} >0$, i = 1,...,N. Traditional statistical methods seek to approximate a function in the feature space. However, support vector regression seeks to increase the dimensionality in the support vector space [58].

Since the support vector regression method only depends on the dot product of the training data, x_i, to approximate more complex functions, the dot product x_ix_j may be replaced by a kernel, k(x_i,x_j). Some efficient options to the linear kernel, k(x_i,x_j) = x_ix_j, include the polynomial kernel, k(x_i,x_j) = (x_ix_j + c)^d, and the radial basis, $k(\mathbf {x}_{i},\mathbf {x}_{j}) = \exp (-\gamma ||\mathbf {x}_{i} - \mathbf {x}_{j}||^{2})$. Obviously, in the dual formulation, the term x_ix_j must be replaced by the chosen kernel, k(x_i,x_j), and the support vector regression model can be expressed according to Eq. 12. According to the selected kernel, the hyperparameters, such as γ for the radial kernel, and the regularization constant, C, must be chosen through cross-validation. In the fashion of the statistical learning principles presented these are the models’ parameters to be selected, α ∈Λ.

1.3 A.3: Tree-based models

The so-called classification and regression trees (CART) approach date at least as far as Morgan and Sonquist [62] as a sequential approach focusing on reducing the prediction error, which is independent of the extent of the linearity in the classifications or the order in which the explanatory factors are introduced. In the case of regression, the algorithm needs to decide which variable to split, from x₁ to x_k, and the splitting point, j, in each step of the procedure, aiming to reduce RSS [41].

Considering the case where the partitions result in M regions, R₁,...,R_m, and that the predicted value is a constant, c_m, in that region, m = 1,...,M, the regression tree model can be written as in Eq. 31, where c_m is the average of the response values of the training data in that region, R_m, and I(x ∈ R_m) is an indicator function which is equal to one if true and 0 on the contrary.

$$ \hat{y} = \sum\limits_{m=1}^{M} c_{m} I(\mathbf{x} \in R_{m}) $$

(31)

To achieve this kind of model, a recursive binary splitting procedure is followed [63]. The procedure seeks to find the predictor, x_j, j = 1,...,k, and the cut point s, which results in the lowest RSS. This binary partitioning, therefore, seeks to minimize the objective function Eq. 32 [41, 63, 64]. The approach is repeated in one of the partitions generated, in which a better decrease in RSS is achieved. The recursive binary splitting continues until a stopping criterion, such as a minimum number of training observations in each partition is reached.

$$ \underset{j,s} {\text{Min}} \bigg\{ \sum\limits_{i: \mathbf{x}_{i} \in R_{1}(j,s)} (y_{i} - c_{1})^{2} + {\sum}_{i: \mathbf{x}_{i} \in R_{2}(j,s)} (y_{i} - c_{2})^{2} \bigg\} $$

(32)

Figure 19 illustrates a tree model in function of two regressors. In this case, three partitions were performed, represented by the three nodes of the tree model representation, Fig. 19a. In this representation, the first node, is at the top, while the branches are at the bottom. Four regions were obtained in the regressors’ space through the recursive binary splitting, entailing four predicted values. These rectangular regions can be observed in Fig. 19b and 19c, in the distinct heights of the surface plot and in the distinct colors of the contour plot color scale, respectively.

Fig. 19

Regression tree model

A promising approach to improve the prediction of the regression tree method is the bagging or bootstrap aggregation method. Through bagged trees, several regression trees are obtained considering resamples of the training data, and the final bagging model is the average of these distinct regression trees obtained. Figure 20 illustrates the bagging procedure. Considering B bootstrap samples generated from the training sample, $\mathbf {Z}_{1}^{*}$, ..., $\mathbf {Z}_{B}^{*}$, it is obtained B models, $\hat {f}_{b}^{*}(x)$, b = 1,...,B. The final bagging model can be written according to Eq. 33. Each bootstrap tree will be different from the one estimated with the complete training data set, with distinct features and terminal nodes [41].

$$ \hat{f}_{bag}(x) = \frac{1}{B} \sum\limits_{b=1}^{B} \hat{f}_{b}^{*}(x) $$

(33)

Fig. 20

Bagging regression trees

A popular modification in the bagging approach for CART is the random forest, proposed by Breiman [65]. In the random forest approach, only a limited number of features or input variables are considered in each step of the recursive binary splitting procedure. This is done to overcome the possible correlation between the features and also to ‘decorrelate’ the B trees generated through bootstrapping of the training data. For regression it is recommended to consider m = k/3 features in each splitting [41, 65].