A penalized blind likelihood Kriging method for surrogate modeling

Abstract

Surrogate modeling is commonly used to replace expensive simulations of engineering problems. Kriging is a popular surrogate for deterministic approximation due to its good nonlinear fitting ability. Previous researches demonstrate that constructing an appropriate trend function or a better stochastic process can improve the prediction accuracy of Kriging. However, they are not improved simultaneously to estimate the model parameters, thus limiting the further improvement on the prediction capability. In this paper, a novel penalized blind likelihood Kriging (PBLK) method is proposed to obtain better model parameters and improve the prediction accuracy. It improves the trend function and stochastic process with regularization techniques simultaneously. First, the formulation of the penalized blind likelihood function is introduced, which penalizes the regression coefficients and correlation parameters at the same time. It is a general expression and therefore can incorporate any type of penalty functions easily. To maximize the penalized blind likelihood function effectively and efficiently, a nested optimization algorithm is proposed to estimate the model parameters sequentially with gradient and Hessian information. As different regularization parameters can lead to different optimal model parameters and influence the prediction accuracy, a cross-validation-based grid search method is proposed to select good regularization parameters. The proposed PBLK method is tested on several analytical functions and two engineering examples, and the experimental results confirm the effectiveness of the proposed method.

Appendix

The gradient and Hessian matrix of the negative profile penalized blind likelihood function are derived in the following. We first assume that the regression coefficients β and process variance σ² are both fixed. Then, the gradient can be written as follows:

$$ \begin{array}{@{}rcl@{}} \frac{\partial Q}{\partial \theta_{j}}\!&=&\!-\frac{1}{2|\sigma^{2}\mathbf{R}|}|\sigma^{2}\mathbf{R}| \text{tr}((\sigma^{2}\mathbf{R})^{-1}\sigma^{2}\dot{\mathbf{R}}_{j}) \\ &&+\frac{1}{2}\mathbf{e}^{\text{T}}(\sigma^{2}\mathbf{R})^{-1}\sigma^{2}\dot{\mathbf{R}}_{j}(\sigma^{2}\mathbf{R})^{-1}\mathbf{e}+N\dot{p}_{j}^{\mu} \\ &=\!&-\frac{1}{2}\text{tr}((\mathbf{R})^{-1}\dot{\mathbf{R}}_{j})+\frac{1}{2}\text{tr}((\sigma^{2}\mathbf{R})^{-1}\mathbf{e}\mathbf{e}^{\text{T}}\mathbf{R}^{-1}\dot{\mathbf{R}}_{j})\!+N\dot{p}_{j}^{\mu} \\ &=&-\frac{1}{2}\text{tr}(\mathbf{R}^{-1}\{\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}-\mathbf{R}\}\mathbf{R}^{-1}\dot{\mathbf{R}}_{j})\\ &&+N\dot{p}_{j}^{\mu} ,\quad \text{\!for} \ j=1,...,D , \end{array} $$

(37)

where e = y −Fβ, $\dot {\mathbf {R}}_{j}=\partial \mathbf {R}/\partial \theta _{j}$, $\dot {p}_{j}^{\mu }=\partial p^{\mu }/\partial \theta _{j}$, and tr(⋅) is the trace of a matrix. Furthermore, the Hessian matrix can be given as follows:

$$ \begin{array}{@{}rcl@{}} \frac{\partial^{2} Q}{\partial \theta_{j} \partial \theta_{m}}&=& -\frac{1}{2}\text{tr}((-\mathbf{R}^{-1}\dot{\mathbf{R}}_{m}\mathbf{R}^{-1}\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}\mathbf{R}^{-1}\\&&-\mathbf{R}^{-1}\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}\mathbf{R}^{-1}\dot{\mathbf{R}}_{m}\mathbf{R}^{-1} \\ &&+\mathbf{R}^{-1}\dot{\mathbf{R}}_{m}\mathbf{R}^{-1})\dot{\mathbf{R}}_{j}+\mathbf{R}^{-1}\{\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}\\&&-\mathbf{R}\}\mathbf{R}^{-1}\ddot{\mathbf{R}}_{jm})+N\ddot{p}_{jm}^{\mu} \\ &=&\ \frac{1}{2}\text{tr}(\mathbf{R}^{-1}\dot{\mathbf{R}}_{j}\mathbf{R}^{-1}\{2\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}-\mathbf{R}\}\mathbf{R}^{-1}\dot{\mathbf{R}}_{m}) \\ &&-\frac{1}{2}\text{tr}(\mathbf{R}^{-1}\{\sigma^{-2}\mathbf{e}\mathbf{e}^{\text{T}}-\mathbf{R}\}\mathbf{R}^{-1}\ddot{\mathbf{R}}_{jm})\\ &&+N\ddot{p}_{jm}^{\mu} , \quad \text{for} \ j,m=1,...,D , \end{array} $$

(38)

where $\ddot {\mathbf {R}}_{jm}=\partial ^{2}\mathbf {R}/\partial \theta _{j} \partial \theta _{m}$ and $\ddot {p}_{jm}^{\mu }=\partial ^{2} p^{\mu }/\partial \theta _{j} \partial \theta _{m}$.