# Improving kriging surrogates of high-dimensional design models by Partial Least Squares dimension reduction

• Mohamed Amine Bouhlel
• Nathalie Bartoli
• Joseph Morlier
REVIEW ARTICLE

## Abstract

Engineering computer codes are often computationally expensive. To lighten this load, we exploit new covariance kernels to replace computationally expensive codes with surrogate models. For input spaces with large dimensions, using the kriging model in the standard way is computationally expensive because a large covariance matrix must be inverted several times to estimate the parameters of the model. We address this issue herein by constructing a covariance kernel that depends on only a few parameters. The new kernel is constructed based on information obtained from the Partial Least Squares method. Promising results are obtained for numerical examples with up to 100 dimensions, and significant computational gain is obtained while maintaining sufficient accuracy.

## Keywords

Kriging Partial Least Squares Experiment design Metamodels

## Symbols and notation

Matrices and

vectors are in bold type.

Symbol

Meaning

det

Determinant of a matrix

|⋅|

Absolute value

Set of real numbers

+

Set of positive real numbers

n

Number of sampling points

d

Dimensions

h

Number of principal components retained

x

d vector

xj

j th element of a vector x

X

n × d matrix containing sampling points

y

n × 1 vector containing simulation of X

x(i)

i th training point for i = 1,…,n (a 1 × d vector)

w(l)

d × 1 vector containing X weights given by the l th PLS iteration for l = 1,…,h

X(0)

X

X(l−1)

Matrix containing residual of inner regression of (l − 1)st PLS iteration for l = 1,…,h

k(⋅, ⋅)

Covariance function

(0, k(⋅, ⋅))

Distribution of a Gaussian process with mean function 0 and covariance function k(⋅, ⋅)

xt

Superscript t denotes the transpose operation of the vector x

## Notes

### Acknowledgments

The authors thank the anonymous reviewers for their insightful and constructive comments. We also extend our grateful thanks to A. Chiplunkar from ISAE SUPAERO, Toulouse and R. G. Regis from Saint Joseph’s University, Philadelphia for their careful correction of the manuscript and to SNECMA for providing the tables of experiment results. Finally, B. Kraabel is gratefully acknowledged for carefully reviewing the paper prior to publication.

## References

1. Alberto P, González F (2012) Partial Least Squares regression on symmetric positive-definite matrices. Rev Col Estad 36(1):177–192
2. Bachoc F (2013) Cross Validation and Maximum Likelihood estimation of hyper-parameters of Gaussian processes with model misspecification. Comput Stat Data Anal 66:55–69
3. Bishop CM (2007) Pattern recognition and machine learning (information science and statistics). SpringerGoogle Scholar
4. Braham H, Ben Jemaa S, Sayrac B, Fort G, Moulines E (2014) Low complexity spatial interpolation for cellular coverage analysis. In: 2014 12th international symposium on modeling and optimization in mobile, ad hoc, and wireless networks (WiOpt). IEEE, pp 188–195Google Scholar
5. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, Cambridge
6. Cressie N (1988) Spatial prediction and ordinary kriging. Math Geol 20(4):405–421
7. Damianou A, Lawrence ND (2013) Deep gaussian processes. In: Proceedings of the sixteenth international conference on artificial intelligence and statistics, AISTATS 2013, Scottsdale, pp 207–215Google Scholar
8. Durrande N (2011) Covariance kernels for simplified and interpretable modeling. A functional and probabilistic approach. Theses, Ecole Nationale Supérieure des Mines de saint-EtienneGoogle Scholar
9. Durrande N, Ginsbourger D, Roustant O (2012) Additive covariance kernels for high-dimensional gaussian process modeling. Ann Fac Sci Toulouse Math 21(3):481–499
10. Forrester A, Sobester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, New York
11. Frank IE, Friedman JH (1993) A statistical view of some chemometrics regression tools. Technometrics 35:109–148
12. Goovaerts P (1997) Geostatistics for natural resources evaluation (applied geostatistics). Oxford University Press, New YorkGoogle Scholar
13. Haykin S (1998) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall PTR, Upper Saddle RiverGoogle Scholar
14. Helland I (1988) On structure of partial least squares regression. Commun Stat - Simul Comput 17:581–607
15. Hensman J, Fusi N, Lawrence ND (2013) Gaussian processes for big data. In: Proceedings of the twenty-ninth conference on uncertainty in artificial intelligence, Bellevue, p 2013Google Scholar
16. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492
17. Lanczos C (1950) An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J Res Natl Bur Stand 45(4):255–282
18. Liem RP, Martins JRRA (2014) Surrogate models and mixtures of experts in aerodynamic performance prediction for mission analysis. In: 15th AIAA/ISSMO multidisciplinary analysis and optimization conference, Atlanta, GA, AIAA-2014-2301Google Scholar
19. Manne R (1987) Analysis of two Partial-Least-Squares algorithms for multivariate calibration. Chemom Intell Lab Syst 2(1–3):187–197
20. Mera NS (2007) Efficient optimization processes using kriging approximation models in electrical impedance tomography. Int J Numer Methods Eng 69(1):202–220
21. Michalewicz Z, Schoenauer M (1996) Evolutionary algorithms for constrained parameter optimization problems. Evol Comput 4: 1–32
22. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830
23. Picheny V, Ginsbourger D, Roustant O, Haftka RT, Kim NH (2010) Adaptive designs of experiments for accurate approximation of a target region. J Mech Des 132(7):071008
24. Powell MJ (1994) A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Advances in optimization and numerical analysis. Springer, pp 51–67Google Scholar
25. Rasmussen C, Williams C (2006) Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, Cambridge
26. Regis R, Shoemaker C (2013) Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization. Eng Optim 45(5):529–555
27. Roustant O, Ginsbourger D, Deville Y (2012) DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J Stat Softw 51(1):1–55
28. Sakata S, Ashida F, Zako M (2004) An efficient algorithm for Kriging approximation and optimization with large-scale sampling data. Comput Methods Appl Mech Eng 193(3):385–404
29. Sasena M (2002) Flexibility and efficiency enhancements for constrained global design optimization with Kriging approximations. PhD thesis, University of MichiganGoogle Scholar
30. Schonlau M (1998) Computer experiments and global optimization. PhD thesis, University of WaterlooGoogle Scholar
31. Wahba G (1990) Spline models for observational data, CBMS-NSF regional conference series in applied mathematics, vol 59. Society for Industrial and Applied Mathematics (SIAM), PhiladelphiaGoogle Scholar
32. Wahba G, Craven P (1978) Smoothing noisy data with spline functions. Estimating the correct degree of smoothing by the method of generalized cross-validation. Numer Math 31:377–404
33. Zimmerman DL, Homer KE (1991) A network design criterion for estimating selected attributes of the semivariogram. Environmetrics 2(4):425–441

## Authors and Affiliations

• Mohamed Amine Bouhlel
• 1
Email author
• Nathalie Bartoli
• 2