Improving kriging surrogates of high-dimensional design models by Partial Least Squares dimension reduction

  • Mohamed Amine BouhlelEmail author
  • Nathalie Bartoli
  • Abdelkader Otsmane
  • Joseph Morlier


Engineering computer codes are often computationally expensive. To lighten this load, we exploit new covariance kernels to replace computationally expensive codes with surrogate models. For input spaces with large dimensions, using the kriging model in the standard way is computationally expensive because a large covariance matrix must be inverted several times to estimate the parameters of the model. We address this issue herein by constructing a covariance kernel that depends on only a few parameters. The new kernel is constructed based on information obtained from the Partial Least Squares method. Promising results are obtained for numerical examples with up to 100 dimensions, and significant computational gain is obtained while maintaining sufficient accuracy.


Kriging Partial Least Squares Experiment design Metamodels 

Symbols and notation

Matrices and

vectors are in bold type.




Determinant of a matrix


Absolute value

Set of real numbers


Set of positive real numbers


Number of sampling points




Number of principal components retained


d vector


j th element of a vector x


n × d matrix containing sampling points


n × 1 vector containing simulation of X


i th training point for i = 1,…,n (a 1 × d vector)


d × 1 vector containing X weights given by the l th PLS iteration for l = 1,…,h




Matrix containing residual of inner regression of (l − 1)st PLS iteration for l = 1,…,h

k(⋅, ⋅)

Covariance function

(0, k(⋅, ⋅))

Distribution of a Gaussian process with mean function 0 and covariance function k(⋅, ⋅)


Superscript t denotes the transpose operation of the vector x



The authors thank the anonymous reviewers for their insightful and constructive comments. We also extend our grateful thanks to A. Chiplunkar from ISAE SUPAERO, Toulouse and R. G. Regis from Saint Joseph’s University, Philadelphia for their careful correction of the manuscript and to SNECMA for providing the tables of experiment results. Finally, B. Kraabel is gratefully acknowledged for carefully reviewing the paper prior to publication.


  1. Alberto P, González F (2012) Partial Least Squares regression on symmetric positive-definite matrices. Rev Col Estad 36(1):177–192MathSciNetzbMATHGoogle Scholar
  2. Bachoc F (2013) Cross Validation and Maximum Likelihood estimation of hyper-parameters of Gaussian processes with model misspecification. Comput Stat Data Anal 66:55–69MathSciNetCrossRefGoogle Scholar
  3. Bishop CM (2007) Pattern recognition and machine learning (information science and statistics). SpringerGoogle Scholar
  4. Braham H, Ben Jemaa S, Sayrac B, Fort G, Moulines E (2014) Low complexity spatial interpolation for cellular coverage analysis. In: 2014 12th international symposium on modeling and optimization in mobile, ad hoc, and wireless networks (WiOpt). IEEE, pp 188–195Google Scholar
  5. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  6. Cressie N (1988) Spatial prediction and ordinary kriging. Math Geol 20(4):405–421MathSciNetCrossRefzbMATHGoogle Scholar
  7. Damianou A, Lawrence ND (2013) Deep gaussian processes. In: Proceedings of the sixteenth international conference on artificial intelligence and statistics, AISTATS 2013, Scottsdale, pp 207–215Google Scholar
  8. Durrande N (2011) Covariance kernels for simplified and interpretable modeling. A functional and probabilistic approach. Theses, Ecole Nationale Supérieure des Mines de saint-EtienneGoogle Scholar
  9. Durrande N, Ginsbourger D, Roustant O (2012) Additive covariance kernels for high-dimensional gaussian process modeling. Ann Fac Sci Toulouse Math 21(3):481–499MathSciNetCrossRefzbMATHGoogle Scholar
  10. Forrester A, Sobester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, New YorkCrossRefGoogle Scholar
  11. Frank IE, Friedman JH (1993) A statistical view of some chemometrics regression tools. Technometrics 35:109–148CrossRefzbMATHGoogle Scholar
  12. Goovaerts P (1997) Geostatistics for natural resources evaluation (applied geostatistics). Oxford University Press, New YorkGoogle Scholar
  13. Haykin S (1998) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall PTR, Upper Saddle RiverGoogle Scholar
  14. Helland I (1988) On structure of partial least squares regression. Commun Stat - Simul Comput 17:581–607MathSciNetCrossRefzbMATHGoogle Scholar
  15. Hensman J, Fusi N, Lawrence ND (2013) Gaussian processes for big data. In: Proceedings of the twenty-ninth conference on uncertainty in artificial intelligence, Bellevue, p 2013Google Scholar
  16. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492MathSciNetCrossRefzbMATHGoogle Scholar
  17. Lanczos C (1950) An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J Res Natl Bur Stand 45(4):255–282MathSciNetCrossRefGoogle Scholar
  18. Liem RP, Martins JRRA (2014) Surrogate models and mixtures of experts in aerodynamic performance prediction for mission analysis. In: 15th AIAA/ISSMO multidisciplinary analysis and optimization conference, Atlanta, GA, AIAA-2014-2301Google Scholar
  19. Manne R (1987) Analysis of two Partial-Least-Squares algorithms for multivariate calibration. Chemom Intell Lab Syst 2(1–3):187–197CrossRefGoogle Scholar
  20. Mera NS (2007) Efficient optimization processes using kriging approximation models in electrical impedance tomography. Int J Numer Methods Eng 69(1):202–220MathSciNetCrossRefzbMATHGoogle Scholar
  21. Michalewicz Z, Schoenauer M (1996) Evolutionary algorithms for constrained parameter optimization problems. Evol Comput 4: 1–32CrossRefGoogle Scholar
  22. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830MathSciNetzbMATHGoogle Scholar
  23. Picheny V, Ginsbourger D, Roustant O, Haftka RT, Kim NH (2010) Adaptive designs of experiments for accurate approximation of a target region. J Mech Des 132(7):071008CrossRefGoogle Scholar
  24. Powell MJ (1994) A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Advances in optimization and numerical analysis. Springer, pp 51–67Google Scholar
  25. Rasmussen C, Williams C (2006) Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, CambridgezbMATHGoogle Scholar
  26. Regis R, Shoemaker C (2013) Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization. Eng Optim 45(5):529–555MathSciNetCrossRefGoogle Scholar
  27. Roustant O, Ginsbourger D, Deville Y (2012) DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J Stat Softw 51(1):1–55CrossRefGoogle Scholar
  28. Sakata S, Ashida F, Zako M (2004) An efficient algorithm for Kriging approximation and optimization with large-scale sampling data. Comput Methods Appl Mech Eng 193(3):385–404CrossRefzbMATHGoogle Scholar
  29. Sasena M (2002) Flexibility and efficiency enhancements for constrained global design optimization with Kriging approximations. PhD thesis, University of MichiganGoogle Scholar
  30. Schonlau M (1998) Computer experiments and global optimization. PhD thesis, University of WaterlooGoogle Scholar
  31. Wahba G (1990) Spline models for observational data, CBMS-NSF regional conference series in applied mathematics, vol 59. Society for Industrial and Applied Mathematics (SIAM), PhiladelphiaGoogle Scholar
  32. Wahba G, Craven P (1978) Smoothing noisy data with spline functions. Estimating the correct degree of smoothing by the method of generalized cross-validation. Numer Math 31:377–404MathSciNetCrossRefzbMATHGoogle Scholar
  33. Zimmerman DL, Homer KE (1991) A network design criterion for estimating selected attributes of the semivariogram. Environmetrics 2(4):425–441CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Mohamed Amine Bouhlel
    • 1
    Email author
  • Nathalie Bartoli
    • 2
  • Abdelkader Otsmane
    • 1
  • Joseph Morlier
    • 3
  1. 1.SNECMA, Rond-point René Ravaud-RéauMoissy-CramayelFrance
  2. 2.ONERAToulouseFrance
  3. 3.Université de Toulouse, CNRS, Institut Clément Ader, ISAE-SUPAEROToulouse Cedex 4France

Personalised recommendations