Engineering with Computers

, Volume 35, Issue 1, pp 157–173 | Cite as

Gradient-enhanced kriging for high-dimensional problems

  • Mohamed A. BouhlelEmail author
  • Joaquim R. R. A. Martins
Original Article


Surrogate models provide an affordable alternative to the evaluation of expensive deterministic functions. However, the construction of accurate surrogate models with many independent variables is currently prohibitive because they require a large number of function evaluations for the desired accuracy. Gradient-enhanced kriging has the potential to reduce the number of evaluations when efficient gradient computation, such as an adjoint method, is available. However, current gradient-enhanced kriging methods do not scale well with the number of sampling points because of the rapid growth in the size of the correlation matrix, where new information is added for each sampling point in each direction of the design space. Furthermore, they do not scale well with the number of independent variables because of the increase in the number of hyperparameters that must be estimated. To address this issue, we develop a new gradient-enhanced surrogate model approach that drastically reduces the number of hyperparameters through the use of the partial least squares method to maintain accuracy. In addition, this method is able to control the size of the correlation matrix by adding only relevant points defined by the information provided by the partial least squares method. To validate our method, we compare the global accuracy of the proposed method with conventional kriging surrogate models on two analytic functions with up to 100 dimensions, as well as engineering problems of varied complexity with up to 15 dimensions. We show that the proposed method requires fewer sampling points than conventional methods to obtain the desired accuracy, or it provides more accuracy for a fixed budget of sampling points. In some cases, we get models that are over three times more accurate than previously developed surrogate models for the same computational time, and over 3200 times faster than standard gradient-enhanced kriging models for the same accuracy.


Gradient-enhanced kriging Partial least squares High-dimensions 

List of symbols

Matrices and vectors are in bold type


Number of dimensions


Hypercube expressed by the product between intervals of each direction space


Number of sampling points


Number of principal components

\(\mathbf{x}\), \(\mathbf{x}'\)

\(1\times d\) vector


\(j{\text {th}}\) element of \(\mathbf{x}\) for \(j=1,\dots ,d\)


\(n\times d\) matrix containing sampling points


\(n\times 1\) vector containing simulation of \(\mathbf{X}\)


\(i{\text {th}}\) sampling point for \(i=1,\dots ,n\) (\(1\times d\) vector)


\(i{\text {th}}\) evaluated output point for \(i=1,\dots ,n\)




Matrix containing residual of the \((l-1){\text {th}}\) inner regression

\(k(\cdot ,\cdot )\)

Covariance function


Spatial correlation between \(\mathbf{x}\) and \(\mathbf{x}'\)


Covariance matrix


Prediction of the kriging variance

\(\sigma ^2\)

Process variance

\(\theta _i\)

ith parameter of the covariance function for \(i=1,\dots ,d\)


Gaussian process


n-vector of ones


lth principal component for \(l=1,\dots ,h\)


Weight vector for partial least squares

\(\Delta x_j\)

First-order Taylor approximation step in the \(j \text{th}\) direction


  1. 1.
    Abraham L (2009) pydoe: the experimental design package for python.
  2. 2.
    Alberto PR, González FG (2012) Partial least squares regression on symmetric positive-definite matrices. Revista Colombiana de Estadística 36(1):177–192MathSciNetzbMATHGoogle Scholar
  3. 3.
    An J, Owen A (2001) Quasi-Regression. J Complex 17(4):588–607MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Barber D (2012) Bayesian reasoning and machine learning. Cambridge University Press, New YorkzbMATHGoogle Scholar
  5. 5.
    Bartoli N, Bouhlel MA, Kurek I, Lafage R, Lefebvre T, Morlier J, Priem R, Stilz V, Regis R (2016) Improvement of efficient global optimization with application to aircraft wing design. In: 17th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Washington, DC. AIAA-2016-4001Google Scholar
  6. 6.
    Bouhlel MA, Bartoli N, Morlier J, Otsmane A (2016a) An improved approach for estimating the hyperparameters of the Kriging model for high-dimensional problems through the partial least squares method. Math Probl En 2016: 6723410.
  7. 7.
    Bouhlel MA, Bartoli N, Otsmane A, Morlier J (2016b) Improving kriging surrogates of high-dimensional design models by pLeast squares dimension reduction. Struct Multidisc Optim 53(5):935–952. ISSN 1615-1488Google Scholar
  8. 8.
    Bouhlel MA, Bartoli N, Regis RG, Otsmane A, Morlier J (2018) Efficient global optimization for high-dimensional constrained problems by using the kriging models combined with the partial least squares method. Eng Optim.
  9. 9.
    Box G, Hunter J, Hunter W (2005) Statistics for experimenters: design, innovation, and discovery. Wiley-Interscience, Wiley Series in Probability and Statistics.
  10. 10.
    Choi S, Chung H, Alonso J (2004) Design of Low-Boom Supersonic Business Jet With Evolutionary Algorithms Using Adaptive Unstructured Mesh. In 45th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference. Palm Springs, California. AIAA-2004-1758Google Scholar
  11. 11.
    Chung HS, Alonso J (2002) Design of a Low-Boom Supersonic Business Jet Using Cokriging Approximation Models. 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Multidisciplinary Analysis Optimization Conferences. AIAA-2002-5598Google Scholar
  12. 12.
    Cressie N (1988) Spatial prediction and ordinary kriging. Math Geol 20(4):405–421MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Deb K (2000) An efficient constraint handling method for genetic algorithms. Comput Methods Appl Mech Eng 186:311–338CrossRefzbMATHGoogle Scholar
  14. 14.
    Forrester AIJ, Sóbester A, Keane AJ (2008) Engineering design via surrogate modeling: a practical guide. Wiley, New YorkCrossRefGoogle Scholar
  15. 15.
    Frank IE, Friedman JH (1993) A statistical view of some chemometrics regression tools. Technometrics 35:109–148CrossRefzbMATHGoogle Scholar
  16. 16.
    Haftka R, Villanueva D, Chaudhuri A (2016) Parallel surrogate-assisted global optimization with expensive functions—a survey. Struct Multidiscipl Optim 54:3–13MathSciNetCrossRefGoogle Scholar
  17. 17.
    Helland IS (1988) On the structure of partial least squares regression. Commun Stat Simul Comput 17:581–607MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Jeong S, Murayama M, Yamamoto K (2005) Efficient optimization design method using Kriging model. J Aircr 42(2):413–420CrossRefGoogle Scholar
  19. 19.
    Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Global Optim 21(4):345–383MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13(4):455–492MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Kenway GKW, Kennedy GJ, Martins JRRA (2014) Scalable parallel approach for high-fidelity steady-state aeroelastic analysis and derivative computations. AIAA J 52(5):935–951. CrossRefGoogle Scholar
  22. 22.
    Kleijnen J, Van Beers W, Van Nieuwenhuyse I (2010) Constrained optimization in expensive simulation: novel approach. Eur J Oper Res 202(1):164–174CrossRefzbMATHGoogle Scholar
  23. 23.
    Kleijnen J, Beers W, Nieuwenhuyse I (2012) Expected improvement in efficient global optimization through bootstrapped kriging. J Global Optim 54(1):59–73MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Kleijnen JPC (2015) Design and analysis of simulation experiments, vol 230. Springer, New YorkzbMATHGoogle Scholar
  25. 25.
    Kleijnen JPC (2017) Regression and kriging metamodels with their experimental designs in simulation: a review. Eur J Oper Res 256(1):1–16. MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Krige DG (1951) A statistical approach to some basic mine valuation problems on the witwatersrand. J Chem Metall Mining Soc 52:119–139Google Scholar
  27. 27.
    Laurenceau J, Sagaut P (2008) Building efficient response surfaces of aerodynamic functions with kriging and cokriging. AIAA J 46(2):498–507CrossRefGoogle Scholar
  28. 28.
    Lewis RM (1998) Using sensitivity information in the construction of Kriging models for design optimization. AIAA-98-4799. 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Multidisciplinary Analysis Optimization Conferences, pp 730–737Google Scholar
  29. 29.
    Liem RP, Kenway GK, Martins JRRA (2012) Multi-point, multi-mission, high-fidelity aerostructural optimization of a long-range aircraft configuration. In: Proceedings of the 14th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference. Indianapolis.
  30. 30.
    Liem RP, Kenway GKW, Martins JRRA (2015a) Multimission aircraft fuel burn minimization via multipoint aerostructural optimization. AIAA J 53(1):104–122.
  31. 31.
    Liem RP, Mader CA, Martins JRRA (2015b) Surrogate models and mixtures of experts in aerodynamic performance prediction for aircraft mission analysis. Aerosp Sci Technol 43:126–151. CrossRefGoogle Scholar
  32. 32.
    Liping W, Don B, Gene W, Mahidhar R (2006) A comparison of metamodeling methods using practical industry requirements. In: Proceedings of the 47th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. Newport, RIGoogle Scholar
  33. 33.
    Liu W (2003) Development of gradient-enhanced Kriging approximations for multidisciplinary design optimization. PhD thesis, University of Notre DameGoogle Scholar
  34. 34.
    Loeppky JL, Sacks S, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376. MathSciNetCrossRefGoogle Scholar
  35. 35.
    Mader CA, Martins JRRA, Alonso JJ, van der Weide E (2008) ADjoint: an approach for the rapid development of discrete adjoint solvers. AIAA J 46(4):863–873. CrossRefGoogle Scholar
  36. 36.
    Mardia KV, Watkins AJ (1989) On multimodality of the likelihood in the spatial linear model. Biometrika 76(2):289. MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Martins JRRA, Hwang JT (2013) Review and unification of methods for computing derivatives of multidisciplinary computational models. AIAA J 51(11):2582–2599. CrossRefGoogle Scholar
  38. 38.
    Matheron G (1963) Principles of geostatistics. Econ Geol 58(8):1246–1266CrossRefGoogle Scholar
  39. 39.
    Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Ollar J, Mortished C, Jones R, Sienz J, Toropov V (2017) Gradient based hyper-parameter optimisation for well conditioned Kriging metamodels. Struct Multidiscipl Optim 55:2029–2044MathSciNetCrossRefGoogle Scholar
  41. 41.
    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Rettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830MathSciNetzbMATHGoogle Scholar
  42. 42.
    Pironneau O (1974) On optimum design in fluid mechanics. J Fluid Mech 64(1):97–110MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Rasmussen CE, Williams CKI (2006) Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, CambridgezbMATHGoogle Scholar
  44. 44.
    Sacks J, Schiller SB, Welch WJ (1989a) Designs for computer experiments. Technometrics 31(1):41–47MathSciNetCrossRefGoogle Scholar
  45. 45.
    Sacks J, Welch WJ, Mitchell WJ, Wynn HP (1989b) Design and analysis of computer experiments. Stat Sci 4(4):409–435MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Sakata S, Ashida F, Zako M (2003) Structural optimization using Kriging approximation. Comput Methods Appl Mech Eng 192(417):923–939CrossRefzbMATHGoogle Scholar
  47. 47.
    Simpson TW, Mauery TM, Korte JJ, Mistree F (2001a) Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA J 39(12):2233–2241CrossRefGoogle Scholar
  48. 48.
    Simpson TW, Poplinski JD, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150CrossRefzbMATHGoogle Scholar
  49. 49.
    Tenenhaus M (1998) La Régression PLS: Théorie et Pratique. Éd, TechnipzbMATHGoogle Scholar
  50. 50.
    Toal DJJ, Bressloff NW, Keane AJ (2008) Geometric filtration using POD for aerodynamic design optimization. In: 26th AIAA Applied Aerodynamics Conference.
  51. 51.
    Ulaganathan S, Couckuyt I, Dhaene T, Laermans E, Degroote J (2014) On the use of gradients in Kriging surrogate models. In: Proceedings of the 2014 Winter Simulation Conference, Savannah, GA, USA, December 7–10, pp 2692–2701.
  52. 52.
    Viana FAC, Simpson TW, Balabanov V, Toropov V (2014) Metamodeling in multidisciplinary design optimization: how far have we really come? AIAA J 52:670–690. CrossRefGoogle Scholar
  53. 53.
    Welch WJ, Buck RJ, Sacks J, Wynn HP, Mitchell TJ, Morris MD (1992) Screening, predicting, and computer experiments. Technometrics 34(1):15–25CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of MichiganAnn ArborUSA

Personalised recommendations