Structural and Multidisciplinary Optimization

, Volume 59, Issue 4, pp 1221–1239 | Cite as

A regularization method for constructing trend function in Kriging model

  • Yi Zhang
  • Wen YaoEmail author
  • Siyu Ye
  • Xiaoqian Chen
Research Paper


Kriging is a popular surrogate for approximating computationally expensive computer experiments. When sample points are limited, it is difficult to identify the overall trend of the problem at hand properly. Thanks to the interpolating characteristic of the Kriging model, a constant is widely used as the trend function, which neglects the overall trend presented by data. However, previous researches prove that an appropriate trend function considering high-order terms is able to enhance the approximation ability of the Kriging model. In this paper, a regularization approach is proposed to construct the trend function in the Kriging model to improve the prediction accuracy. First, a new weighting scheme, which is formulated as an optimization problem with regularization terms, is used to solve the regression coefficients. Then, the other model parameters are estimated by maximizing the likelihood function, which leads to a nested optimization problem. It is solved iteratively to obtain the final estimation of the model parameters. From a Bayesian point of view, the proposed regularization method can adaptively tune the parameter of the prior distribution on the regression coefficients in the iterative algorithm. To select good regularization parameters, a cross-validation method is used. The implementation is tested on several analytical examples and physical examples, and the experimental results confirm the effectiveness of the proposed approach.


Kriging Trend function Regularization Bayesian analysis Computer experiments 



We would like to thank four reviewers for thoughtful comments that lead to improvements in the manuscript.

Funding information

This work is supported by the National Natural Science Foundation of China (No. 11725211 and 51675525).


  1. Acar E (2013) Effects of the correlation model, the trend model, and the number of training points on the accuracy of kriging metamodels. Expert Syst 30(5):418–428Google Scholar
  2. Acar E (2014) Simultaneous optimization of shape parameters and weight factors in ensemble of radial basis functions. Struct Multidiscip Optim 49(6):969–978Google Scholar
  3. Aute V, Saleh K, Abdelaziz O, Azarm S, Radermacher R (2013) Cross-validation based single response adaptive design of experiments for Kriging metamodeling of deterministic computer simulations. Struct Multidiscip Optim 48(3):581–605Google Scholar
  4. Balabanov VO, Weckner O, Wu J (2014) Reducing error of polynomial approximation outside of designated design space for practical problems. In: 15th AIAA/ISSMO multidisciplinary analysis and optimization conference, p 2303Google Scholar
  5. Ben-Ari EN, Steinberg DM (2007) Modeling data from computer experiments: sn empirical comparison of Kriging with mars and projection pursuit regression. Qual Eng 19(4):327–338Google Scholar
  6. Bouhlel MA, Bartoli N, Otsmane A, Morlier J (2016) Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction. Struct Multidiscip Optim 53(5):935–952MathSciNetzbMATHGoogle Scholar
  7. Clark LD, Bae H, Gobal K, Penmetsa RC (2016) Engineering design exploration using locally optimized covariance Kriging. AIAA J 54(10):3160–3175Google Scholar
  8. Clarke SM, Griebsch JH, Simpson TW (2005) Analysis of support vector regression for approximation of complex engineering analyses. J Mech Des 127(6):1077–1087Google Scholar
  9. Dette H, Pepelyshev A (2010) Generalized latin hypercube design for computer experiments. Technometrics 52(4):421–429MathSciNetGoogle Scholar
  10. Echard B, Gayton N, Lemaire M (2011) Ak-mcs: an active learning reliability method combining Kriging and monte carlo simulation. Struct Saf 33(2):145–154Google Scholar
  11. Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization. Progress Aerosp Sci 45(1):50–79Google Scholar
  12. Forrester AIJ, Sóbester A, Keane AJ (2008) Engineering design via surrogate modelling: a practical guide. WileyGoogle Scholar
  13. Friedman JH (1991) Multivariate adaptive regression splines. Ann Statist 19(1):1–67MathSciNetzbMATHGoogle Scholar
  14. Galatsanos NP, Katsaggelos AK (1992) Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation. IEEE Trans Image Process 1(3):322–336Google Scholar
  15. Ginsbourger D, Dupuy D, Badea A, Carraro L, Roustant O (2010) A note on the choice and the estimation of Kriging models for the analysis of computer experiments. Appl Stoch Model Bus Ind 25(2):115–131MathSciNetzbMATHGoogle Scholar
  16. Goel T, Hafkta RT, Shyy W (2009) Comparing error estimation measures for polynomial and Kriging approximation of noise-free functions. Struct Multidiscip Optim 38(5):429–442Google Scholar
  17. Golub GH, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21(2):215–223MathSciNetzbMATHGoogle Scholar
  18. Hoerl AE, Kennard RW (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67zbMATHGoogle Scholar
  19. Hung Y (2011) Penalized blind Kriging in computer experiments. Stat Sin 21(3):1171–1190MathSciNetzbMATHGoogle Scholar
  20. Joseph VR, Hung Y, Sudjianto A (2008) Blind Kriging: a new method for developing metamodels. J Mech Des 130(3):1–8Google Scholar
  21. Kersaudy P, Sudret B, Varsier N, Picon O, Wiart J (2015) A new surrogate modeling technique combining Kriging and polynomial chaos expansions–application to uncertainty analysis in computational dosimetry. J Comput Phys 286:103–117MathSciNetzbMATHGoogle Scholar
  22. Kwon H, Choi S (2015) A trended kriging model with r2 indicator and application to design optimization. Aerosp Sci Technol 43:111–125Google Scholar
  23. Lee H, Lee DJ, Kwon H (2018) Development of an optimized trend Kriging model using regression analysis and selection process for optimal subset of basis functions. Aerosp Sci Technol 77:273–285Google Scholar
  24. Li R, Sudjianto A (2005) Analysis of computer experiments using penalized likelihood in gaussian Kriging models. Technometrics 47(2):111–120MathSciNetGoogle Scholar
  25. Liang H, Zhu M (2013) Comment on “metamodeling method using dynamic Kriging for design optimization”. AIAA J 51(12):2988–2989Google Scholar
  26. Liang H, Zhu M, Wu Z (2014) Using cross-validation to design trend function in Kriging surrogate modeling. AIAA J 52(10):2313–2327Google Scholar
  27. Liu H, Ong YS, Cai J (2018) A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct Multidiscip Optim 57(1):393–416Google Scholar
  28. Lophaven SN, Nielsen HB, Søndergaard J (2002) Dace–a matlab kriging toolbox–version 2.0. Tech. repGoogle Scholar
  29. Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA J 43(4):853–863Google Scholar
  30. McIlhagga W (2016) Penalized: a matlab toolbox for fitting generalized linear models with penalties. J Statist Softw 72(6):1–21Google Scholar
  31. Mehmani A, Chowdhury S, Messac A (2015) Predictive quantification of surrogate model fidelity based on modal variations with sample density. Struct Multidiscip Optim 52(2):353–373Google Scholar
  32. Park MY, Hastie T (2007) L1-regularization path algorithm for generalized linear models. J R Stat Soc 69(4):659–677MathSciNetGoogle Scholar
  33. Parmar M, Haselbacher A, Balachandar S (2010) Improved drag correlation for spheres and application to shock-tube experiments. AIAA J 48(6):1273–1277Google Scholar
  34. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–423MathSciNetzbMATHGoogle Scholar
  35. Schobi R, Sudret B, Wiart J (2015) Polynomial-chaos-based Kriging. Int J Uncertain Quantif 5(2):171–193MathSciNetGoogle Scholar
  36. Sóbester A, Forrester AI, Toal DJ, Tresidder E, Tucker S (2014) Engineering design applications of surrogate-assisted optimization techniques. Optim Eng 15(1):243–265zbMATHGoogle Scholar
  37. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc 58(1):267–288MathSciNetzbMATHGoogle Scholar
  38. Viana FAC (2011) Surrogates toolbox user’s guide. Gainesville, FL, Version 3.0.
  39. Viana FAC, Haftka RT, Steffen V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39(4):439–457Google Scholar
  40. Viana FAC, Simpson TW, Balabanov V, Toropov V (2014) Metamodeling in multidisciplinary design optimization: how far have we really come? AIAA J 52(4):670–690Google Scholar
  41. Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129(4):370–380Google Scholar
  42. Yao W, Chen X, Ouyang Q, Tooren MV (2012a) A surrogate based multistage-multilevel optimization procedure for multidisciplinary design optimization. Struct Multidiscip Optim 45(4):559– 574Google Scholar
  43. Yao W, Chen X, Zhao Y, Tooren MV (2012b) Concurrent subspace width optimization method for rbf neural network modeling. IEEE Trans Neural Netw Learn Syst 23(2):247–259Google Scholar
  44. Yao W, Chen X, Ouyang Q, Tooren MV (2013) A reliability-based multidisciplinary design optimization procedure based on combined probability and evidence theory. Struct Multidiscip Optim 48(2):339–354MathSciNetGoogle Scholar
  45. Zhang S, Zhu P, Chen W, Arendt P (2013) Concurrent treatment of parametric uncertainty and metamodeling uncertainty in robust design. Struct Multidiscip Optim 47(1):63–76MathSciNetzbMATHGoogle Scholar
  46. Zhang Y, Kim NH, Park C, Haftka RT (2016) Function extrapolation of noisy data using converging lines. In: AIAA Modeling and simulation technologies conference, p 2144Google Scholar
  47. Zhang Y, Park C, Kim NH, Haftka RT (2017) Function prediction at one inaccessible point using converging lines. J Mech Des 139(5):051,402Google Scholar
  48. Zhao L, Choi K, Lee I (2011) Metamodeling method using dynamic Kriging for design optimization. AIAA J 49(9):2034–2046Google Scholar
  49. Zhao L, Choi K, Lee I (2013) Reply by the authors to the comment by H. Liang and M. Zhu. AIAA J 51(12):2989–2990Google Scholar
  50. Zou H (2006) The adaptive lasso and its oracle properties. J Am Stat Assoc 101:1418–1429MathSciNetzbMATHGoogle Scholar
  51. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc 67(2):301–320MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Aerospace Science and EngineeringNational University of Defense TechnologyChangshaChina
  2. 2.National Innovation Institute of Defense TechnologyChinese Academy of Military ScienceBeijingChina

Personalised recommendations