An efficient kriging modeling method for high-dimensional design problems based on maximal information coefficient

Abstract

Kriging, one of the most popular surrogate models, is widely used in computationally expensive optimization problems to improve the design efficiency. However, due to the “curse-of-dimensionality,” the time for generating the kriging model increases exponentially as the dimension of the problem grows. When it comes to the cases that the kriging model needs to be frequently constructed, such as sequential sampling for kriging modeling or global optimization based on kriging model, the increased modeling time should be taken into consideration. To overcome this challenge, we propose a novel kriging modeling method which combines kriging with maximal information coefficient (MIC). Taking the features of the optimized hyper-parameters into consideration, MIC is utilized for estimating the relative magnitude of hyper-parameters. Then this knowledge of hyper-parameters is incorporated into the maximum likelihood estimation problem to reduce the dimensionality. In this way, the high dimensional optimization can be transformed into a one-dimensional optimization, which can significantly improve the modeling efficiency. Five representative numerical examples from 20-D to 80-D and an industrial example with 35 variables are used to show the effectiveness of the proposed method. Results show that compared with the conventional kriging, the modeling time of the proposed method can be ignored, while the loss of accuracy is acceptable. For the problems with more than 40 variables, the proposed method can even obtain a more accurate kriging model with given computational resources. Besides, the proposed method is also compared with KPLS (kriging combined with the partial least squares method), another state-of-the-art kriging modeling method for high-dimensional problems. Results show that the proposed method is more competitive than KPLS, which means the proposed method is an efficient kriging modeling method for high-dimensional problems.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Notes

  1. 1.

    http://minepy.readthedocs.io/

  2. 2.

    https://github.com/SMTorg/smt.

Abbreviations

Symbols:

Meaning

:

Set of real numbers

+ :

Set of positive real numbers

d :

Dimensions

m :

Number of sample points

x :

A sample point (1 × d vector)

x j :

jth element of x for j = 1, ..., d

X :

m × d matrix of sample points

x (i) :

ith sample point for i = 1, ..., m (1 × d vector)

x i :

ith column of X for i = 1, ..., d (m × 1 vector)

\( {x}_l^{(i)} \) :

ith element of xl for i = 1, ..., m.

y :

m × 1 vector of response values

y (i) :

Response value of x(i) for i = 1, ..., m

\( \hat{y}\left(\mathbf{x}\right) \) :

Prediction of a sample point

z(x):

Realization of a stochastic process

R :

Spatial correlation function

R :

Correlation matrix

1 :

n−vector of ones

s 2(x):

Prediction of the kriging variance

σ 2 :

Process variance

θ :

Hyper-parameters (1 × d vector)

θ i :

ith element of θ for i = 1, ..., d

S i :

First-order Sobol’ index for i = 1, ..., d

w i :

MIC value of xi and y for i = 1, ..., d

λ :

Auxiliary parameter

References

  1. Albanese D, Filosi M, Visintainer R, Riccadonna S, Jurman G, Furlanello C (2013) Minerva and minepy: a c engine for the mine suite and its r, python and matlab wrappers. Bioinformatics 29(3):407–408

    Google Scholar 

  2. Bouhlel MA, Martins JRRA (2019) Gradient-enhanced kriging for high-dimensional problems. Engineering with Computers 35(1):157–173

    Google Scholar 

  3. Bouhlel MA, Bartoli N, Otsmane A, Morlier J (2016) Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction. Struct Multidiscip Optim 53(5):935–952

    MathSciNet  MATH  Google Scholar 

  4. Box GE, Draper NR (1987) Empirical model-building and response surfaces. J R Stat Soc 30(2):229–231

    MATH  Google Scholar 

  5. Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Trans Evol Comput 10(6):646–657

    Google Scholar 

  6. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, Cambridge

    Google Scholar 

  7. Cai X, Qiu H, Gao L, Shao X (2017) Metamodeling for high dimensional design problems by multi-fidelity simulations. Struct Multidiscip Optim 56(1):151–166

    MathSciNet  Google Scholar 

  8. Chen L, Qiu H, Gao L, Jiang C, Yang Z (2019) A screening-based gradient-enhanced kriging modeling method for high-dimensional problems. Appl Math Model 69:15–31

    MathSciNet  Google Scholar 

  9. Da Veiga S (2015) Global sensitivity analysis with dependence measures. J Stat Comput Simul 85(7):1283–1305

    MathSciNet  Google Scholar 

  10. Dong H, Sun S, Song B, Wang P (2019) Multi-surrogate-based global optimization using a score-based infill criterion. Struct Multidiscip Optim 59(2):485–506

    MathSciNet  Google Scholar 

  11. Dong H, Song B, Dong Z, Wang P (2018) Scgosr: surrogate-based constrained global optimization using space reduction. Appl Soft Comput 65:462–477

    Google Scholar 

  12. Emmerich MT, Giannakoglou KC, Naujoks B (2006) Single-and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10(4):421–439

    Google Scholar 

  13. Forrester AI, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45(1–3):50–79

    Google Scholar 

  14. Forrester A, Sobester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, Hoboken

    Google Scholar 

  15. Haftka RT, Mroz Z (1986) First-and second-order sensitivity analysis of linear and nonlinearstructures. AIAA J 24(7):1187–1192

    MathSciNet  MATH  Google Scholar 

  16. Han ZH, Zhang Y, Song CX, Zhang KS (2017) Weighted gradient-enhanced kriging for high-dimensional surrogate modeling and design optimization. AIAA J 55(12):4330–4346

    Google Scholar 

  17. Hartwig L, Bestle D (2017) Compressor blade design for stationary gas turbines using dimension reduced surrogate modeling. Evol Comput

  18. Hemmateenejad B, Baumann K (2018) Screening for linearly and nonlinearly related variables in predictive cheminformatic models. J Chemom 32:e3009

    Google Scholar 

  19. Hollingsworth P, Mavris D (2003) Gaussian process meta-modeling: comparison of Gaussian process training methods. In AIAA's 3rd Annual Aviation Technology, Integration, and Operations (ATIO) Forum (p. 6761)

  20. Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21(4):345–383

    MathSciNet  MATH  Google Scholar 

  21. Kinney JB, Atwal GS (2014) Equitability, mutual information, and the maximal information coefficient. Proc Natl Acad Sci 111(9):3354–3359

    MathSciNet  MATH  Google Scholar 

  22. Krige DG (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J South Afr Inst Min Metall 52(6):119–139

    Google Scholar 

  23. Kulfan B, Bussoletti J (2006) “Fundamental” parameteric geometry representations for aircraft component shapes. Paper presented at the 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference: The Modeling and Simulation Frontier for Multidisciplinary Design Optimization

  24. Lee K, Cho H, Lee I (2019) Variable selection using Gaussian process regression-based metrics for high-dimensional model approximation with limited data. Struct Multidiscip Optim 59(5):1439–1454

    MathSciNet  Google Scholar 

  25. Li C, Wang P, Dong H, Wang X (2018) A simplified shape optimization strategy for blended-wing-body underwater gliders. Struct Multidiscip Optim 58(5):2189–2202

    Google Scholar 

  26. Liu B, Zhang Q, Gielen GG (2014) A Gaussian process surrogate model assisted evolutionary algorithm for medium scale expensive optimization problems. IEEE Trans Evol Comput 18(2):180–192

    Google Scholar 

  27. Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. TECHNOMETRICS 51(4):366–376

    MathSciNet  Google Scholar 

  28. Lophaven SN, Nielsen HB, Søndergaard J (2002) Aspects of the matlab toolbox DACE. IMM, Informatics and Mathematical Modelling, The Technical University of Denmark

  29. Matheron G (1963) Principles of geostatistics. Econ Geol 58(8):1246–1266

    Google Scholar 

  30. Mckay MD, Beckman RJ, Conover WJ (1979) Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245

    MathSciNet  MATH  Google Scholar 

  31. Michalewicz Z, Schoenauer M (2014) Evolutionary algorithms for constrained parameter optimization problems. Evol Comput 4(1):1–32

    Google Scholar 

  32. Mullur A, Messac A (2005) Extended radial basis functions: more flexible and effective metamodeling. AIAA J 43(6):1306–1315

    Google Scholar 

  33. Powell MJ (1994) A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Advances in optimization and numerical analysis. Springer, Dordrecht, pp 51–67

    Google Scholar 

  34. Rasmussen CE, Williams CKI (2005) Gaussian processes for machine learning. MIT Press, Cambridge

    Google Scholar 

  35. Reshef DN, Reshef YA, Finucane HK, Grossman SR, Mcvean G, Turnbaugh PJ et al (2011) Detecting novel associations in large data sets. Science 334(6062):1518–1524

    MATH  Google Scholar 

  36. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci:409–423

    MathSciNet  MATH  Google Scholar 

  37. Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D et al (2008) Global sensitivity analysis: the primer. Wiley, Hoboken

    Google Scholar 

  38. Sasena MJ (2002) Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations (Doctoral dissertation, University of Michigan)

  39. Schmit LA, Farshi B (1974) Some approximation concepts for structural synthesis. AIAA J 12(5):692–699

    Google Scholar 

  40. Shan S, Wang GG (2010) Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multidiscip Optim 41(2):219–241

    MathSciNet  MATH  Google Scholar 

  41. Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–222

    MathSciNet  Google Scholar 

  42. Sobol IM (2001) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math Comput Simul 55(1–3):271–280

    MathSciNet  MATH  Google Scholar 

  43. Speed T (2011) A correlation for the 21st century. Science 334(6062):1502–1503

    Google Scholar 

  44. Sun GL, Li JB, Dai J, Song ZC, Lang F (2018). Feature selection for IoT based on maximal information coefficient. Futur Gener Comput Syst 89:606–616

    Google Scholar 

  45. Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E (2016a) High dimensional kriging metamodelling utilising gradient information. Appl Math Model 40(9–10):5256–5270

    Google Scholar 

  46. Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E (2016b) Performance study of gradient-enhanced kriging. Eng Comput 32(1):15–34

    Google Scholar 

  47. Wang H, Jin Y, Doherty J (2017) Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems. IEEE Trans Cybern 47(9):2664–2677

    Google Scholar 

  48. Wu D, Wang GG (2018) Knowledge assisted optimization for large-scale problems: a review and proposition. In: ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp V02BT03A032–V02BT03A032. American Society of Mechanical Engineers

  49. Wu D, Coatanea E, Wang GG (2017) Dimension reduction and decomposition using causal graph and qualitative analysis for aircraft concept design optimization. In: ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, New York, pp V02BT03A035–V02BT03A035

    Google Scholar 

  50. Zhao X, Deng W, Shi Y (2013) Feature selection with attributes clustering by maximal information coefficient. Procedia Comput Sci 17(2):70–79

    Google Scholar 

Download references

Funding

This research was financially supported by the National Natural Science Foundation of China (Grant No. 51875466 and Grant No. 51805436).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Peng Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Responsible Editor: Nam Ho Kim

Appendix: Examples of spatial correlation functions

Appendix: Examples of spatial correlation functions

The proposed method (KMIC) combines kriging with maximal information coefficient and constructs a new maximum likelihood estimation problem (Eq. 22). Then the number of parameters we need to optimize when estimating hyper-parameters is reduced to one. It seems that KMIC constructs a new spatial correlation function which depends on only one parameter. In this article, Gaussian exponential correlation function is applied with the proposed method. For the other spatial correlation functions, the proposed method is also suitable. Appendix Table 8 presents the most popular examples of spatial correlation functions. Appendix Table 9 presents the new KMIC spatial correlation functions based on the examples given in Appendix Table 8.

Table 8 Examples of commonly used spatial correlation functions
Table 9 Examples of KMIC spatial correlation functions

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhao, L., Wang, P., Song, B. et al. An efficient kriging modeling method for high-dimensional design problems based on maximal information coefficient. Struct Multidisc Optim 61, 39–57 (2020). https://doi.org/10.1007/s00158-019-02342-3

Download citation

Keywords

  • Kriging
  • Maximal information coefficient
  • High-dimensional problems
  • Metamodels