# An efficient kriging modeling method for high-dimensional design problems based on maximal information coefficient

## Abstract

Kriging, one of the most popular surrogate models, is widely used in computationally expensive optimization problems to improve the design efficiency. However, due to the “curse-of-dimensionality,” the time for generating the kriging model increases exponentially as the dimension of the problem grows. When it comes to the cases that the kriging model needs to be frequently constructed, such as sequential sampling for kriging modeling or global optimization based on kriging model, the increased modeling time should be taken into consideration. To overcome this challenge, we propose a novel kriging modeling method which combines kriging with maximal information coefficient (MIC). Taking the features of the optimized hyper-parameters into consideration, MIC is utilized for estimating the relative magnitude of hyper-parameters. Then this knowledge of hyper-parameters is incorporated into the maximum likelihood estimation problem to reduce the dimensionality. In this way, the high dimensional optimization can be transformed into a one-dimensional optimization, which can significantly improve the modeling efficiency. Five representative numerical examples from 20-D to 80-D and an industrial example with 35 variables are used to show the effectiveness of the proposed method. Results show that compared with the conventional kriging, the modeling time of the proposed method can be ignored, while the loss of accuracy is acceptable. For the problems with more than 40 variables, the proposed method can even obtain a more accurate kriging model with given computational resources. Besides, the proposed method is also compared with KPLS (kriging combined with the partial least squares method), another state-of-the-art kriging modeling method for high-dimensional problems. Results show that the proposed method is more competitive than KPLS, which means the proposed method is an efficient kriging modeling method for high-dimensional problems.

This is a preview of subscription content, log in to check access.

## Abbreviations

Symbols:

Meaning

:

Set of real numbers

+ :

Set of positive real numbers

d :

Dimensions

m :

Number of sample points

x :

A sample point (1 × d vector)

x j :

jth element of x for j = 1, ..., d

X :

m × d matrix of sample points

x (i) :

ith sample point for i = 1, ..., m (1 × d vector)

x i :

ith column of X for i = 1, ..., d (m × 1 vector)

$${x}_l^{(i)}$$ :

ith element of xl for i = 1, ..., m.

y :

m × 1 vector of response values

y (i) :

Response value of x(i) for i = 1, ..., m

$$\hat{y}\left(\mathbf{x}\right)$$ :

Prediction of a sample point

z(x):

Realization of a stochastic process

R :

Spatial correlation function

R :

Correlation matrix

1 :

n−vector of ones

s 2(x):

Prediction of the kriging variance

σ 2 :

Process variance

θ :

Hyper-parameters (1 × d vector)

θ i :

ith element of θ for i = 1, ..., d

S i :

First-order Sobol’ index for i = 1, ..., d

w i :

MIC value of xi and y for i = 1, ..., d

λ :

Auxiliary parameter

## References

1. Albanese D, Filosi M, Visintainer R, Riccadonna S, Jurman G, Furlanello C (2013) Minerva and minepy: a c engine for the mine suite and its r, python and matlab wrappers. Bioinformatics 29(3):407–408

2. Bouhlel MA, Martins JRRA (2019) Gradient-enhanced kriging for high-dimensional problems. Engineering with Computers 35(1):157–173

3. Bouhlel MA, Bartoli N, Otsmane A, Morlier J (2016) Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction. Struct Multidiscip Optim 53(5):935–952

4. Box GE, Draper NR (1987) Empirical model-building and response surfaces. J R Stat Soc 30(2):229–231

5. Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Trans Evol Comput 10(6):646–657

6. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, Cambridge

7. Cai X, Qiu H, Gao L, Shao X (2017) Metamodeling for high dimensional design problems by multi-fidelity simulations. Struct Multidiscip Optim 56(1):151–166

8. Chen L, Qiu H, Gao L, Jiang C, Yang Z (2019) A screening-based gradient-enhanced kriging modeling method for high-dimensional problems. Appl Math Model 69:15–31

9. Da Veiga S (2015) Global sensitivity analysis with dependence measures. J Stat Comput Simul 85(7):1283–1305

10. Dong H, Sun S, Song B, Wang P (2019) Multi-surrogate-based global optimization using a score-based infill criterion. Struct Multidiscip Optim 59(2):485–506

11. Dong H, Song B, Dong Z, Wang P (2018) Scgosr: surrogate-based constrained global optimization using space reduction. Appl Soft Comput 65:462–477

12. Emmerich MT, Giannakoglou KC, Naujoks B (2006) Single-and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10(4):421–439

13. Forrester AI, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45(1–3):50–79

14. Forrester A, Sobester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, Hoboken

15. Haftka RT, Mroz Z (1986) First-and second-order sensitivity analysis of linear and nonlinearstructures. AIAA J 24(7):1187–1192

16. Han ZH, Zhang Y, Song CX, Zhang KS (2017) Weighted gradient-enhanced kriging for high-dimensional surrogate modeling and design optimization. AIAA J 55(12):4330–4346

17. Hartwig L, Bestle D (2017) Compressor blade design for stationary gas turbines using dimension reduced surrogate modeling. Evol Comput

18. Hemmateenejad B, Baumann K (2018) Screening for linearly and nonlinearly related variables in predictive cheminformatic models. J Chemom 32:e3009

19. Hollingsworth P, Mavris D (2003) Gaussian process meta-modeling: comparison of Gaussian process training methods. In AIAA's 3rd Annual Aviation Technology, Integration, and Operations (ATIO) Forum (p. 6761)

20. Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21(4):345–383

21. Kinney JB, Atwal GS (2014) Equitability, mutual information, and the maximal information coefficient. Proc Natl Acad Sci 111(9):3354–3359

22. Krige DG (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J South Afr Inst Min Metall 52(6):119–139

23. Kulfan B, Bussoletti J (2006) “Fundamental” parameteric geometry representations for aircraft component shapes. Paper presented at the 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference: The Modeling and Simulation Frontier for Multidisciplinary Design Optimization

24. Lee K, Cho H, Lee I (2019) Variable selection using Gaussian process regression-based metrics for high-dimensional model approximation with limited data. Struct Multidiscip Optim 59(5):1439–1454

25. Li C, Wang P, Dong H, Wang X (2018) A simplified shape optimization strategy for blended-wing-body underwater gliders. Struct Multidiscip Optim 58(5):2189–2202

26. Liu B, Zhang Q, Gielen GG (2014) A Gaussian process surrogate model assisted evolutionary algorithm for medium scale expensive optimization problems. IEEE Trans Evol Comput 18(2):180–192

27. Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. TECHNOMETRICS 51(4):366–376

28. Lophaven SN, Nielsen HB, Søndergaard J (2002) Aspects of the matlab toolbox DACE. IMM, Informatics and Mathematical Modelling, The Technical University of Denmark

29. Matheron G (1963) Principles of geostatistics. Econ Geol 58(8):1246–1266

30. Mckay MD, Beckman RJ, Conover WJ (1979) Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245

31. Michalewicz Z, Schoenauer M (2014) Evolutionary algorithms for constrained parameter optimization problems. Evol Comput 4(1):1–32

32. Mullur A, Messac A (2005) Extended radial basis functions: more flexible and effective metamodeling. AIAA J 43(6):1306–1315

33. Powell MJ (1994) A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Advances in optimization and numerical analysis. Springer, Dordrecht, pp 51–67

34. Rasmussen CE, Williams CKI (2005) Gaussian processes for machine learning. MIT Press, Cambridge

35. Reshef DN, Reshef YA, Finucane HK, Grossman SR, Mcvean G, Turnbaugh PJ et al (2011) Detecting novel associations in large data sets. Science 334(6062):1518–1524

36. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci:409–423

37. Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D et al (2008) Global sensitivity analysis: the primer. Wiley, Hoboken

38. Sasena MJ (2002) Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations (Doctoral dissertation, University of Michigan)

39. Schmit LA, Farshi B (1974) Some approximation concepts for structural synthesis. AIAA J 12(5):692–699

40. Shan S, Wang GG (2010) Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multidiscip Optim 41(2):219–241

41. Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–222

42. Sobol IM (2001) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math Comput Simul 55(1–3):271–280

43. Speed T (2011) A correlation for the 21st century. Science 334(6062):1502–1503

44. Sun GL, Li JB, Dai J, Song ZC, Lang F (2018). Feature selection for IoT based on maximal information coefficient. Futur Gener Comput Syst 89:606–616

45. Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E (2016a) High dimensional kriging metamodelling utilising gradient information. Appl Math Model 40(9–10):5256–5270

46. Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E (2016b) Performance study of gradient-enhanced kriging. Eng Comput 32(1):15–34

47. Wang H, Jin Y, Doherty J (2017) Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems. IEEE Trans Cybern 47(9):2664–2677

48. Wu D, Wang GG (2018) Knowledge assisted optimization for large-scale problems: a review and proposition. In: ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp V02BT03A032–V02BT03A032. American Society of Mechanical Engineers

49. Wu D, Coatanea E, Wang GG (2017) Dimension reduction and decomposition using causal graph and qualitative analysis for aircraft concept design optimization. In: ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, New York, pp V02BT03A035–V02BT03A035

50. Zhao X, Deng W, Shi Y (2013) Feature selection with attributes clustering by maximal information coefficient. Procedia Comput Sci 17(2):70–79

## Funding

This research was financially supported by the National Natural Science Foundation of China (Grant No. 51875466 and Grant No. 51805436).

## Author information

Authors

### Corresponding author

Correspondence to Peng Wang.

## Ethics declarations

### Conflict of interest

The authors declare that they have no conflict of interest.

### Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Responsible Editor: Nam Ho Kim

## Appendix: Examples of spatial correlation functions

### Appendix: Examples of spatial correlation functions

The proposed method (KMIC) combines kriging with maximal information coefficient and constructs a new maximum likelihood estimation problem (Eq. 22). Then the number of parameters we need to optimize when estimating hyper-parameters is reduced to one. It seems that KMIC constructs a new spatial correlation function which depends on only one parameter. In this article, Gaussian exponential correlation function is applied with the proposed method. For the other spatial correlation functions, the proposed method is also suitable. Appendix Table 8 presents the most popular examples of spatial correlation functions. Appendix Table 9 presents the new KMIC spatial correlation functions based on the examples given in Appendix Table 8.

## Rights and permissions

Reprints and Permissions