A benchmark of kriging-based infill criteria for noisy optimization

Abstract

Responses of many real-world problems can only be evaluated perturbed by noise. In order to make an efficient optimization of these problems possible, intelligent optimization strategies successfully coping with noisy evaluations are required. In this article, a comprehensive review of existing kriging-based methods for the optimization of noisy functions is provided. In summary, ten methods for choosing the sequential samples are described using a unified formalism. They are compared on analytical benchmark problems, whereby the usual assumption of homoscedastic Gaussian noise made in the underlying models is meet. Different problem configurations (noise level, maximum number of observations, initial number of observations) and setups (covariance functions, budget, initial sample size) are considered. It is found that the choices of the initial sample size and the covariance function are not critical. The choice of the method, however, can result in significant differences in the performance. In particular, the three most intuitive criteria are found as poor alternatives. Although no criterion is found consistently more efficient than the others, two specialized methods appear more robust on average.

Appendices

Appendix A: Sample results for four infill criteria

The research questions addressed in the actual benchmark are particularly interesting for a practical user. For the research on NKO, also the effect of using specific information in the infill criteria is of interest. This section aims at illustrating some differences between the infill criteria and provide some insight on the performance differences observed in Section 5. We particularly focus on the effects of the consideration of the current accuracy of an observation (e. g., by using a Kriging quantile rather than the Kriging mean) and the use of replications (Forrester’s approach does not perform any replications while other infill criteria may do). The runs results shown here are somehow typical, but there are of course many situations for which the above comments do not apply.

We show here four runs of the AKG, AEI, EQ90 and RI criteria on the Goldsteinprice with 50 % noise, large initial DOE and small budget. Out of the four, EQ90 (Fig. 5c) is clearly less exploratory, since almost all the added observations form a single cluster. This cluster reduces a lot the kriging uncertainty locally, but does not allow global exploration or accurate convergence. Inversely, RI (Fig. 5d) does not allow repetitions which results here in very good exploration, but prevents from identifying the valley of the global optimum. The two other criteria, on this example, offer the right trade-off, with a relatively wide exploration and accurate identification of the optimum region.

Fig. 5

Sample of optimization results obtained on the Goldsteinprice with 50 % noise, large initial DOE and small budget. Initial observations are represented with filled triangles. The contour lines represent the actual function

Appendix B: ANOVA tables of the linear models

Table 7 ANOVA table for the Branin test function

Table 8 ANOVA table for the Goldstein-Price test function

Table 9 ANOVA table for the Hartman4 test function

Table 10 ANOVA table for the Rosenbrock test function

Table 11 ANOVA table for the Hartman6 test function

Table 12 ANOVA table for the Sphere test function