Skip to main content
SpringerLink
Log in

Statistics of Nadaraya-Watson estimator errors in surrogate-based optimization

  • Published:
Optimization and Engineering Aims and scope Submit manuscript

Abstract

For the optimization under uncertainty problem, there has been recent interest in coupling trust-region methods with surrogate surfaces or function approximations. There are many theoretical and statistical issues that must be carefully considered in following such an approach. Herein, the Nadaraya-Watson estimator is used for the smooth function approximation, and the effects of observation noise and random sampling on estimator error are examined. For the fundamental optimization problem where the exact function is quadratic, analytical results are derived for the mean-square error of the difference and gradient of the function. It is also shown how these statistics are related to the trust-region method, how the analytical results can be used to determine the bandwidth of the kernel of the estimator, and how third-order terms can affect the error statistics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alexandrov N, JED Jr, Lewis RM, Torczon V (1997) A trust region framework for managing the use of approximation models in optimization. ICASE Report 97-50, NASA

  • Carter RG (1991) On the global convergence of trust region algorithms using inexact gradient information. J Numer Anal 28(1):251–265

    Article  MATH  MathSciNet  Google Scholar 

  • Carter RG (1993) Numerical experience with a class of algorithms for nonlinear optimization using inexact fuction and gradient information. J Sci Comput 14:368–388

    MATH  MathSciNet  Google Scholar 

  • Collomb G (1985) Nonparametric regression—an up-to-date bibliography. Statistics 2:309–324

    MathSciNet  Google Scholar 

  • Conn AR, Scheinberg K, Toint P (1997) On the convergence of derivative-free methods for unconstrained opimization. In: Buhmann MD, Iserles A (eds) Approximation theory and optimization: tributes to M.J.D. Powell. Cambridge University Press, pp 83–108

  • Eldred M, Giunta A, Wojtkiewicz JSF, Trucano T (2002) Formulations for surrogate-based optimization under uncertainty. In: Proceedings of the 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, number paper AIAA-2002-5585, Atlanta, GA

  • Gasser T, Maller H (1984) Estimating regression functions and their derivatives by the kernel method. Scand J Stat 11:171–185

    MATH  Google Scholar 

  • Giunta AA, Eldred MS (2000) Implementation of a trust region model management strategy in the DAKOTA optimization toolkit. In: Proceedings of the 8th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA

  • Hardle W (1990) Applied nonparametric regression, econometric society monographs No. 19. Cambridge University Press, London

  • Hardle W, Marron JS (1985) Optimal bandwidth selection in nonparametric regression function estimation. Ann Stat 13:1465–1481

    MathSciNet  Google Scholar 

  • More JJ (1983) Recent developments in algorithms and software for trust region methods. In: Bachem A, Grotschel M, Korte B (eds) Mathematical Programming, the State of the Art. Springer-Verlag, New York, pp 258–287

  • Nadaraya E (1964) On non-parametric estimates of density functions and regression curves. Theory Probab Appl 10:186–190

    Article  Google Scholar 

  • Rodriguez JF, Renaud JE, Watson LT (1997) Trust region augmented Lagrangian methods for sequential response surface approximation and optimization. In: Proceedings of the ASME Design Engineering Technical Conference

  • Stone C (1982) Optimal rates of convergence for nonparametric regression. Ann Stat 10:1040–1053

    MATH  Google Scholar 

  • Trosset M, Alexandrov N, Watson L (2003) New methods for robust design using computer simulation. In: Proceedings of the Section on Physical and Engineering Sciences. American Statistical Association

  • Venter G, Haftka RT, Starnes JH (1998) Construction of response surface approximations for design optimization. AIAA Journal 36(12):2242–2249

    Article  Google Scholar 

  • Vieu P (1991) Nonparametric regression: Optimal local bandwidth choice. J R Stat Soc B, 453–464

  • Wand M (1992) Error analysis for general multivariate kernel estimatiors. Nonparametr Stat 2:1–15

    MathSciNet  MATH  Google Scholar 

  • Watson GS (1964) Smooth regression analysis. Sankhya 26:359–372

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Johns Hopkins University, Baltimore, Maryland, 21218, USA

    Zailong Wan & Takeru Igusa

Authors
  1. Zailong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Takeru Igusa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Takeru Igusa.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wan, Z., Igusa, T. Statistics of Nadaraya-Watson estimator errors in surrogate-based optimization. Optim Eng 7, 385–397 (2006). https://doi.org/10.1007/s11081-006-9980-9

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11081-006-9980-9

Keywords

Search

Navigation