Optimization and Engineering

, Volume 19, Issue 1, pp 213–238 | Cite as

Locally weighted regression models for surrogate-assisted design optimization

  • Bastien TalgornEmail author
  • Charles Audet
  • Sébastien Le Digabel
  • Michael Kokkolaras


We consider engineering design optimization problems where the objective and/or constraint functions are evaluated by means of computationally expensive blackboxes. Our practical optimization strategy consists of solving surrogate optimization problems in the search step of the mesh adaptive direct search algorithm. In this paper, we consider locally weighted regression models to build the necessary surrogates, and present three ideas for appropriate and effective use of locally weighted scatterplot smoothing (LOWESS) models for surrogate optimization. First, a method is proposed to reduce the computational cost of LOWESS models. Second, a local scaling coefficient is introduced to adapt LOWESS models to the density of neighboring points while retaining smoothness. Finally, an appropriate order error metric is used to select the optimal shape coefficient of the LOWESS model. Our surrogate-assisted optimization approach utilizes LOWESS models to both generate and rank promising candidates found in the search and poll steps. The “real” blackbox functions that govern the original optimization problem are then evaluated at these ranked candidates with an opportunistic strategy, reducing CPU time significantly. Computational results are reported for four engineering design problems with up to six variables and six constraints. The results demonstrate the effectiveness of the LOWESS models as well as the order error metric for surrogate optimization.


Local regression Order error Surrogate models Derivative-free optimization MADS 



All the authors acknowledge the partial support of FRQNT Grant PR-182098; B. Talgorn and M. Kokkolaras are also grateful for the partial support of NSERC/Hydro-Québec Grant EGP2 498903-16; such support does not constitute an endorsement by the sponsors of the opinions expressed in this article. B. Talgorn would like to thank Stéphane Alarie of IREQ, Patricia Gillett-Kawamoto of GERAD and Sylvain Arreckx of GERAD for their invaluable insights and comments.


  1. Abramson MA, Audet C, Couture G, Dennis JE Jr, Le Digabel S, Tribes C (2015) The NOMAD project. Software available at
  2. Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37(3):279–294CrossRefGoogle Scholar
  3. Adams BM, Ebeida MS, Eldred MS, Geraci G, Jakeman JD, Maupin KA, Monschke JA, Swiler LP, Stephens JA, Vigil DM, Wildey TM, Bohnhoff WJ, Dalbey KR, Eddy JP, Frye JR, Hooper RW, Hu KT, Hough PD, Khalil M, Ridgway EM, Rushdi A (2014) Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: version 6.6 users manual. Technical report SAND2014-4633, Sandia National LaboratoriesGoogle Scholar
  4. Arora J (2004) Introduction to optimum design. Academic Press, LondonGoogle Scholar
  5. Atkeson CG, Moore AW, Schaal S (1997) Locally weighted learning for control. Artif Intell Rev 11(1):11–73CrossRefGoogle Scholar
  6. Audet C, Dennis JE Jr (2006) Mesh adaptive direct search algorithms for constrained optimization. SIAM J Optim 17(1):188–217MathSciNetCrossRefzbMATHGoogle Scholar
  7. Audet C, Dennis JE Jr (2009) A progressive barrier for derivative-free nonlinear programming. SIAM J Optim 20(1):445–472MathSciNetCrossRefzbMATHGoogle Scholar
  8. Audet C, Kokkolaras M, Le Digabel S, Talgorn B (2017) Order-based error for managing ensembles of surrogates in mesh adaptive direct search. J Glob Optim. ISSN: 1573-2916.
  9. Belegundu AD (1982) A study of mathematical programming methods for structural optimization. University of Iowa, Iowa CityGoogle Scholar
  10. Booker AJ, Dennis JE Jr, Frank PD, Serafini DB, Torczon V, Trosset MW (1999) A rigorous framework for optimization of expensive functions by surrogates. Struct Multidiscip Optim 17(1):1–13CrossRefGoogle Scholar
  11. Chen X, Kelley CT (2016) Optimization with hidden constraints and embedded Monte Carlo computations. Optim Eng 17(1):157–175MathSciNetCrossRefzbMATHGoogle Scholar
  12. Choi SC, Wette R (1969) Maximum likelihood estimation of the parameters of the Gamma distribution and their bias. Technometrics 11(4):683–690CrossRefzbMATHGoogle Scholar
  13. Cleveland WS (1979) Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc 74:829–836MathSciNetCrossRefzbMATHGoogle Scholar
  14. Cleveland WS (1981) LOWESS: a program for smoothing scatterplots by robust locally weighted regression. Am Stat 35(1):54CrossRefGoogle Scholar
  15. Cleveland WS, Devlin SJ (1988) Locally weighted regression: an approach to regression analysis by local fitting. J Am Stat Assoc 83:596–610CrossRefzbMATHGoogle Scholar
  16. Cleveland WS, Devlin SJ, Grosse E (1988) Regression by local fitting: methods, properties, and computational algorithms. J Econ 37(1):87–114MathSciNetCrossRefGoogle Scholar
  17. Cohen RA (1999) An introduction to PROC LOESS for local regression. In: Proceedings of the 24th SAS users group international conference.
  18. Conn AR, Le Digabel S (2013) Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim Methods Softw 28(1):139–158MathSciNetCrossRefzbMATHGoogle Scholar
  19. Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78(382):316–331MathSciNetCrossRefzbMATHGoogle Scholar
  20. Fan J (1993) Local linear regression smoothers and their minimax efficiencies. Ann Stat 21(1):196–216MathSciNetCrossRefzbMATHGoogle Scholar
  21. Fan J, Gijbels I (1992) Variable bandwidth and local linear regression smoothers. Ann Stat 20(4):2008–2036MathSciNetCrossRefzbMATHGoogle Scholar
  22. Fletcher R, Leyffer S (2002) Nonlinear programming without a penalty function. Math Program Ser A 91:239–269MathSciNetCrossRefzbMATHGoogle Scholar
  23. Gablonsky J (2001) DIRECT version 2.0 userguide technical report. Technical report CRSC-TR01-08, North Carolina State University, Center for Research in Scientific Computation, Raleigh, NCGoogle Scholar
  24. Garg H (2014) Solving structural engineering design optimization problems using an artificial bee colony algorithm. J Ind Manag Optim 10(3):777–794MathSciNetCrossRefzbMATHGoogle Scholar
  25. Garneau M Lemyre (2015) Modelling of a solar thermal power plant for benchmarking blackbox optimization solvers. Master’s thesis, École Polytechnique de Montréal.
  26. Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning, 1st edn. Addison-Wesley, BostonzbMATHGoogle Scholar
  27. Gramacy RB, Gray GA, Le Digabel S, Lee HKH, Ranjan P, Wells G, Wild SM (2016) Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics 58(1):1–11MathSciNetCrossRefGoogle Scholar
  28. Gramacy RB, Le Digabel S (2015) The mesh adaptive direct search algorithm with treed Gaussian process surrogates. Pac J Optim 11(3):419–447MathSciNetzbMATHGoogle Scholar
  29. Haftka RT, Gurdal Z (1992) Elements of structural optimization, volume 11 of solid mechanics and its applications. Springer, BerlinCrossRefzbMATHGoogle Scholar
  30. Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning. Springer series in statistics. Springer, New YorkCrossRefzbMATHGoogle Scholar
  31. Kannan A, Wild SM (2012) Benefits of deeper analysis in simulation-based groundwater optimization problems. In: Proceedings of the XIX international conference on computational methods in water resources (CMWR 2012)Google Scholar
  32. Lall U, Moon YI, Kwon HH, Bosworth K (2006) Locally weighted polynomial regression: parameter choice and application to forecasts of the Great Salt Lake. Water Resour Res 42(W05422):1–11Google Scholar
  33. Le Digabel S (2011) Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans Math Softw 37(4):44:1–44:15MathSciNetCrossRefzbMATHGoogle Scholar
  34. Le Digabel S, Wild SM (2015) A taxonomy of constraints in simulation-based optimization. Technical report G-2015-57, Les cahiers du GERADGoogle Scholar
  35. Loader C (1999) Local regression and likelihood. Springer, New YorkzbMATHGoogle Scholar
  36. Macaulay FR (1931) Curve fitting and graduation. In: The smoothing of time series. National Bureau of Economic Research. NBER, pp 31–42Google Scholar
  37. Matott LS, Rabideau AJ, Craig JR (2006) Pump-and-treat optimization using analytic element method flow models. Adv Water Resour 29(5):760–775CrossRefGoogle Scholar
  38. Matott LS, Leung K, Sim J (2011) Application of MATLAB and Python optimizers to two case studies involving groundwater flow and contaminant transport modeling. Comput Geosci 37(11):1894–1899CrossRefGoogle Scholar
  39. McKay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245MathSciNetzbMATHGoogle Scholar
  40. Minka TP (2002) Estimating a Gamma distribution.
  41. Moré JJ, Wild SM (2009) Benchmarking derivative-free optimization algorithms. SIAM J Optim 20(1):172–191MathSciNetCrossRefzbMATHGoogle Scholar
  42. Müller H-G (1987) Weighted local regression and Kernel methods for nonparametric curve fitting. J Am Stat Assoc 82(397):231–238MathSciNetzbMATHGoogle Scholar
  43. Müller J, Piché R (2011) Mixture surrogate models based on Dempster–Shafer theory for global optimization problems. J Global Optim 51(1):79–104MathSciNetCrossRefzbMATHGoogle Scholar
  44. Natrella M (2010) NIST/SEMATECH e-handbook of statistical methods. NIST/SEMATECH.
  45. Orr MJL (1996) Introduction to radial basis function networks. Technical report, Center for Cognitive Science, University of EdinburghGoogle Scholar
  46. Pourmohamad T (2016) Combining multivariate stochastic process models with filter methods for constrained optimization. Ph.D. thesis. UC Santa Cruz: Statistics and Applied MathematicsGoogle Scholar
  47. Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucher PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28CrossRefGoogle Scholar
  48. Ruppert D, Wand MP (1994) Multivariate locally weighted least squares regression. Ann Stat 22(3):1346–1370MathSciNetCrossRefzbMATHGoogle Scholar
  49. Stone M (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. J Roy Stat Soc Ser B (Methodol) 39(1):44–47MathSciNetzbMATHGoogle Scholar
  50. Ting J-A, Vijayakumar S, Schaal S (2010) Locally weighted regression for control. Springer, Boston, pp 613–624Google Scholar
  51. Wilks DS (1990) Maximum likelihood estimation for the Gamma distribution using data containing zeros. J Clim 3(12):1495–1501CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  • Bastien Talgorn
    • 1
    Email author
  • Charles Audet
    • 2
  • Sébastien Le Digabel
    • 2
  • Michael Kokkolaras
    • 1
  1. 1.Department of Mechanical EngineeringMcGill University and GERADMontréalCanada
  2. 2.Department of Mathematics and Industrial EngineeringÉcole Polytechnique de Montréal and GERADMontréalCanada

Personalised recommendations