Skip to main content
SpringerLink
Log in

Derivative-Free Optimization

  • Chapter
Computational Optimization, Methods and Algorithms

Part of the book series: Studies in Computational Intelligence ((SCI,volume 356))

Abstract

In many engineering applications it is common to find optimization problems where the cost function and/or constraints require complex simulations. Though it is often, but not always, theoretically possible in these cases to extract derivative information efficiently, the associated implementation procedures are typically non-trivial and time-consuming (e.g., adjoint-based methodologies). Derivative-free (non-invasive, black-box) optimization has lately received considerable attention within the optimization community, including the establishment of solid mathematical foundations for many of the methods considered in practice. In this chapter we will describe some of the most conspicuous derivative-free optimization techniques. Our depiction will concentrate first on local optimization such as pattern search techniques, and other methods based on interpolation/approximation. Then, we will survey a number of global search methodologies, and finally give guidelines on constraint handling approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Pironneau, O.: On optimum design in fluid mechanics. Journal of Fluid Mechanics 64, 97–110 (1974)

    Article  MATH  MathSciNet  Google Scholar 

  2. Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Review 45(3), 385–482 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  3. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. MPS-SIAM (2009)

    Google Scholar 

  4. Gilmore, P., Kelley, C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM Journal on Optimization 5, 269–285 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  5. Kelley, C.T.: Iterative Methods for Optimization. In: Frontiers in Applied Mathematics, SIAM, Philadelphia (1999)

    Google Scholar 

  6. Dennis Jr., J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. SIAM’s Classics in Applied Mathematics Series. SIAM, Philadelphia (1996)

    MATH  Google Scholar 

  7. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  8. Schilders, W.H.A., van der Vorst, H.A., Rommes, J.: Model Order Reduction: Theory, Research Aspects and Applications. Mathematics in Industry Series. Springer, Heidelberg (2008)

    Book  MATH  Google Scholar 

  9. Conn, A.R., Gould, N.I.M.: Toint, Ph.L.: Trust-Region Methods. MPS-SIAM Series on Optimization. MPS-SIAM (2000)

    Google Scholar 

  10. Meza, J.C., Martinez, M.L.: On the use of direct search methods for the molecular conformation problem. Journal of Computational Chemistry 15, 627–632 (1994)

    Article  Google Scholar 

  11. Booker, A.J., Dennis Jr., J.E., Frank, P.D., Moore, D.W., Serafini, D.B.: Optimization using surrogate objectives on a helicopter test example. In: Borggaard, J.T., Burns, J., Cliff, E., Schreck, S. (eds.) Computational Methods for Optimal Design and Control, pp. 49–58. Birkháuser, Basel (1998)

    Google Scholar 

  12. Marsden, A.L., Wang, M., Dennis Jr., J.E., Moin, P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. Journal of Fluid Mechanics 572, 13–36 (2003)

    Article  MathSciNet  Google Scholar 

  13. Duvigneau, R., Visonneau, M.: Hydrodynamic design using a derivative-free method. Structural and Multidisciplinary Optimization 28, 195–205 (2004)

    Article  Google Scholar 

  14. Fowler, K.R., Reese, J.P., Kees, C.E., Dennis Jr., J.E., Kelley, C.T., Miller, C.T., Audet, C., Booker, A.J., Couture, G., Darwin, R.W., Farthing, M.W., Finkel, D.E., Gablonsky, J.M., Gray, G., Kolda, T.G.: Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Advances in Water Resources 31(5), 743–757 (2008)

    Article  Google Scholar 

  15. Oeuvray, R., Bierlaire, M.: A new derivative-free algorithm for the medical image registration problem. International Journal of Modelling and Simulation 27, 115–124 (2007)

    Article  Google Scholar 

  16. Marsden, A.L., Feinstein, J.A., Taylor, C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Computational Methods in Applied Mechanics and Engineering 197, 1890–1905 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  17. Artus, V., Durlofsky, L.J., Onwunalu, J.E., Aziz, K.: Optimization of nonconventional wells under uncertainty using statistical proxies. Computational Geosciences 10, 389–404 (2006)

    Article  MATH  Google Scholar 

  18. Dadashpour, M., Echeverría Ciaurri, D., Mukerji, T., Kleppe, J., Landrø, M.: A derivative-free approach for the estimation of porosity and permeability using time-lapse seismic and production data. Journal of Geophysics and Engineering 7, 351–368 (2010)

    Article  Google Scholar 

  19. Echeverría Ciaurri, D., Isebor, O.J., Durlofsky, L.J.: Application of derivativefree methodologies for generally constrained oil production optimization problems. International Journal of Mathematical Modelling and Numerical Optimisation 2(2), 134–161 (2011)

    Article  Google Scholar 

  20. Onwunalu, J.E., Durlofsky, L.J.: Application of a particle swarm optimization algorithm for determining optimum well location and type. Computational Geosciences 14, 183–198 (2010)

    Article  MATH  Google Scholar 

  21. Zhang, H., Conn, A.R., Scheinberg, K.: A derivative-free algorithm for leastsquares minimization. SIAM Journal on Optimization 20(6), 3555–3576 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  22. Torczon, V.: On the convergence of pattern search algorithms. SIAM Journal on Optimization 7(1), 1–25 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  23. Audet, C., Dennis Jr., J.E.: Analysis of generalized pattern searches. SIAM Journal on Optimization 13(3), 889–903 (2002)

    Article  MathSciNet  Google Scholar 

  24. Audet, C., Dennis Jr., J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM Journal on Optimization 17(1), 188–217 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  25. Hooke, R., Jeeves, T.A.: Direct search solution of numerical and statistical problems. Journal of the ACM 8(2), 212–229 (1961)

    Article  MATH  Google Scholar 

  26. Powell, M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. Technical report DAMTP 2004/NA5, Dept. of Applied Mathematics and Theoretical Physics, University of Cambridge (2004)

    Google Scholar 

  27. Oeuvray, R., Bierlaire, M.: BOOSTERS: a derivative-free algorithm based on radial basis functions. International Journal of Modelling and Simulatio 29(1), 26–36 (2009)

    Google Scholar 

  28. Metropolis, N., Rosenbluth, A., Teller, A., Teller, E.: Equation of state calculations by fast computing machines. Chemical Physics 21(6), 1087–1092 (1953)

    Google Scholar 

  29. Kirkpatrick, S., Gelatt Jr., C.D., Vecchi, M.: Optimization by simulated annealing. Science 220(4498), 671–680 (1983)

    Article  MathSciNet  Google Scholar 

  30. Glover, F.: Tabu search – part I. ORSA Journal on Computing 1(3), 190–206 (1990)

    Google Scholar 

  31. Glover, F.: Tabu search – part II. ORSA Journal on Computing 2(1), 4–32 (1990)

    MATH  Google Scholar 

  32. Dorigo, M.: Optimization, Learning and Natural Algorithms. PhD thesis, Dept. of Electronics, Politecnico di Milano (1992)

    Google Scholar 

  33. Dorigo, M., Stützle, T.: Ant Colony Optimization. Prentice-Hall, Englewood Cliffs (2004)

    Book  MATH  Google Scholar 

  34. Farmer, J., Packard, N., Perelson, A.: The immune system, adaptation and machine learning. Physica 2, 187–204 (1986)

    MathSciNet  Google Scholar 

  35. Castro, L.N.D., Timmis, J.: Artificial Immune Systems: A New Computational Intelligence. Springer, Heidelberg (2002)

    MATH  Google Scholar 

  36. Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press (1975)

    Google Scholar 

  37. Fogel, D.B.: Artificial Intelligence through Simulated Evolution. Wiley, Chichester (1966)

    MATH  Google Scholar 

  38. Beyer, H.-G., Schwefel, H.-P.: Evolution strategies - a comprehensive introduction. Natural Computing 1, 3–52 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  39. Rechenberg, I.: Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution. Frommann-Holzboog (1973)

    Google Scholar 

  40. Schwefel, H.-P.: Numerische Optimierung von Computer-Modellen mittel der Evolutionsstrategie. Birkhäuser, Basel (1977)

    Google Scholar 

  41. Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, Reading (1989)

    MATH  Google Scholar 

  42. Holland, J.H.: Hidden Order: How Adaptation Builds Complexity. Addison- Wesley, London (1995)

    Google Scholar 

  43. Beyer, H.-G.: An alternative explanation for the manner in which genetic algorithms operate. BioSystems 41(1), 1–15 (1997)

    Article  Google Scholar 

  44. Schwefel, H.-P.: Adaptive mechanismen in der biologischen evolution und ihr einfluss auf die evolutionsgeschwindigkeit. In: Interner Bericht der Arbeitsgruppe Bionik und Evolutionstechnik am Institut für Mess- und Regelungstechnik, TU Berlin (1974)

    Google Scholar 

  45. Beyer, H.-G., Sendhoff, B.: Covariance matrix adaptation revisited – the CMSA evolution strategy –. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 123–132. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  46. Ostermeier, A., Gawelczyk, A., Hansen, N.: A derandomized approach to selfadaptation of evolution strategies. Evolutionary Computation 2(4), 369–380 (1994)

    Article  Google Scholar 

  47. Teytaud, F., Teytaud, O.: Why one must use reweighting in estimation of distributionalgorithms. In: Proceedings of the 11th Annual conference on Genetic and Evolutionary Computation (GECCO 2009), pp. 453–460 (2009)

    Google Scholar 

  48. Grahl, J., Bosman, P.A.N., Rothlauf, F.: The correlation-triggered adaptive variance scaling idea. In: Proceedings of the 8th Annual conference on Genetic and Evolutionary Computation (GECCO 2006), pp. 397–404 (2006)

    Google Scholar 

  49. Bosman, P.A.N., Grahl, J., Thierens, D.: Enhancing the performance of maximum–likelihood gaussian eDAs using anticipated mean shift. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 133–143. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  50. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, pp. 1942–1948 (1995)

    Google Scholar 

  51. Storn, R., Price, K.: Differential evolution - a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization 11, 341–359 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  52. Chakraborty, U.: Advances in Differential Evolution. SCI. Springer, Heidelberg (2008)

    Book  MATH  Google Scholar 

  53. Griffin, J.D., Kolda, T.G.: Nonlinearly-constrained optimization using asynchronous parallel generating set search. Technical report SAND2007-3257, Sandia National Laboratories (2007)

    Google Scholar 

  54. Hestenes, M.R.: Multiplier and gradients methods. Journal of Optimization Theory and Applications 4(5), 303–320 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  55. Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, London (1969)

    Google Scholar 

  56. Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM Journal on Numerical Analysis 28(2), 545–572 (1991)

    Article  MATH  MathSciNet  Google Scholar 

  57. Conn, A.R., Gould, N.I.M., Toint, P.L.: LANCELOT: A Fortran Package for Large-Scale Nonlinear Optimization (Release A). Computational Mathematics. Springer, Heidelberg (1992)

    MATH  Google Scholar 

  58. Lewis, R.M., Torczon, V.: A direct search approach to nonlinear programming problems using an augmented Lagrangian method with explicit treatment of the linear constraints. Technical report WM-CS-2010-01, Dept. of Computer Science, College of William & Mary (2010)

    Google Scholar 

  59. Fletcher, R., Leyffer, S., Toint, P.L.: A brief history of filter methods. Technical report ANL/MCS/JA-58300, Argonne National Laboratory (2006)

    Google Scholar 

  60. Audet, C., Dennis Jr., J.E.: A pattern search filter method for nonlinear programming without derivatives. SIAM Journal on Optimization 14(4), 980–1010 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  61. Abramson, M.A.: NOMADm version 4.6 User’s Guide. Dept. of Mathematics and Statistics, Air Force Institute of Technology (2007)

    Google Scholar 

  62. Belur, S.V.: CORE: constrained optimization by random evolution. In: Koza, J.R. (ed.) Late Breaking Papers at the Genetic Programming 1997 Conference, pp. 280–286 (1997)

    Google Scholar 

  63. Coello Coello, C.A.: Theoretical and numerical constraint handling techniques used with evolutionary algorithms: a survey of the state of the art. Computer Methods in Applied Mechanics and Engineering 191(11-12), 1245–1287 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  64. Parmee, I.C., Purchase, G.: The development of a directed genetic search technique for heavily constrained design spaces. In: Parmee, I.C. (ed.) Proceedings of the Conference on Adaptive Computing in Engineering Design and Control, pp. 97–102. University of Plymouth (1994)

    Google Scholar 

  65. Surry, P.D., Radcliffe, N.J., Boyd, I.D.: A multi-objective approach to constrained optimisation of gas supply networks: the COMOGA method. In: Fogarty, T.C. (ed.) AISB-WS 1995. LNCS, vol. 993, pp. 166–180. Springer, Heidelberg (1995)

    Google Scholar 

  66. Coello Coello, C.A.: Treating constraints as objectives for single-objective evolutionary optimization. Engineering Optimization 32(3), 275–308 (2000)

    Article  Google Scholar 

  67. Coello Coello, C.A.: Constraint handling through a multiobjective optimization technique. In: Proceedings of the 8th Annual conference on Genetic and Evolutionary Computation (GECCO 1999), pp. 117–118 (1999)

    Google Scholar 

  68. Jimenez, F., Verdegay, J.L.: Evolutionary techniques for constrained optimization problems. In: Zimmermann, H.J. (ed.) 7th European Congress on Intelligent Techniques and Soft Computing (EUFIT 1999). Springer, Heidelberg (1999)

    Google Scholar 

  69. Mezura-Montes, E., Coello Coello, C.A.: Constrained optimization via multiobjective evolutionary algorithms. In: Knowles, J., Corne, D., Deb, K., Deva, R. (eds.) Multiobjective Problem Solving from Nature. Natural Computing Series, pp. 53–75. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  70. Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6(2), 182–197 (2002)

    Article  Google Scholar 

  71. Zitzler, E., Laumanns, M., Thiele, L.: SPEA2: improving the strength Pareto evolutionary algorithm for multiobjective optimization. In: Evolutionary Methods for Design, Optimisation and Control with Application to Industrial Problems (EUROGEN 2001), pp. 95–100 (2002)

    Google Scholar 

  72. Schoenauer, M., Xanthakis, S.: Constrained GA optimization. In: Forrest, S. (ed.) Proceedings of the 5th International Conference on Genetic Algorithms (ICGA 1993), pp. 573–580. Morgan Kaufmann, San Francisco (1993)

    Google Scholar 

  73. Montes, E.M., Coello Coello, C.A.: A simple multi-membered evolution strategy to solve constrained optimization problems. IEEE Transactions on Evolutionary Computation 9(1), 1–17 (2005)

    Article  Google Scholar 

  74. Liang, J., Suganthan, P.: Dynamic multi-swarm particle swarm optimizer with a novel constraint-handling mechanism. In: Yen, G.G., Lucas, S.M., Fogel, G., Kendall, G., Salomon, R., Zhang, B.-T., Coello Coello, C.A., Runarsson, T.P. (eds.) Proceedings of the 2006 IEEE Congress on Evolutionary Computation (CEC 2006), pp. 9–16. IEEE Press, Los Alamitos (2006)

    Google Scholar 

  75. Raidl, G.R.: A unified view on hybrid metaheuristics. In: Almeida, F., Blesa Aguilera, M.J., Blum, C., Moreno Vega, J.M., Pérez Pérez, M., Roli, A., Sampels, M. (eds.) HM 2006. LNCS, vol. 4030, pp. 1–12. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  76. Talbi, E.G.: A taxonomy of hybrid metaheuristics. Journal of Heuristics 8(5), 541–564 (2002)

    Article  Google Scholar 

  77. Griewank, A.: Generalized descent for global optimization. Journal of Optimization Theory and Applications 34, 11–39 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  78. Duran Toksari, M., Güner, E.: Solving the unconstrained optimization problem by a variable neighborhood search. Journal of Mathematical Analysis and Applications 328(2), 1178–1187 (2007)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. UC Berkeley, Berkeley, CA, 94704, USA

    Oliver Kramer

  2. Department of Energy Resources Engineering, Stanford University, Stanford, CA, 94305-2220, USA

    David Echeverría Ciaurri

  3. Engineering Optimization & Modeling Center, School of Science and Engineering, Reykjavik University, Menntavegur 1, 101, Reykjavik, Iceland

    Slawomir Koziel

Authors
  1. Oliver Kramer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Echeverría Ciaurri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Slawomir Koziel
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Science and Engineering Engineering Optimization & Modeling Center, Reykjavik University, Menntavegur 1, 101, Reykjavik, Iceland

    Slawomir Koziel

  2. Mathematics and Scientific Computing, National Physical Laboratory, TW11 0LW, Teddington, UK

    Xin-She Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Kramer, O., Ciaurri, D.E., Koziel, S. (2011). Derivative-Free Optimization. In: Koziel, S., Yang, XS. (eds) Computational Optimization, Methods and Algorithms. Studies in Computational Intelligence, vol 356. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20859-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-20859-1_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-20858-4

  • Online ISBN: 978-3-642-20859-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation