Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Derivative-free optimization: a review of algorithms and comparison of software implementations

Abstract

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

References

  1. 1

    Aarts E.H.L., van Laarhoven P.J.M.: Statistical cooling: a general approach to combinatorial optimization problems. Phillips J. Res. 40, 193–226 (1985)

  2. 2

    Abramson, M.A.: Pattern Search Algorithms for Mixed Variable General Constrained Optimization Problems. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (2002, Aug)

  3. 3

    Abramson, M.A.: NOMADm Version 4.5 User’s Guide. Air Force Institute of Technology, Wright-Patterson AFB, OH (2007)

  4. 4

    Abramson M.A., Asaki T.J., Dennis J.E. Jr., O’Reilly K.R., Pingel R.L.: Quantitative object reconstruction via Abel-based X-ray tomography and mixed variable optimization. SIAM J. Imaging Sci. 1, 322–342 (2008)

  5. 5

    Abramson M.A., Audet C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17, 606–609 (2006)

  6. 6

    Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., LeDigabel, S.: The Nomad project. http://www.gerad.ca/nomad/

  7. 7

    Abramson M.A., Audet C., Dennis J.E. Jr: Filter pattern search algorithms for mixed variable constrained optimization problems. Pac. J. Optim. 3, 477–500 (2007)

  8. 8

    Abramson M.A., Audet C., Dennis J.E. Jr, Le Digabel S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20, 948–966 (2009)

  9. 9

    Audet C.: Convergence results for generalized pattern search algorithms are tight. Optim. Eng. 5, 101–122 (2004)

  10. 10

    Audet C., Béchard V., Chaouki J.: Spent potliner treatment process optimization using a MADS algorithm. Optim. Eng. 9, 143–160 (2008)

  11. 11

    Audet C., Dennis J.E. Jr: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)

  12. 12

    Audet C., Dennis J.E. Jr: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20, 445–472 (2009)

  13. 13

    Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008)

  14. 14

    Barros P.A. Jr, Kirby M.R., Mavris D.N.: Impact of sampling techniques selection on the creation of response surface models. SAE Trans. J. Aerosp. 113, 1682–1693 (2004)

  15. 15

    Bartholomew-Biggs M.C., Parkhurst S.C., Wilson S.P.: Using DIRECT to solve an aircraft routing problem. Comput. Optim. Appl. 21, 311–323 (2002)

  16. 16

    Barton, R.R.: Metamodeling: A state of the art review. In: Proceedings of the 1994 Winter Simulation Conference, pp. 237–244 (1994)

  17. 17

    Bélisle C.J., Romeijn H.E., Smith R.L.: Hit-and-run algorithms for generating multivariate distributions. Math. Oper. Res. 18, 255–266 (1993)

  18. 18

    Bethke, J.D.: Genetic Algorithms as Function Optimizers. PhD thesis, Department of Computer and Communication Sciences, University of Michigan, Ann Arbor (1980)

  19. 19

    Björkman M., Holmström K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1, 373–397 (2000)

  20. 20

    Boender C.G.E., Rinnooy Kan A.H.G., Timmer G.T.: A stochastic method for global optimization. Math. Program. 22, 125–140 (1982)

  21. 21

    Boneh, A., Golan, A.: Constraints’ redundancy and feasible region boundedness by random feasible point generator (RFPG). In: 3rd European Congress on Operations Research (EURO III), Amsterdam (1979)

  22. 22

    Booker, A.J., Dennis, J.E., Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. In: ICASE Report, pp. 1–24 (1998)

  23. 23

    Booker A.J., Dennis J.E. Jr, Frank P.D., Serafini D.B., Torczon V.J., Trosset M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Optim. 17, 1–13 (1999)

  24. 24

    Booker A.J., Meckesheimer M., Torng T.: Reliability based design optimization using design explorer. Optim. Eng. 5, 179–205 (2004)

  25. 25

    Brent R.P.: Algorithms for Minimization without Derivatives. Prentice-Hall, Englewood Cliffs (1973)

  26. 26

    Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  27. 27

    Chiang T., Chow Y.: A limit theorem for a class of inhomogeneous Markov processes. Ann. Probab. 17, 1483–1502 (1989)

  28. 28

    COIN-OR Project. Derivative Free Optimization. http://projects.coin-or.org/Dfo

  29. 29

    COIN-OR Project. IPOPT 2.3.x A software package for large-scale nonlinear optimization. http://www.coin-or.org/Ipopt/ipopt-fortran.html

  30. 30

    Conn A.R., Gould N., Lescrenier M., Toint Ph.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer, Dordrecht (1994)

  31. 31

    Conn A.R., Scheinberg K., Toint P.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds) Approximation Theory and Optimization, Tribute to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1996)

  32. 32

    Conn A.R., Scheinberg K., Toint P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)

  33. 33

    Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings of AIAA St Louis Conference, pp. 1–11 (1998)

  34. 34

    Conn A.R., Scheinberg K., Vicente L.N.: Global convergence of general derivative-free trust-region algorithms to first and second order critical points. SIAM J. Optim. 20, 387–415 (2009)

  35. 35

    Conn A.R., Scheinberg K., Vicente L.N.: Introduction to derivative-free optimization. SIAM, Philadelphia (2009)

  36. 36

    Cox, D.D., John, S.: SDO: A statistical method for global optimization. In: Multidisciplinary Design Optimization (Hampton, VA, 1995), pp. 315–329. SIAM, Philadelphia (1997)

  37. 37

    Csendes T., Pál L., Sendín J.O.H., Banga J.R.: The GLOBAL optimization method revisited. Optim. Lett. 2, 445–454 (2008)

  38. 38

    Custódio A.L., Dennis J.E. Jr, Vicente L.N.: Using simplex gradients of nonsmooth functions in direct search methods. IMA J. Numer. Anal. 28, 770–784 (2008)

  39. 39

    Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. (to appear)

  40. 40

    Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18, 537–555 (2007)

  41. 41

    Custódio, A.L., Vicente, L.N.: SID-PSM: A Pattern Search Method Guided by Simplex Derivatives for Use in Derivative-Free Optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra (2008)

  42. 42

    Deming S.N., Parker L.R. Jr, Denton M.B.: A review of simplex optimization in analytical chemistry. Crit. Rev. Anal. Chem. 7, 187–202 (1974)

  43. 43

    Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010)

  44. 44

    Eberhart, R.,Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the 6th International Symposium on Micro Machine and Human Science, Nagoya, pp. 39–43 (1995)

  45. 45

    Eldred, M.S., Adams, B.M., Gay, D.M., Swiler, L.P., Haskell, K., Bohnhoff, W.J., Eddy, J.P., Hart, W.E., Watson, J-P, Hough, P.D., Kolda, T.G., Williams, P.J., Martinez-Canales, M.L., DAKOTA, A.: Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 4.2 User’s Manual. Sandia National Laboratories, Albuquerque (2008)

  46. 46

    Fan S.S., Zahara E.: A hybrid simplex search and particle swarm optimization for unconstrained optimization. Eur. J. Oper. Res. 181, 527–548 (2007)

  47. 47

    Finkel D.E., Kelley C.T.: Additive scaling and the DIRECT algorithm. J. Glob. Optim. 36, 597–608 (2006)

  48. 48

    Fowler K.R., Reese J.P., Kees C.E., Dennis J.E. Jr, Kelley C.T., Miller C.T., Audet C., Booker A.J., Couture G., Darwin R.W., Farthing M.W., Finkel D.E., Gablonsky J.M., Gray G., Kolda T.G.: A comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31, 743–757 (2008)

  49. 49

    Gablonsky, J.M.: Modifications of the DIRECT Algorithm. PhD thesis, Department of Mathematics, North Carolina State University, Raleigh (2001)

  50. 50

    Gilmore P., Kelley C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269–285 (1995)

  51. 51

    GLOBAL Library. http://www.gamsworld.org/global/globallib.htm

  52. 52

    Gray G., Kolda T., Sale K., Young M.: Optimizing an empirical scoring function for transmembrane protein structure determination. INFORMS J. Comput. 16, 406–418 (2004)

  53. 53

    Gutmann H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001)

  54. 54

    Han J., Kokkolaras M., Papalambros P.Y.: Optimal design of hybrid fuel cell vehicles. J. Fuel Cell Sci. Technol. 5, 041014 (2008)

  55. 55

    Hansen, N.: The CMA Evolution Strategy: A tutorial. http://www.lri.fr/hansen/cmaesintro.html

  56. 56

    Hansen N.: The CMA evolution strategy: a comparing review. In: Lozano, J.A., Larranaga, P., Inza, I., Bengoetxea, E. (eds) Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms, pp. 75–102. Springer, Berlin (2006)

  57. 57

    Hayes R.E., Bertrand F.H., Audet C., Kolaczkowski S.T.: Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81, 1192–1199 (2003)

  58. 58

    Holland J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)

  59. 59

    Holmström, K.: Private Communication (2009)

  60. 60

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com

  61. 61

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/CGO. Tomlab Optimization (2007). http://tomopt.com

  62. 62

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/OQNLP. Tomlab Optimization (2007). http://tomopt.com

  63. 63

    Holmström K., Quttineh N.-H., Edvall M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9, 311–339 (2008)

  64. 64

    Hooke R., Jeeves T.A.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8, 212–219 (1961)

  65. 65

    Huyer W., Neumaier A.: Global optimization by multilevel coordinate search. J. Glob. Optim. 14, 331–355 (1999)

  66. 66

    Huyer W., Neumaier A.: SNOBFIT—Stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35, 1–25 (2008)

  67. 67

    Hvattum L.M., Glover F.: Finding local optima of high-dimensional functions using direct search methods. Eur. J. Oper. Res. 195, 31–45 (2009)

  68. 68

    Ingber, L.: Adaptive Simulated Annealing (ASA). http://www.ingber.com/#ASA

  69. 69

    Järvi, T.: A Random Search Optimizer with an Application to a Max–Min Problem. Technical report, Pulications of the Institute for Applied Mathematics, University of Turku (1973)

  70. 70

    Jones D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21, 345–383 (2001)

  71. 71

    Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431–440. Kluwer, Boston (2001)

  72. 72

    Jones D.R., Perttunen C.D., Stuckman B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79, 157–181 (1993)

  73. 73

    Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)

  74. 74

    Kelley, C.T.: Users Guide for IMFIL version 1.0. http://www4.ncsu.edu/ctk/imfil.html

  75. 75

    Kelley C.T.: Detection and remediation of stagnation in the Nelder–Mead algorithm using a sufficient decrease condition. SIAM J. Optim. 10, 43–55 (1999)

  76. 76

    Kelley C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999)

  77. 77

    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, pp. 1942–1948 (1995)

  78. 78

    Kirkpatrick S., Gelatt C.D., Vecchi M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)

  79. 79

    Kolda T.G., Lewis R.M., Torczon V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385–482 (2003)

  80. 80

    Kolda T.G., Torczon V.J.: On the convergence of asynchronous parallel pattern search. SIAM J. Optim. 14, 939–964 (2004)

  81. 81

    Lagarias J.C., Reeds J.A., Wright M.H., Wright P.E.: Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112–147 (1998)

  82. 82

    LeDigabel, S.: NOMAD User Guide Version 3.3. Technical report, Les Cahiers du GERAD (2009)

  83. 83

    Lewis R.M., Torczon V.J.: Pattern search algorithms for bound constrained minimization. SIAM J. Optim. 9, 1082–1099 (1999)

  84. 84

    Lewis R.M., Torczon V.J.: Pattern search algorithms for linearly constrained minimization. SIAM J. Optim. 10, 917–941 (2000)

  85. 85

    Lewis R.M., Torczon V.J.: A globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds. SIAM J. Optim. 12, 1075–1089 (2002)

  86. 86

    Liepins G.E., Hilliard M.R.: Genetic algorithms: foundations and applications. Ann. Oper. Res. 21, 31–58 (1989)

  87. 87

    Lin Y., Schrage L.: The global solver in the LINDO API. Optim. Methods Softw. 24, 657–668 (2009)

  88. 88

    Lucidi S., Sciandrone M.: On the global convergence of derivative-free methods for unconstrained minimization. SIAM J. Optim. 13, 97–116 (2002)

  89. 89

    Lukšan, L., Vlček, J.: Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical report, Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps

  90. 90

    Marsden A.L., Feinstein J.A., Taylor C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)

  91. 91

    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Optimal aeroacustic shape design using the surrogate management framework. Optim. Eng. 5, 235–262 (2004)

  92. 92

    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. J. Fluid Mech. 5, 235–262 (2007)

  93. 93

    Matheron G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1967)

  94. 94

    McKinnon K.I.M.: Convergence of the Nelder–Mead simplex method to a nonstationary point. SIAM J. Optim. 9, 148–158 (1998)

  95. 95

    Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)

  96. 96

    Mongeau M., Karsenty H., Rouzé V., Hiriart-Urruty J.B.: Comparison of public-domain software for black box global optimization. Optim. Methods Softw. 13, 203–226 (2000)

  97. 97

    Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)

  98. 98

    Mugunthan P., Shoemaker C.A., Regis R.G.: Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models. Water Resour. Res. 41, W11427 (2005)

  99. 99

    Nelder J.A., Mead R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

  100. 100

    Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper 2007/76 (2007)

  101. 101

    Neumaier, A.: MCS: Global Optimization by Multilevel Coordinate Search. http://www.mat.univie.ac.at/neum/software/mcs/

  102. 102

    Neumaier A., Shcherbina O., Huyer W., Vinkó T.: A comparison of complete global optimization solvers. Math. Program. 103, 335–356 (2005)

  103. 103

    Oeuvray, R.: Trust-Region Methods Based on Radial Basis Functions with Application to Biomedical Imaging. PhD thesis, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne (2005, March)

  104. 104

    Orosz J.E., Jacobson S.H.: Finite-time performance analysis of static simulated annealing algorithms. Comput. Optim. Appl. 21, 21–53 (2002)

  105. 105

    Pintér, J.: Homepage of Pintér Consulting Services. http://www.pinterconsulting.com/

  106. 106

    Pintér J.D.: Global Optimization in Action: Continuous and Lipschitz Optimization. Algorithms, Implementations and Applications. Nonconvex Optimization and its Applications. Kluwer, Dordrecht (1995)

  107. 107

    Pintér, J.D., Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/LGO. Tomlab Optimization (2006). http://tomopt.com

  108. 108

    Plantenga, T.D.: HOPSPACK 2.0 User Manual. Technical Report SAND2009-6265, Sandia National Laboratories, Albuquerque (2009)

  109. 109

    Powell M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J-P. (eds) (eds.) Advances in Optimization and Numerical Analysis, pp. 51–67. Kluwer, Dordrecht (1994)

  110. 110

    Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994)

  111. 111

    Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998)

  112. 112

    Powell M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582 (2002)

  113. 113

    Powell M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds) Large-Scale Nonlinear Optimization, pp. 255–297. Springer, New York (2006)

  114. 114

    Powell M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 28, 649–664 (2008)

  115. 115

    Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)

  116. 116

    Princeton Library. http://www.gamsworld.org/performance/princetonlib/princetonlib.htm

  117. 117

    Regis R.G., Shoemaker C.A.: Constrained global optimization of expensive black box functions using radial basis functions. J. Glob. Optim. 31, 153–171 (2005)

  118. 118

    Regis R.G., Shoemaker C.A.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007)

  119. 119

    Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. (2010, to appear). http://www.optimization-online.org/DB_FILE/2009/02/2226.pdf

  120. 120

    Rios, L.M.: Algorithms for Derivative-Free Optimization. PhD thesis, Department of Industrial and Enterprise Systems Engineering, University of Illinois, Urbana (2009, May)

  121. 121

    Romeo F., Sangiovanni-Vincentelli A.: A theoretical framework for simulated annealing. Algorithmica 6, 302–345 (1991)

  122. 122

    Sacks J., Welch W.J., Mitchell T.J., Wynn H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–423 (1989)

  123. 123

    Sahinidis, N.V., Tawarmalani, M.: BARON 7.5: Global Optimization of Mixed-Integer Nonlinear Programs, User’s Manual (2005)

  124. 124

    Sandia National Laboratories: The Coliny Project. https://software.sandia.gov/trac/acro/wiki/Overview/Projects

  125. 125

    Scheinberg, K.: Manual for Fortran Software Package DFO v2.0 (2003)

  126. 126

    Schonlau, M.: Computer Experiments and Global Optimization. PhD thesis, Department of Statistics, University of Waterloo, Waterloo (1997)

  127. 127

    Serafini, D.B.: A Framework for Managing Models in Nonlinear Optimization of Computationally Expensive Functions. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (1998, Nov)

  128. 128

    Shah, S.B., Sahinidis, N.V.: SAS-Pro: Simultaneous residue assignment and structure superposition for protein structure alignment. PLoS ONE 7(5), e37493 (2012)

  129. 129

    Shubert B.O.: A sequential method seeking the global maximum of a function. SIAM J. Numer. Anal. 9, 379–388 (1972)

  130. 130

    Smith R.L.: Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions. Oper. Res. 32, 1296–1308 (1984)

  131. 131

    Søndergaard, J.: Optimization Using Surrogate Models—by the Space Mapping Technique. PhD thesis, Technical University of Denmark, Department of Mathematical Modelling, Lingby (2003)

  132. 132

    Spendley W., Hext G.R., Himsworth F.R.: Sequential application for simplex designs in optimisation and evolutionary operation. Technometrics 4, 441–461 (1962)

  133. 133

    Tawarmalani M., Sahinidis N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103, 225–249 (2005)

  134. 134

    Torczon V.J.: On the convergence of multidirectional search algorithms. SIAM J. Optim. 1, 123–145 (1991)

  135. 135

    Torczon V.J.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)

  136. 136

    Tseng P.: Fortified-descent simplicial search method: a general approach. SIAM J. Optim. 10, 269–288 (1999)

  137. 137

    Vaz, A.I.F.: PSwarm Home Page. http://www.norg.uminho.pt/aivaz/pswarm/

  138. 138

    Vaz A.I.F., Vicente L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39, 197–219 (2007)

  139. 139

    Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  140. 140

    Wild S.M., Regis R.G., Shoemaker C.A.: ORBIT: Optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2008)

  141. 141

    Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969)

  142. 142

    Winslow, T.A., Trew, R.J., Gilmore, P., Kelley, C.T.: Simulated performance optimization of gaas mesfet amplifiers. In: IEEE/Cornell Conference on Advanced Concepts in High Speed Semiconductor Devices and Circuits, Piscataway, pp. 393–402 (1991)

  143. 143

    Zhao Z., Meza J.C., Van Hove M.: Using pattern search methods for surface structure determination of nanomaterials. J. Phys. Condens. Matter 18, 8693–8706 (2006)

  144. 144

    Zheng, Y.: Pairs Trading and Portfolio Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

Download references

Author information

Correspondence to Nikolaos V. Sahinidis.

Electronic Supplementary Material

The Below is the Electronic Supplementary Material.

ESM 1 (PDF 159 kb)

Rights and permissions

This article is published under an open access license. Please check the 'Copyright Information' section for details of this license and what re-use is permitted. If your intended use exceeds what is permitted by the license or if you are unable to locate the licence and re-use information, please contact the Rights and Permissions team.

About this article

Cite this article

Rios, L.M., Sahinidis, N.V. Derivative-free optimization: a review of algorithms and comparison of software implementations. J Glob Optim 56, 1247–1293 (2013). https://doi.org/10.1007/s10898-012-9951-y

Download citation

Keywords

  • Derivative-free algorithms
  • Direct search methods
  • Surrogate models