Skip to main content
SpringerLink
Log in

Best practices for comparing optimization algorithms

  • Published:
Optimization and Engineering Aims and scope Submit manuscript

Abstract

Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Note that this example is artificially constructed to emphasize the results; the recommended starting point for the Beale test problem is (1, 1).

  2. Wall clock time refers to the amount of time the human tester has to wait to get an answer from the computer. Conversely, CPU time is the amount of time the CPU spends on the algorithm, excluding operating system tasks and other processes.

  3. We thank “Mathematics Referee #1” for pointing out that reference.

  4. We thank “Engineering Referee #3” for pointing out that reference.

  5. We thank “Mathematics Referee #1” for pointing out this challenge.

  6. We thank “Engineering Referee #3” and “Mathematics Referee #2” for pointing out this challenge.

  7. We thank “Mathematics Referee #2” for pointing out this challenge.

  8. We thank “Mathematics Referee #1” for pointing out this challenge.

References

  • Addis B, Locatelli M (2007) A new class of test functions for global optimization. J Glob Optim 38(3):479–501

    Article  MATH  MathSciNet  Google Scholar 

  • Ali MM, Khompatraporn C, Zabinsky ZB (2005) A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems. J Glob Optim 31(4):635–672

    Article  MATH  MathSciNet  Google Scholar 

  • Andrei N (2008) An unconstrained optimization test functions collection. Adv Model Optim 10(1):147–161

    MATH  MathSciNet  Google Scholar 

  • Asaadi J (1973) A computational comparison of some non-linear programs. Math Program 4(1):144–154

    Article  MATH  MathSciNet  Google Scholar 

  • Audet C, Orban D (2006) Finding optimal algorithmic parameters using derivative-free optimization. SIAM J Optim 17(3):642–664

    Article  MATH  MathSciNet  Google Scholar 

  • Audet C, Dang CK, Orban D (2010) Algorithmic parameter optimization of the DFO method with the OPAL framework. In: Ken N, Keita T, John C, Reiji S (eds) Software automatic tuning. Springer, New York, pp 255–274

    Google Scholar 

  • Audet C, Dang K-C, Orban D (2014a) Optimization of algorithms with OPAL. Math Program Comput 6(3):233–254

    Article  MATH  MathSciNet  Google Scholar 

  • Audet C, Le Digabel S, Peyrega M (2014b) Linear equalities in blackbox optimization. Technical report, Les Cahiers du GERAD

  • Averick BM, Carter RG, Moré JJ (1991) The MINPACK-2 test problem collection. Technical report, Argonne National Laboratory, Argonne

  • Balint A, Gall D, Kapler G, Retz R (2010) Experiment design and administration for computer clusters for SAT-solvers (EDACC), system description. J Satisf Boolean Model Comput 7:77–82

    Google Scholar 

  • Bard Y (1970) Comparison of gradient methods for the solution of nonlinear parameter estimation problems. SIAM J Numer Anal 7(1):157–186

    Article  MATH  MathSciNet  Google Scholar 

  • Barr RS, Hickman BL (1993) Reporting computational experiments with parallel algorithms: Issues, measures, and experts’ opinions. INFORMS J Comput 5(1):2–18

    Article  MATH  Google Scholar 

  • Barr RS, Golden BL, Kelly JP, Resende MGC, Stewart WR Jr (1995) Designing and reporting on computational experiments with heuristic methods. J Heuristics 1(1):9–32

    Article  MATH  Google Scholar 

  • Barton RR (1987) Testing strategies for simulation optimization. In Proceedings of the 19th conference on winter simulation, WSC’87, New York, NY, USA. ACM, pp 391–401

  • Bartz-Beielstein T, Preuss M (2014) Experimental analysis of optimization algorithms: tuning and beyond. In: Borenstein Y, Moraglio A (eds) Theory and principled methods for the design of metaheuristics. Natural computing series. Springer, Berlin, pp 205–245

    Chapter  Google Scholar 

  • Bartz-Beielstein T, Lasarczyk CWG, Preuss M (2005) Sequential parameter optimization. In: The 2005 IEEE congress on evolutionary computation, vol 1, pp 773–780

  • Baz M, Hunsaker B, Brooks P, Gosavi A (2007) Automated tuning of optimization software parameters. Technical report, University of Pittsburgh, Department of Industrial Engineering

  • Beiranvand V, Hare W, Lucet Y, Hossain S (2015) Multi-haul quasi network flow model for vertical alignment optimization. Technical report, Computer Science, University of British Columbia, Kelowna, BC, Canada

  • Beltrami EJ (1969) A comparison of some recent iterative methods for the numerical solution of nonlinear programs. In: Beckmann M, Künzi HP (eds) Computing methods in optimization problems. Lecture notes in operations research and mathematical economics, vol 14. Springer, Berlin, pp 20–29

    Chapter  Google Scholar 

  • Benson HY, Shanno DF, Vanderbei RJ (2003) A comparative study of large-scale nonlinear optimization algorithms. In: Di Pillo G, Murli A (eds) High performance algorithms and software for nonlinear optimization. Applied optimization, vol 82. Springer, New York, pp 95–127

    Chapter  Google Scholar 

  • Benson HY, Shanno DF, Vanderbei RJ (2004) Interior-point methods for nonconvex nonlinear programming: jamming and numerical testing. Math Progr 99:35–48

    Article  MATH  MathSciNet  Google Scholar 

  • Berthold T (2013) Measuring the impact of primal heuristics. Oper Res Lett 41(6):611–614

    Article  MATH  MathSciNet  Google Scholar 

  • Billups SC, Dirkse SP, Ferris MC (1997) A comparison of large scale mixed complementarity problem solvers. Comput Optim Appl 7(1):3–25

    Article  MATH  MathSciNet  Google Scholar 

  • Birattari M (2009) Tuning metaheuristics: a machine learning perspective. Springer, Berlin (1st ed. 2005. 2nd printing edition)

    Book  MATH  Google Scholar 

  • Bondarenko AS, Bortz DM, Moré JJ (1999) COPS: large-scale nonlinearly constrained optimization problems. Technical report, Mathematics and Computer Science Division, Argonne National Laboratory. Technical report ANL/MCS-TM-237

  • Bongartz I, Conn AR, Gould N, Toint PL (1995) CUTE: constrained and unconstrained testing environment. ACM Trans Math Softw 21(1):123–160

    Article  MATH  Google Scholar 

  • Bongartz I, Conn AR, Gould NIM, Saunders MA, Toint PL (1997) A numerical comparison between the LANCELOT and MINOS packages for large scale constrained optimization. Technical report, SCAN-9711063

  • Box MJ (1966) A comparison of several current optimization methods, and the use of transformations in constrained problems. Comput J 9(1):67–77

    Article  MATH  MathSciNet  Google Scholar 

  • Buckley AG (1992) Algorithm 709: testing algorithm implementations. ACM Trans Math Softw 18(4):375–391

    Article  MATH  Google Scholar 

  • Bussieck MR, Drud AS, Meeraus A, Pruessner A (2003) Quality assurance and global optimization. In: Bliek C, Jermann C, Neumaier A (eds) Global optimization and constraint satisfaction. Lecture notes in computer science, vol 2861. Springer, Berlin, pp 223–238

    Chapter  Google Scholar 

  • Bussieck MR, Dirkse SP, Vigerske S (2014) PAVER 2.0: an open source environment for automated performance analysis of benchmarking data. J Glob Optim 59(2–3):259–275

    Article  MATH  Google Scholar 

  • CPLEX’s automatic tuning tool. Technical report, IBM Corporation, 2014

  • Colville AR (1968) A comparative study of nonlinear programming codes. Technical report 320-2949, IBM Scientific Center, New York

  • Conn AR, Gould N, Toint PL (1996) Numerical experiments with the LANCELOT package (release A) for large-scale nonlinear optimization. Math Program 73(1):73–110

    Article  MATH  MathSciNet  Google Scholar 

  • Crowder H, Dembo RS, Mulvey JM (1979) On reporting computational experiments with mathematical software. ACM Trans Math Softw 5(2):193–203

    Article  Google Scholar 

  • Dannenbring DG (1977) Procedures for estimating optimal solution values for large combinatorial problems. Manag Sci 23(12):1273–1283

    Article  MATH  Google Scholar 

  • Dembo RS (1976) A set of geometric programming test problems and their solutions. Math Program 10(1):192–213

    Article  MATH  MathSciNet  Google Scholar 

  • Dembo RS (1978) Current state of the art of algorithms and computer software for geometric programming. J Optim Theory Appl 26(2):149–183

    Article  MATH  MathSciNet  Google Scholar 

  • Derigs U (1985) Using confidence limits for the global optimum in combinatorial optimization. Oper Res 33(5):1024–1049

    Article  MATH  MathSciNet  Google Scholar 

  • Dixon LCW, Szegö GP (1978) Towards global optimisation 2. North-Holland, Amsterdam

    MATH  Google Scholar 

  • Dolan ED, Moré JJ (2000) Benchmarking optimization software with COPS. Technical report, Argonne National Laboratory research report

  • Dolan ED, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213

    Article  MATH  MathSciNet  Google Scholar 

  • Dolan ED, Moré JJ (2004) Benchmarking optimization software with COPS 3.0. Argonne National Laboratory research report

  • Domes F, Fuchs M, Schichl H, Neumaier A (2014) The optimization test environment. Optim Eng 15(2):443–468

    Article  MATH  MathSciNet  Google Scholar 

  • Eason ED (1982) Evidence of fundamental difficulties in nonlinear optimization code comparisons. In: Mulvey JM (ed) Evaluating mathematical programming techniques. Lecture notes in economics and mathematical systems, vol 199. Springer, Berlin, pp 60–71

    Chapter  Google Scholar 

  • Eason ED, Fenton RG (1974) A comparison of numerical optimization methods for engineering design. J Manuf Sci Eng 96(1):196–200

    Google Scholar 

  • Elam JJ, Klingman D (1982) NETGEN-II: a system for generating structured network-based mathematical programming test problems. In: Mulvey JM (ed) Evaluating mathematical programming techniques. Lecture notes in economics and mathematical systems, vol 199. Springer, Berlin, pp 16–23

    Chapter  Google Scholar 

  • Evtushenko YG (1985) Numerical optimization techniques. Translation series in mathematics and engineering. Optimization Software, Inc., Publications Division, New York (distributed by Springer, New York, Translated from the Russian, Translation edited and with a foreword by J. Stoer)

  • Famularo D, Pugliese P, Sergeyev YD (2002) Test problems for Lipschitz univariate global optimization with multiextremal constraints. In: Dzemyda G, Šaltenis V, Žilinskas A (eds) Stochastic and global optimization. Nonconvex optimization and its applications, vol 59. Kluwer Academic Publishers, Dordrecht, pp 93–109

    Chapter  Google Scholar 

  • Floudas CA, Pardalos PM (1990) A collection of test problems for constrained global optimization algorithms, vol 455. Springer, Berlin

    MATH  Google Scholar 

  • Floudas CA, Pardalos PM, Adjiman CS, Esposito WR, Gëmës ZH, Harding ST, Klepeis JL, Meyer CA, Schweiger CA (1999) Handbook of test problems in local and global optimization. Springer, New York

    Book  MATH  Google Scholar 

  • Fourer R, Gay DM, Kernighan BW (2002) AMPL: a modeling language for mathematical programming, 2nd edn. Duxbury Press, Belmont

    MATH  Google Scholar 

  • Fowler KR, Reese JP, Kees CE, Dennis JE Jr, Kelley CT, Miller CT, Audet C, Booker AJ, Couture G, Darwin RW, Farthing MW, Finkel DE, Gablonsky JM, Gray G, Kolda TG (2008) Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv Water Resour 31(5):743–757

    Article  Google Scholar 

  • Gaviano M, Kvasov DE, Lera D, Sergeyev YD (2003) Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization. ACM Trans Math Softw 29(4):469–480

    Article  MATH  MathSciNet  Google Scholar 

  • Gilbert JC, Jonsson X (2009) LIBOPT an environment for testing solvers on heterogeneous collections of problems the manual, version 2.1. Technical report, INRIA, Le Chesnay

  • Gillard JW, Kvasov DE (2017) Lipschitz optimization methods for fitting a sum of damped sinusoids to a series of observations. Stat Interface 10:59–70

    Article  MathSciNet  Google Scholar 

  • Golden BL, Stewart WR (1985) Empirical analysis of heuristics. In: Lawler EL, Lenstra JK, Rinnooy Kan AHG, Shmoys DB (eds) The traveling salesman problem: a guided tour of combinatorial optimization. Wiley, New York, pp 207–249

    Google Scholar 

  • Gould N, Scott J (2016) A note on performance profiles for benchmarking software. ACM Trans Math Softw 43(2):15:1–15:5

    Article  MATH  MathSciNet  Google Scholar 

  • Gould N, Orban D, Toint PL (2003) CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans Math Softw 29(4):373–394

    Article  MATH  Google Scholar 

  • Gould NM, Orban D, Toint PL (2015) CUTEst: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput Optim Appl 60(3):545–557

    Article  MATH  MathSciNet  Google Scholar 

  • Grishagin VA (1978) Operating characteristics of some global search algorithms. In: Problems of stochastic search, vol 7. Zinatne, Riga, pp 198–206 (in Russian)

    Google Scholar 

  • Grundel D, Jeffcoat D (2009) Combinatorial test problems and problem generators. In: Floudas CA, Pardalos PM (eds) Encyclopedia of optimization. Springer, New York, pp 391–394

    Chapter  Google Scholar 

  • Hansen P, Jaumard B, Lu SH (1992) Global optimization of univariate Lipschitz functions: II. New algorithms and computational comparison. Math Program 55(1–3):273–292

    Article  MATH  MathSciNet  Google Scholar 

  • Hare W, Planiden C (2014) The NC-proximal average for multiple functions. Optim Lett 8(3):849–860

    Article  MATH  MathSciNet  Google Scholar 

  • Hare W, Sagastizábal C (2006) Benchmark of some nonsmooth optimization solvers for computing nonconvex proximal points. Pac J Optim 2(3):545–573

    MATH  MathSciNet  Google Scholar 

  • Hare W, Sagastizábal C (2010) A redistributed proximal bundle method for nonconvex optimization. SIAM J Optim 20(5):2442–2473

    Article  MATH  MathSciNet  Google Scholar 

  • Hare WL, Wang Y (2010) Fairer benchmarking of optimization algorithms via derivative free optimization. Technical report, optimization-online

  • Hare WL, Koch VR, Lucet Y (2011) Models and algorithms to improve earthwork operations in road design using mixed integer linear programming. Eur J Oper Res 215(2):470–480

    Article  MATH  MathSciNet  Google Scholar 

  • Hillstrom KE (1977) A simulation test approach to the evaluation of nonlinear optimization algorithms. ACM Trans Math Softw 3(4):305–315

    Article  Google Scholar 

  • Hiriart-Urruty J-B, Lemaréchal C (1993) Convex analysis and minimization algorithms. II. Grundlehren der Mathematischen Wissenschaften [Fundamental principles of mathematical sciences]. Advanced theory and bundle methods, vol 306. Springer, Berlin

    MATH  Google Scholar 

  • Hock W, Schittkowski K (1981) Test examples for nonlinear programming codes. Lecture notes in economics and mathematical systems. Springer, Berlin

    Book  MATH  Google Scholar 

  • Hock W, Schittkowski K (1983) A comparative performance evaluation of 27 nonlinear programming codes. Computing 30(4):335–358

    Article  MATH  MathSciNet  Google Scholar 

  • Hoffman KL, Jackson RHF (1982) In pursuit of a methodology for testing mathematical programming software. In: Mulvey JM (ed) Evaluating mathematical programming techniques. Lecture notes in economics and mathematical systems, vol 199. Springer, Berlin, pp 177–199

    Chapter  Google Scholar 

  • Hoffman A, Mannos M, Sokolowsky D, Wiegmann N (1953) Computational experience in solving linear programs. J Soc Ind Appl Math 1(1):17–33

    Article  MATH  MathSciNet  Google Scholar 

  • Hough P, Kolda T, Torczon V (2001) Asynchronous parallel pattern search for nonlinear optimization. SIAM J Sci Comput 23(1):134–156

    Article  MATH  MathSciNet  Google Scholar 

  • Houstis EN, Rice JR, Christara CC, Vavalis EA (1988) Performance of scientific software. In: Rice JR (ed) Mathematical aspects of scientific software. The IMA volumes in mathematics and its applications, vol 14. Springer, New York, pp 123–155

    Chapter  Google Scholar 

  • Huang HY, Levy AV (1970) Numerical experiments on quadratically convergent algorithms for function minimization. J Optim Theory Appl 6(3):269–282

    Article  MATH  MathSciNet  Google Scholar 

  • Hutter F, Hoos HH, Leyton-Brown K, Stützle T (2009) ParamILS: an automatic algorithm configuration framework. J Artif Intell Res 36(1):267–306

    MATH  Google Scholar 

  • Hutter F, Hoos HH, Leyton-Brown K (2010) Automated configuration of mixed integer programming solvers. In: Lodi A, Milano M, Toth P (eds) Integration of AI and OR techniques in constraint programming for combinatorial optimization problems. Lecture notes in computer science, vol 6140. Springer, Berlin, pp 186–202

    Chapter  Google Scholar 

  • Jackson RHF, Boggs PT, Nash SG, Powell S (1990) Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Math Program 49(1–3):413–425

    Article  MathSciNet  Google Scholar 

  • Jamil M, Yang XS (2013) A literature survey of benchmark functions for global optimisation problems. Int J Math Model Numer Optim 4(2):150–194

    MATH  Google Scholar 

  • Johnson DS, McGeoch LA, Rothberg EE (1996) Asymptotic experimental analysis for the Held–Karp traveling salesman bound. In: Proceedings of the seventh annual ACM-SIAM symposium on discrete algorithms, SODA’96, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics, pp 341–350

  • Khompatraporn C, Pinter JD, Zabinsky ZB (2005) Comparative assessment of algorithms and software for global optimization. J Glob Optim 31(4):613–633

    Article  MATH  MathSciNet  Google Scholar 

  • Knuth DE (1994) The Stanford GraphBase: a platform for combinatorial computing, vol 37. Addison-Wesley Publishing Company, Boston

    MATH  Google Scholar 

  • Koch T, Achterberg T, Andersen E, Bastert O, Berthold T, Bixby RE, Danna E, Gamrath G, Gleixner AM, Heinz S, Lodi A, Mittelmann H, Ralphs T, Salvagnin D, Steffy DE, Wolter K (2011) MIPLIB 2010. Math Program Comput 3(2):103–163

    Article  MathSciNet  Google Scholar 

  • Kolda TG, Lewis RM, Torczon V (2003) Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev 45(3):385–482

    Article  MATH  MathSciNet  Google Scholar 

  • Kortelainen M, Lesinski T, Moré J, Nazarewicz W, Sarich J, Schunck N, Stoitsov MV, Wild S (2010) Nuclear energy density optimization. Phys Rev C 82(2):024313

    Article  Google Scholar 

  • Kvasov DE, Mukhametzhanov MS (2016) One-dimensional global search: nature-inspired vs. lipschitz methods. AIP Conf Proc 1738(1):400012

    Article  Google Scholar 

  • Kvasov DE, Mukhametzhanov MS (2017) Metaheuristic vs. deterministic global optimization algorithms: the univariate case. Appl Math Comput

  • Kvasov DE, Sergeyev YD (2015) Deterministic approaches for solving practical black-box global optimization problems. Adv Eng Softw 80:58–66

    Article  Google Scholar 

  • LaMarca A, Ladner R (1996) The influence of caches on the performance of heaps. J Exp Algorithmics 1:4

    Article  MATH  MathSciNet  Google Scholar 

  • Lenard ML, Minkoff M (1984) Randomly generated test problems for positive definite quadratic programming. ACM Trans Math Softw 10(1):86–96

    Article  MATH  MathSciNet  Google Scholar 

  • Liu D, Zhang XS (2000) Test problem generator by neural network for algorithms that try solving nonlinear programming problems globally. J Glob Optim 16(3):229–243

    Article  MATH  MathSciNet  Google Scholar 

  • McGeoch CC (1996) Toward an experimental method for algorithm simulation. INFORMS J Comput 8(1):1–15

    Article  MATH  Google Scholar 

  • McGeoch CC (2002) Experimental analysis of algorithms. In: Pardalos PM, Romeijn HE (eds) Handbook of global optimization. Nonconvex optimization and its applications, vol 62. Springer, New York, pp 489–513

    Chapter  Google Scholar 

  • Miele A, Tietze JL, Levy AV (1972) Comparison of several gradient algorithms for mathematical programming problems. Aero-astronautics report no. 94, Rice University, Houston

  • Mittelmann HD (2003) An independent benchmarking of SDP and SOCP solvers. Math Program 95(2):407–430

    Article  MATH  MathSciNet  Google Scholar 

  • Mittelmann HD, Pruessner A (2006) A server for automated performance analysis of benchmarking data. Optim Methods Softw 21(1):105–120

    Article  MATH  Google Scholar 

  • Moré JJ, Wild S (2009) Benchmarking derivative-free optimization algorithms. SIAM J Optim 20(1):172–191

    Article  MATH  MathSciNet  Google Scholar 

  • Moré JJ, Garbow BS, Hillstrom KE (1981) Testing unconstrained optimization software. ACM Trans Math Softw 7(1):17–41

    Article  MATH  MathSciNet  Google Scholar 

  • Mulvey JM (ed) (1982) Evaluating mathematical programming techniques, vol 199. Springer, Berlin

    MATH  Google Scholar 

  • Nannen V, Eiben AE (2006) A method for parameter calibration and relevance estimation in evolutionary algorithms. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06, New York, NY, USA. ACM, pp 183–190

  • Nash S, Nocedal J (1991) A numerical study of the limited memory BFGS method and the truncated-Newton method for large scale optimization. SIAM J Optim 1(3):358–372

    Article  MATH  MathSciNet  Google Scholar 

  • Nell C, Fawcett C, Hoos HH, Leyton-Brown K (2011) HAL: a framework for the automated analysis and design of high-performance algorithms. In: Coello CAC (ed) Learning and intelligent optimization. Lecture notes in computer science, vol 6683. Springer, Berlin, pp 600–615

    Chapter  Google Scholar 

  • Netlib: Netlib linear programming library. http://netlib.org/

  • Neumaier A, Shcherbina O, Huyer W, Vinkó T (2005) A comparison of complete global optimization solvers. Math Program 103(2):335–356

    Article  MATH  MathSciNet  Google Scholar 

  • Ng C-K, Li D (2014) Test problem generator for unconstrained global optimization. Comput Oper Res 51:338–349

    Article  MATH  MathSciNet  Google Scholar 

  • Nocedal J, Wright S (2006) Numerical optimization. Springer series in operations research and financial engineering. Springer, New York

  • Opara K, Arabas J (2011) Benchmarking procedures for continuous optimization algorithms. J Telecommun Inf Technol 4:73–80

    Google Scholar 

  • Parejo JA, Ruiz-Cortés A, Lozano S, Fernandez P (2012) Metaheuristic optimization frameworks: a survey and benchmarking. Soft Comput 16(3):527–561

    Article  Google Scholar 

  • Paulavičius R, Sergeyev YD, Kvasov DE, Žilinskas J (2014) Globally-biased Disimpl algorithm for expensive global optimization. J Glob Optim 59(2–3):545–567

    Article  MATH  MathSciNet  Google Scholar 

  • Pintér JD (2002) Global optimization: software, test problems, and applications. In: Pardalos PM, Romeijn HE (eds) Handbook of global optimization. Nonconvex optimization and its applications, vol 62. Springer, New York, pp 515–569

    Chapter  Google Scholar 

  • Pintér JD (2007) Nonlinear optimization with GAMS /LGO. J Glob Optim 38(1):79–101

    Article  MATH  MathSciNet  Google Scholar 

  • Pintér JD, Kampas FJ (2013) Benchmarking nonlinear optimization software in technical computing environments. Top 21(1):133–162

    Article  MATH  Google Scholar 

  • Ramsin H, Wedin P (1977) A comparison of some algorithms for the nonlinear least squares problem. BIT Numer Math 17(1):72–90

    Article  MATH  MathSciNet  Google Scholar 

  • Rardin RL, Uzsoy R (2001) Experimental evaluation of heuristic optimization algorithms: a tutorial. J Heuristics 7(3):261–304

    Article  MATH  Google Scholar 

  • Regis RG, Shoemaker CA (2007) A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J Comput 19(4):497–509

    Article  MATH  MathSciNet  Google Scholar 

  • Regis RG, Wild SM (2017) Conorbit: constrained optimization by radial basis function interpolation in trust regions. Optim Methods Softw 32(3):1–29

    Article  MathSciNet  Google Scholar 

  • Ridge E (2007) Design of experiments for the tuning of optimisation algorithms. Department of Computer Science, University of York, Heslington

    MATH  Google Scholar 

  • Ridge E, Kudenko D (2010) Tuning an algorithm using design of experiments. In: Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (eds) Experimental methods for the analysis of optimization algorithms. Springer, Berlin, pp 265–286

    Chapter  Google Scholar 

  • Rijckaert MJ, Martens XM (1978) Comparison of generalized geometric programming algorithms. J Optim Theory Appl 26(2):205–242

    Article  MATH  Google Scholar 

  • Rios L, Sahinidis NV (2013) Derivative-free optimization: a review of algorithms and comparison of software implementations. J Glob Optim 56(3):1247–1293

    Article  MATH  MathSciNet  Google Scholar 

  • Romesis M, Xie M, Minkovich K, Cong J (2003) Optimality study project. Technical report, UCLA Computer Science Department. http://cadlab.cs.ucla.edu/~pubbench/

  • Rosenthal RE (2014) GAMS—a user’s guide. Technical report, GAMS Development Corporation

  • Sandgren E, Ragsdell KM (1980a) The utility of nonlinear programming algorithms: a comparative study, part 1. J Mech Des 102(3):540–546

    Article  Google Scholar 

  • Sandgren E, Ragsdell KM (1980b) The utility of nonlinear programming algorithms: a comparative study, part 2. J Mech Des 102(3):547–551

    Article  Google Scholar 

  • Schichl H, Markót MC (2012) Algorithmic differentiation techniques for global optimization in the COCONUT environment. Optim Methods Softw 27(2):359–372

    Article  MATH  MathSciNet  Google Scholar 

  • Schittkowski K (1980) Nonlinear programming codes: information, tests, performance. Lecture notes in economics and mathematical systems. Springer, Berlin

    Book  MATH  Google Scholar 

  • Schittkowski K (2008) An updated set of 306 test problems for nonlinear programming with validated optimal solutions—user’s guide. Technical report, Department of Computer Science, University of Bayreuth

  • Schittkowski K, Stoer J (1978) A factorization method for the solution of constrained linear least squares problems allowing subsequent data changes. Numer Math 31(4):431–463

    Article  MATH  MathSciNet  Google Scholar 

  • Schoen F (1993) A wide class of test functions for global optimization. J Glob Optim 3(2):133–137

    Article  MATH  MathSciNet  Google Scholar 

  • Sergeyev YD, Kvasov DE (2006) Global search based on efficient diagonal partitions and a set of Lipschitz constants. SIAM J Optim 16(3):910–937

    Article  MATH  MathSciNet  Google Scholar 

  • Sergeyev YD, Kvasov DE (2015) A deterministic global optimization using smooth diagonal auxiliary functions. Commun Nonlinear Sci Numer Simul 21(1–3):99–111

    Article  MATH  MathSciNet  Google Scholar 

  • Sergeyev YD, Strongin RG, Lera D (2013) Introduction to global optimization exploiting space-filling curves. Springer briefs in optimization. Springer, New York

    Book  MATH  Google Scholar 

  • Sergeyev YD, Kvasov DE, Mukhametzhanov MS (2016) Operational zones for comparing metaheuristic and deterministic one-dimensional global optimization algorithms. Math Comput Simul 141:96–109

    Article  MathSciNet  Google Scholar 

  • Shcherbina O, Neumaier A, Sam-Haroud D, Vu XH, Nguyen TV (2003) Benchmarking global optimization and constraint satisfaction codes. In: Bliek C, Jermann C, Neumaier A (eds) Global optimization and constraint satisfaction. Lecture notes in computer science, vol 6683. Springer, Berlin, pp 211–222

    Chapter  Google Scholar 

  • Strongin RG, Sergeyev YD (2000) Global optimization with non-convex constraints: sequential and parallel algorithms. Springer, New York

    Book  MATH  Google Scholar 

  • Tabak D (1969) Comparative study of various minimization techniques used in mathematical programming. IEEE Trans Autom Control 14(5):572–572

    Article  MathSciNet  Google Scholar 

  • Tedford NP, Martins JRRA (2010) Benchmarking multidisciplinary design optimization algorithms. Optim Eng 11(1):159–183

    Article  MATH  MathSciNet  Google Scholar 

  • Törn A, Žilinskas A (1989) Global optimization. Lecture notes in computer science, vol 350. Springer, Berlin

    MATH  Google Scholar 

  • Törn A, Ali MM, Viitanen S (1999) Stochastic global optimization: problem classes and solution techniques. J Glob Optim 14(4):437–447

    Article  MATH  MathSciNet  Google Scholar 

  • Tufte ER, Graves-Morris PR (1983) The visual display of quantitative information, vol 2. Graphics Press, Cheshire

    Google Scholar 

  • Tukey JW (1977) Exploratory data analysis. Pearson, Reading

    MATH  Google Scholar 

  • Vanden Berghen F, Bersini H (2005) CONDOR, a new parallel, constrained extension of powell’s UOBYQA algorithm: experimental results and comparison with the DFO algorithm. J Comput Appl Math 181(1):157–175

    Article  MATH  MathSciNet  Google Scholar 

  • Vanderbei RJ, Shanno DF (1999) An interior-point algorithm for nonconvex nonlinear programming. Comput Optim Appl 13(1–3):231–252

    Article  MATH  MathSciNet  Google Scholar 

  • Vaz AIF, Vicente LN (2007) A particle swarm pattern search method for bound constrained global optimization. J Glob Optim 39(2):197–219

    Article  MATH  MathSciNet  Google Scholar 

  • Yeniay O (2005) A comparative study on optimization methods for the constrained nonlinear programming problems. Math Probl Eng 2005(2):165–173

    Article  MATH  MathSciNet  Google Scholar 

  • Zhang Z (2014) Sobolev seminorm of quadratic functions with applications to derivative-free optimization. Math Program 146(1–2):77–96

    Article  MATH  MathSciNet  Google Scholar 

  • Zhigljavsky A, Žilinskas A (2008) Stochastic global optimization. Springer optimization and its applications, vol 9. Springer, New York

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of British Columbia, Kelowna, BC, Canada

    Vahid Beiranvand & Yves Lucet

  2. Department of Mathematics, University of British Columbia, Kelowna, BC, Canada

    Warren Hare

Authors
  1. Vahid Beiranvand
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Warren Hare
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yves Lucet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yves Lucet.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Beiranvand, V., Hare, W. & Lucet, Y. Best practices for comparing optimization algorithms. Optim Eng 18, 815–848 (2017). https://doi.org/10.1007/s11081-017-9366-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11081-017-9366-1

Keywords

Search

Navigation