Computational Comparison of Metaheuristics

  • John Silberholz
  • Bruce GoldenEmail author
  • Swati Gupta
  • Xingyin Wang
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 272)


Metaheuristics are truly diverse in nature—under the overarching theme of performing operations to escape local optima, algorithms as different as ant colony optimization, tabu search, harmony search, and genetic algorithms have emerged. Due to the unique functionality of each type of metaheuristic, the computational comparison of metaheuristics is in many ways more difficult than other algorithmic comparisons. In this chapter, we discuss techniques for the meaningful computational comparison of metaheuristics. We discuss how to create and classify instances in a new testbed and how to make sure other researchers have access to these test instances for future metaheuristic comparisons. In addition, we discuss the disadvantages of large parameter sets and how to measure complicated parameter interactions in a metaheuristic’s parameter space. Finally, we explain how to compare metaheuristics in terms of both solution quality and runtime and how to compare parallel metaheuristics.


  1. 1.
    R. Ahuja, J. Orlin, Use of representative operation counts in computational testing of algorithms.INFORMS J. Comput. 8(3), 318–330 (1996)Google Scholar
  2. 2.
    R.K. Ahuja, M. Kodialam, A.K. Mishra, J.B. Orlin, Computational investigations of maximum flow algorithms. Eur. J. Oper. Res. 97(3), 509–542 (1997)CrossRefGoogle Scholar
  3. 3.
    T. Akhtar, C.A. Shoemaker, Multi objective optimization of computationally expensive multi-modal functions with RBF surrogates and multi-rule selection. J. Glob. Optim. 64(1), 17–32 (2016)CrossRefGoogle Scholar
  4. 4.
    E. Alba, G. Luque, S. Nesmachnow, Parallel metaheuristics: recent advances and new trends. Int. Trans. Oper. Res. 20(1), 1–48 (2013)CrossRefGoogle Scholar
  5. 5.
    D. Bailey, Twelve ways to fool the masses when giving performance results on parallel computers. Supercomput. Rev. 4(8), 54–55 (1991)Google Scholar
  6. 6.
    M. Bull, L. Smith, L. Pottage, R. Freeman, Benchmarking Java against C and Fortran for scientific applications, in ACM 2001 Java Grande/ISCOPE Conference (2001), pp. 97–105Google Scholar
  7. 7.
    I.-M. Chao, Algorithms and solutions to multi-level vehicle routing problems. PhD thesis, University of Maryland, College Park, MD, 1993Google Scholar
  8. 8.
    S. Chen, B. Golden, E. Wasil, The split delivery vehicle routing problem: applications, algorithms, test problems, and computational results. Networks 49(4), 318–329 (2007)CrossRefGoogle Scholar
  9. 9.
    P. Chen, B. Golden, X. Wang, E. Wasil, A novel approach to solve the split delivery vehicle routing problem. Int. Trans. Oper. Res. 24(1–2), 27–41 (2017)CrossRefGoogle Scholar
  10. 10.
    F.R.K. Chung, Spectral Graph Theory, vol. 92 (American Mathematical Society, Providence, 1997)Google Scholar
  11. 11.
    C. Coello, Evolutionary multi-objective optimization: a historical view of the field. IEEE Comput. Intell. Mag. 1(1), 28–36 (2006)CrossRefGoogle Scholar
  12. 12.
    M. Coffin, M.J. Saltzman, Statistical analysis of computational tests of algorithms and heuristics. INFORMS J. Comput. 12(1), 24–44 (2000)CrossRefGoogle Scholar
  13. 13.
    D.W. Corne, A.P. Reynolds, Optimisation and generalisation: footprints in instance space, in International Conference on Parallel Problem Solving from Nature (Springer, Berlin, 2010), pp. 22–31Google Scholar
  14. 14.
    S. Coy, B. Golden, G. Runger, E. Wasil, Using experimental design to find effective parameter settings for heuristics. J. Heuristics 7(1), 77–97 (2001)CrossRefGoogle Scholar
  15. 15.
    J. Culberson, A. Beacham, D. Papp, Hiding our colors, in CP95 Workshop on Studying and Solving Really Hard Problems (1995), pp. 31–42Google Scholar
  16. 16.
    K. Deb, S. Agarwal, Understanding interactions among genetic algorithm parameters, in Foundations of Genetic Algorithms (Morgan Kauffman, San Mateo, 1998), pp. 265–286Google Scholar
  17. 17.
    J. Dongarra, Performance of various computers using standard linear equations software. Technical Report CS-89-85, University of Tennessee, 2014Google Scholar
  18. 18.
    M.M. Drugan, Generating QAP instances with known optimum solution and additively decomposable cost function. J. Comb. Optim. 30(4), 1138–1172 (2015)CrossRefGoogle Scholar
  19. 19.
    I. Dunning, S. Gupta, J. Silberholz, What works best when? A systematic evaluation of heuristics for Max-Cut and QUBO. INFORMS J. Comput. (2018, to appear)Google Scholar
  20. 20.
    G. Erdoğan, G. Laporte, A.M. Rodríguez Chía, Exact and heuristic algorithms for the Hamiltonian p-median problem. Eur. J. Oper. Res. 253(1), 280–289 (2016)CrossRefGoogle Scholar
  21. 21.
    M. Fischetti, J.J. Salazar González, P. Toth, A branch-and-cut algorithm for the symmetric generalized traveling salesman problem. Oper. Res. 45(3), 378–394 (1997)CrossRefGoogle Scholar
  22. 22.
    I. Gamvros, B. Golden, S. Raghavan, The multilevel capacitated minimum spanning tree problem. INFORMS J. Comput. 18(3), 348–365 (2006)CrossRefGoogle Scholar
  23. 23.
    M. Gendreau, G. Laporte, F. Semet, A tabu search heuristic for the undirected selective travelling salesman problem. Eur. J. Oper. Res. 106(2–3), 539–545 (1998)CrossRefGoogle Scholar
  24. 24.
    C. Groër, B. Golden, E. Wasil, A parallel algorithm for the vehicle routing problem. INFORMS J. Comput. 23(2), 315–330 (2011)CrossRefGoogle Scholar
  25. 25.
    A.A. Hagberg, D.A. Schult, P.J. Swart, Exploring network structure, dynamics, and function using NetworkX, in Proceedings of the 7th Python in Science Conference, Pasadena, 2008, pp. 11–15Google Scholar
  26. 26.
    M. Held, R.M. Karp, The traveling-salesman problem and minimum spanning trees. Oper. Res. 18(6), 1138–1162 (1970)CrossRefGoogle Scholar
  27. 27.
    M. Held, R.M. Karp, The traveling-salesman problem and minimum spanning trees: Part II. Math. Program. 1(1), 6–25 (1971)CrossRefGoogle Scholar
  28. 28.
    R. Jans, Z. Degraeve, Meta-heuristics for dynamic lot sizing: a review and comparison of solution approaches. Eur. J. Oper. Res. 177(3), 1855–1875 (2007)CrossRefGoogle Scholar
  29. 29.
    A. Jaszkiewicz, Do multi-objective metaheuristics deliver on their promises? A computational experiment on the set-covering problem. IEEE Trans. Evol. Comput. 7(2), 133–143 (2003)CrossRefGoogle Scholar
  30. 30.
    D.S. Johnson, A theoretician’s guide to experimental analysis of algorithms, in Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, Providence, 2002, ed. by M.H. Goldwasser, D.S. Johnson, C.C. McGeoch, pp. 215–250Google Scholar
  31. 31.
    D.S. Johnson, C.R. Aragon, L.A. McGeoch, C. Schevon, Optimization by simulated annealing: an experimental evaluation: part II, graph coloring and number partitioning. Oper. Res. 37(6), 865–892 (1989)CrossRefGoogle Scholar
  32. 32.
    D.F. Jones, S.K. Mirrazavi, M. Tamiz, Multi-objective meta-heuristics: an overview of the current state-of-the-art. Eur. J. Oper. Res. 137(1), 1–9 (2002)CrossRefGoogle Scholar
  33. 33.
    R.M. Karp, Reducibility among combinatorial problems, in Complexity of Computer Computations (Springer, Berlin, 1972), pp. 85–103CrossRefGoogle Scholar
  34. 34.
    P. King, How the NFL schedule was made, 2017. Retrieved from
  35. 35.
    F. Krzkakała, A. Pagnani, M. Weigt, Threshold values, stability analysis, and high-q asymptotics for the coloring problem on random graphs. Phys. Rev. E 70(4), 046705 (2004)Google Scholar
  36. 36.
    M. Kulich, J.J. Miranda-Bront, L. Pr̆euc̆il, A meta-heuristic based goal-selection strategy for mobile robot search in an unknown environment. Comput. Oper. Res. 84, 178–187 (2017)Google Scholar
  37. 37.
    F. Li, B. Golden, E. Wasil, Very large-scale vehicle routing: new test problems, algorithms, and results. Comput. Oper. Res. 32(5), 1165–1179 (2005)CrossRefGoogle Scholar
  38. 38.
    F. Li, B. Golden, E. Wasil, The open vehicle routing problem: algorithms, large-scale test problems, and computational results. Comput. Oper. Res. 34(10), 2918–2930 (2007)CrossRefGoogle Scholar
  39. 39.
    F. Li, B. Golden, E. Wasil, A record-to-record travel algorithm for solving the heterogeneous fleet vehicle routing problem. Comput. Oper. Res. 34(9), 2734–2742 (2007)CrossRefGoogle Scholar
  40. 40.
    J. Liu, D. Wang, K. He, Y. Xue, Combining Wang-Landau sampling algorithm and heuristics for solving the unequal-area dynamic facility layout problem. Eur. J. Oper. Res. 262(3), 1052–1063 (2017)CrossRefGoogle Scholar
  41. 41.
    M. López-Ibánez, J. Dubois-Lacoste, L. Pérez Cáceres, M. Birattari, T. Stützle, The irace package: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)CrossRefGoogle Scholar
  42. 42.
    G. Luque, E. Alba, Parallel Genetic Algorithms: Theory and Real World Applications (Springer, Berlin, 2011)CrossRefGoogle Scholar
  43. 43.
    D. Montgomery, Design and Analysis of Experiments (Wiley, New York, 2006)Google Scholar
  44. 44.
    J. Nummela, B. Julstrom, An effective genetic algorithm for the minimum-label spanning tree problem, in Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation (ACM, New York, 2006), pp. 553–557Google Scholar
  45. 45.
    Y.W. Park, Y. Jiang, D. Klabjan, L. Williams, Algorithms for generalized clusterwise linear regression. INFORMS J. Comput. 29(2), 301–317 (2017)CrossRefGoogle Scholar
  46. 46.
    A. Paul, D. Freund, A. Ferber, D.B. Shmoys, D.P. Williamson, Prize-collecting TSP with a budget constraint, in 25th Annual European Symposium on Algorithms (ESA 2017), ed. by K. Pruhs, C. Sohler. Leibniz International Proceedings in Informatics (LIPIcs), vol. 87 (Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, 2017), pp. 62:1–62:14Google Scholar
  47. 47.
    N. Pholdee, S. Bureerat, Comparative performance of meta-heuristic algorithms for mass minimisation of trusses with dynamic constraints. Adv. Eng. Softw. 75(1), 1–13 (2014)CrossRefGoogle Scholar
  48. 48.
    R. Plackett, J. Burman, The design of optimum multifactorial experiments. Biometrika 33, 305–325 (1946)CrossRefGoogle Scholar
  49. 49.
    R.L. Rardin, R. Uzsoy, Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(3), 261–304 (2001)CrossRefGoogle Scholar
  50. 50.
    G. Reinelt, TSPLIB—a traveling salesman problem library. ORSA J. Comput. 3(4), 376–384 (1991)CrossRefGoogle Scholar
  51. 51.
    G. Rinaldi, RUDY: a generator for random graphs (1996). Accessed 30 Sept 2014
  52. 52.
    K.L. Sadowski, D. Thierens, P.A.N. Bosman, GAMBIT: a parameterless model-based evolutionary algorithm for mixed-integer problems. Evol. Comput. (2018, to appear)Google Scholar
  53. 53.
    H. Sawai, S. Kizu, Parameter-free genetic algorithm inspired by “disparity theory of evolution”, in Parallel Problem Solving from Nature – PPSN V, ed. by A. Eiben, T. Bäck, M. Schoenauer, H.-P. Schwefel. LNCS, vol. 1498 (Springer, Berlin, 1998), pp. 702–711Google Scholar
  54. 54.
    J. Silberholz, B. Golden, The effective application of a new approach to the generalized orienteering problem. J. Heuristics 16(3), 393–415 (2010)CrossRefGoogle Scholar
  55. 55.
    K. Smith-Miles, S. Bowly, Generating new test instances by evolving in instance space. Comput. Oper. Res. 63, 102–113 (2015)CrossRefGoogle Scholar
  56. 56.
    K. Smith-Miles, L. Lopes, Measuring instance difficulty for combinatorial optimization problems. Comput. Oper. Res. 39(5), 875–889 (2012)CrossRefGoogle Scholar
  57. 57.
    K. Smith-Miles, J. van Hemert, Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann. Math. Artif. Intell. 61(2), 87–104 (2011)CrossRefGoogle Scholar
  58. 58.
    K. Smith-Miles, D. Baatar, B. Wreford, R. Lewis, Towards objective measures of algorithm performance across instance space. Comput. Oper. Res. 45, 12–24 (2014)CrossRefGoogle Scholar
  59. 59.
    S. Subramanian, M.C. Jeffrey, M. Abeydeera, H.R. Lee, V.A. Ying, J. Emer, D. Sanchez, Fractal: an execution model for fine-grain nested speculative parallelism, in Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17 (ACM, New York, 2017), pp. 587–599Google Scholar
  60. 60.
    D. Sudholt, Parallel evolutionary algorithms, in Springer Handbook of Computational Intelligence, ed. by J. Kacprzyk, W. Pedrycz (Springer, Berlin, 2015), pp. 929–959CrossRefGoogle Scholar
  61. 61.
    E.-G. Talbi, M. Basseur, A.J. Nebro, E. Alba, Multi-objective optimization using metaheuristics: non-standard algorithms. Int. Trans. Oper. Res. 19(1–2), 283–305 (2012)CrossRefGoogle Scholar
  62. 62.
    D. Taş, M. Gendreau, O. Jabali, G. Laporte, The traveling salesman problem with time-dependent service times. Eur. J. Oper. Res. 248(2), 372–383 (2016)CrossRefGoogle Scholar
  63. 63.
    L. Tierney, A.J. Rossini, N. Li, H. Sevcikova, Simple network of workstations (Package ‘snow’), 2016.
  64. 64.
    V.V. Vazirani, Approximation Algorithms (Springer, Berlin, 2013)Google Scholar
  65. 65.
    T. Vidal, T.G. Crainic, M. Gendreau, C. Prins, Heuristics for multi-attribute vehicle routing problems: a survey and synthesis. Eur. J. Oper. Res. 231(1), 1–21 (2013)CrossRefGoogle Scholar
  66. 66.
    X. Wang, B. Golden, E. Wasil, The min-max multi-depot vehicle routing problem: heuristics and computational results. J. Oper. Res. Soc. 66(9), 1430–1441 (2015)CrossRefGoogle Scholar
  67. 67.
    D.H. Wolpert, W.G. Macready, No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)CrossRefGoogle Scholar
  68. 68.
    Y. Xiong, B. Golden, E. Wasil, A one-parameter genetic algorithm for the minimum labeling spanning tree problem. IEEE Trans. Evol. Comput. 9(1), 55–60 (2005)CrossRefGoogle Scholar
  69. 69.
    J. Xu, J. Kelly, A network flow-based tabu search heuristic for the vehicle routing problem. Transp. Sci. 30(4), 379–393 (1996)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  • John Silberholz
    • 1
  • Bruce Golden
    • 2
    Email author
  • Swati Gupta
    • 3
  • Xingyin Wang
    • 4
  1. 1.Ross School of BusinessUniversity of MichiganAnn ArborUSA
  2. 2.R. H. Smith School of BusinessUniversity of MarylandCollege ParkUSA
  3. 3.Simons Institute for the Theory of ComputingUC BerkeleyUSA
  4. 4.Engineering Systems and DesignSingapore University of Technology and DesignSingaporeSingapore

Personalised recommendations