Computational Comparison of Metaheuristics

Part of the International Series in Operations Research & Management Science book series (ISOR, volume 272)


Metaheuristics are truly diverse in nature—under the overarching theme of performing operations to escape local optima, algorithms as different as ant colony optimization, tabu search, harmony search, and genetic algorithms have emerged. Due to the unique functionality of each type of metaheuristic, the computational comparison of metaheuristics is in many ways more difficult than other algorithmic comparisons. In this chapter, we discuss techniques for the meaningful computational comparison of metaheuristics. We discuss how to create and classify instances in a new testbed and how to make sure other researchers have access to these test instances for future metaheuristic comparisons. In addition, we discuss the disadvantages of large parameter sets and how to measure complicated parameter interactions in a metaheuristic’s parameter space. Finally, we explain how to compare metaheuristics in terms of both solution quality and runtime and how to compare parallel metaheuristics.


Parallel Metaheuristics Comparable Solution Quality Problem Instances Best-known Solution Basic Combinatorial Operations 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    R. Ahuja, J. Orlin, Use of representative operation counts in computational testing of algorithms.INFORMS J. Comput. 8(3), 318–330 (1996)Google Scholar
  2. 2.
    R.K. Ahuja, M. Kodialam, A.K. Mishra, J.B. Orlin, Computational investigations of maximum flow algorithms. Eur. J. Oper. Res. 97(3), 509–542 (1997)Google Scholar
  3. 3.
    T. Akhtar, C.A. Shoemaker, Multi objective optimization of computationally expensive multi-modal functions with RBF surrogates and multi-rule selection. J. Glob. Optim. 64(1), 17–32 (2016)Google Scholar
  4. 4.
    E. Alba, G. Luque, S. Nesmachnow, Parallel metaheuristics: recent advances and new trends. Int. Trans. Oper. Res. 20(1), 1–48 (2013)Google Scholar
  5. 5.
    D. Bailey, Twelve ways to fool the masses when giving performance results on parallel computers. Supercomput. Rev. 4(8), 54–55 (1991)Google Scholar
  6. 6.
    M. Bull, L. Smith, L. Pottage, R. Freeman, Benchmarking Java against C and Fortran for scientific applications, in ACM 2001 Java Grande/ISCOPE Conference (2001), pp. 97–105Google Scholar
  7. 7.
    I.-M. Chao, Algorithms and solutions to multi-level vehicle routing problems. PhD thesis, University of Maryland, College Park, MD, 1993Google Scholar
  8. 8.
    S. Chen, B. Golden, E. Wasil, The split delivery vehicle routing problem: applications, algorithms, test problems, and computational results. Networks 49(4), 318–329 (2007)Google Scholar
  9. 9.
    P. Chen, B. Golden, X. Wang, E. Wasil, A novel approach to solve the split delivery vehicle routing problem. Int. Trans. Oper. Res. 24(1–2), 27–41 (2017)Google Scholar
  10. 10.
    F.R.K. Chung, Spectral Graph Theory, vol. 92 (American Mathematical Society, Providence, 1997)Google Scholar
  11. 11.
    C. Coello, Evolutionary multi-objective optimization: a historical view of the field. IEEE Comput. Intell. Mag. 1(1), 28–36 (2006)Google Scholar
  12. 12.
    M. Coffin, M.J. Saltzman, Statistical analysis of computational tests of algorithms and heuristics. INFORMS J. Comput. 12(1), 24–44 (2000)Google Scholar
  13. 13.
    D.W. Corne, A.P. Reynolds, Optimisation and generalisation: footprints in instance space, in International Conference on Parallel Problem Solving from Nature (Springer, Berlin, 2010), pp. 22–31Google Scholar
  14. 14.
    S. Coy, B. Golden, G. Runger, E. Wasil, Using experimental design to find effective parameter settings for heuristics. J. Heuristics 7(1), 77–97 (2001)Google Scholar
  15. 15.
    J. Culberson, A. Beacham, D. Papp, Hiding our colors, in CP95 Workshop on Studying and Solving Really Hard Problems (1995), pp. 31–42Google Scholar
  16. 16.
    K. Deb, S. Agarwal, Understanding interactions among genetic algorithm parameters, in Foundations of Genetic Algorithms (Morgan Kauffman, San Mateo, 1998), pp. 265–286Google Scholar
  17. 17.
    J. Dongarra, Performance of various computers using standard linear equations software. Technical Report CS-89-85, University of Tennessee, 2014Google Scholar
  18. 18.
    M.M. Drugan, Generating QAP instances with known optimum solution and additively decomposable cost function. J. Comb. Optim. 30(4), 1138–1172 (2015)Google Scholar
  19. 19.
    I. Dunning, S. Gupta, J. Silberholz, What works best when? A systematic evaluation of heuristics for Max-Cut and QUBO. INFORMS J. Comput. (2018, to appear)Google Scholar
  20. 20.
    G. Erdoğan, G. Laporte, A.M. Rodríguez Chía, Exact and heuristic algorithms for the Hamiltonian p-median problem. Eur. J. Oper. Res. 253(1), 280–289 (2016)Google Scholar
  21. 21.
    M. Fischetti, J.J. Salazar González, P. Toth, A branch-and-cut algorithm for the symmetric generalized traveling salesman problem. Oper. Res. 45(3), 378–394 (1997)Google Scholar
  22. 22.
    I. Gamvros, B. Golden, S. Raghavan, The multilevel capacitated minimum spanning tree problem. INFORMS J. Comput. 18(3), 348–365 (2006)Google Scholar
  23. 23.
    M. Gendreau, G. Laporte, F. Semet, A tabu search heuristic for the undirected selective travelling salesman problem. Eur. J. Oper. Res. 106(2–3), 539–545 (1998)Google Scholar
  24. 24.
    C. Groër, B. Golden, E. Wasil, A parallel algorithm for the vehicle routing problem. INFORMS J. Comput. 23(2), 315–330 (2011)Google Scholar
  25. 25.
    A.A. Hagberg, D.A. Schult, P.J. Swart, Exploring network structure, dynamics, and function using NetworkX, in Proceedings of the 7th Python in Science Conference, Pasadena, 2008, pp. 11–15Google Scholar
  26. 26.
    M. Held, R.M. Karp, The traveling-salesman problem and minimum spanning trees. Oper. Res. 18(6), 1138–1162 (1970)Google Scholar
  27. 27.
    M. Held, R.M. Karp, The traveling-salesman problem and minimum spanning trees: Part II. Math. Program. 1(1), 6–25 (1971)Google Scholar
  28. 28.
    R. Jans, Z. Degraeve, Meta-heuristics for dynamic lot sizing: a review and comparison of solution approaches. Eur. J. Oper. Res. 177(3), 1855–1875 (2007)Google Scholar
  29. 29.
    A. Jaszkiewicz, Do multi-objective metaheuristics deliver on their promises? A computational experiment on the set-covering problem. IEEE Trans. Evol. Comput. 7(2), 133–143 (2003)Google Scholar
  30. 30.
    D.S. Johnson, A theoretician’s guide to experimental analysis of algorithms, in Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, Providence, 2002, ed. by M.H. Goldwasser, D.S. Johnson, C.C. McGeoch, pp. 215–250Google Scholar
  31. 31.
    D.S. Johnson, C.R. Aragon, L.A. McGeoch, C. Schevon, Optimization by simulated annealing: an experimental evaluation: part II, graph coloring and number partitioning. Oper. Res. 37(6), 865–892 (1989)Google Scholar
  32. 32.
    D.F. Jones, S.K. Mirrazavi, M. Tamiz, Multi-objective meta-heuristics: an overview of the current state-of-the-art. Eur. J. Oper. Res. 137(1), 1–9 (2002)Google Scholar
  33. 33.
    R.M. Karp, Reducibility among combinatorial problems, in Complexity of Computer Computations (Springer, Berlin, 1972), pp. 85–103Google Scholar
  34. 34.
    P. King, How the NFL schedule was made, 2017. Retrieved from
  35. 35.
    F. Krzkakała, A. Pagnani, M. Weigt, Threshold values, stability analysis, and high-q asymptotics for the coloring problem on random graphs. Phys. Rev. E 70(4), 046705 (2004)Google Scholar
  36. 36.
    M. Kulich, J.J. Miranda-Bront, L. Pr̆euc̆il, A meta-heuristic based goal-selection strategy for mobile robot search in an unknown environment. Comput. Oper. Res. 84, 178–187 (2017)Google Scholar
  37. 37.
    F. Li, B. Golden, E. Wasil, Very large-scale vehicle routing: new test problems, algorithms, and results. Comput. Oper. Res. 32(5), 1165–1179 (2005)Google Scholar
  38. 38.
    F. Li, B. Golden, E. Wasil, The open vehicle routing problem: algorithms, large-scale test problems, and computational results. Comput. Oper. Res. 34(10), 2918–2930 (2007)Google Scholar
  39. 39.
    F. Li, B. Golden, E. Wasil, A record-to-record travel algorithm for solving the heterogeneous fleet vehicle routing problem. Comput. Oper. Res. 34(9), 2734–2742 (2007)Google Scholar
  40. 40.
    J. Liu, D. Wang, K. He, Y. Xue, Combining Wang-Landau sampling algorithm and heuristics for solving the unequal-area dynamic facility layout problem. Eur. J. Oper. Res. 262(3), 1052–1063 (2017)Google Scholar
  41. 41.
    M. López-Ibánez, J. Dubois-Lacoste, L. Pérez Cáceres, M. Birattari, T. Stützle, The irace package: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)Google Scholar
  42. 42.
    G. Luque, E. Alba, Parallel Genetic Algorithms: Theory and Real World Applications (Springer, Berlin, 2011)Google Scholar
  43. 43.
    D. Montgomery, Design and Analysis of Experiments (Wiley, New York, 2006)Google Scholar
  44. 44.
    J. Nummela, B. Julstrom, An effective genetic algorithm for the minimum-label spanning tree problem, in Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation (ACM, New York, 2006), pp. 553–557Google Scholar
  45. 45.
    Y.W. Park, Y. Jiang, D. Klabjan, L. Williams, Algorithms for generalized clusterwise linear regression. INFORMS J. Comput. 29(2), 301–317 (2017)Google Scholar
  46. 46.
    A. Paul, D. Freund, A. Ferber, D.B. Shmoys, D.P. Williamson, Prize-collecting TSP with a budget constraint, in 25th Annual European Symposium on Algorithms (ESA 2017), ed. by K. Pruhs, C. Sohler. Leibniz International Proceedings in Informatics (LIPIcs), vol. 87 (Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, 2017), pp. 62:1–62:14Google Scholar
  47. 47.
    N. Pholdee, S. Bureerat, Comparative performance of meta-heuristic algorithms for mass minimisation of trusses with dynamic constraints. Adv. Eng. Softw. 75(1), 1–13 (2014)Google Scholar
  48. 48.
    R. Plackett, J. Burman, The design of optimum multifactorial experiments. Biometrika 33, 305–325 (1946)Google Scholar
  49. 49.
    R.L. Rardin, R. Uzsoy, Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(3), 261–304 (2001)Google Scholar
  50. 50.
    G. Reinelt, TSPLIB—a traveling salesman problem library. ORSA J. Comput. 3(4), 376–384 (1991)Google Scholar
  51. 51.
    G. Rinaldi, RUDY: a generator for random graphs (1996). Accessed 30 Sept 2014
  52. 52.
    K.L. Sadowski, D. Thierens, P.A.N. Bosman, GAMBIT: a parameterless model-based evolutionary algorithm for mixed-integer problems. Evol. Comput. (2018, to appear)Google Scholar
  53. 53.
    H. Sawai, S. Kizu, Parameter-free genetic algorithm inspired by “disparity theory of evolution”, in Parallel Problem Solving from Nature – PPSN V, ed. by A. Eiben, T. Bäck, M. Schoenauer, H.-P. Schwefel. LNCS, vol. 1498 (Springer, Berlin, 1998), pp. 702–711Google Scholar
  54. 54.
    J. Silberholz, B. Golden, The effective application of a new approach to the generalized orienteering problem. J. Heuristics 16(3), 393–415 (2010)Google Scholar
  55. 55.
    K. Smith-Miles, S. Bowly, Generating new test instances by evolving in instance space. Comput. Oper. Res. 63, 102–113 (2015)Google Scholar
  56. 56.
    K. Smith-Miles, L. Lopes, Measuring instance difficulty for combinatorial optimization problems. Comput. Oper. Res. 39(5), 875–889 (2012)Google Scholar
  57. 57.
    K. Smith-Miles, J. van Hemert, Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann. Math. Artif. Intell. 61(2), 87–104 (2011)Google Scholar
  58. 58.
    K. Smith-Miles, D. Baatar, B. Wreford, R. Lewis, Towards objective measures of algorithm performance across instance space. Comput. Oper. Res. 45, 12–24 (2014)Google Scholar
  59. 59.
    S. Subramanian, M.C. Jeffrey, M. Abeydeera, H.R. Lee, V.A. Ying, J. Emer, D. Sanchez, Fractal: an execution model for fine-grain nested speculative parallelism, in Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17 (ACM, New York, 2017), pp. 587–599Google Scholar
  60. 60.
    D. Sudholt, Parallel evolutionary algorithms, in Springer Handbook of Computational Intelligence, ed. by J. Kacprzyk, W. Pedrycz (Springer, Berlin, 2015), pp. 929–959Google Scholar
  61. 61.
    E.-G. Talbi, M. Basseur, A.J. Nebro, E. Alba, Multi-objective optimization using metaheuristics: non-standard algorithms. Int. Trans. Oper. Res. 19(1–2), 283–305 (2012)Google Scholar
  62. 62.
    D. Taş, M. Gendreau, O. Jabali, G. Laporte, The traveling salesman problem with time-dependent service times. Eur. J. Oper. Res. 248(2), 372–383 (2016)Google Scholar
  63. 63.
    L. Tierney, A.J. Rossini, N. Li, H. Sevcikova, Simple network of workstations (Package ‘snow’), 2016.
  64. 64.
    V.V. Vazirani, Approximation Algorithms (Springer, Berlin, 2013)Google Scholar
  65. 65.
    T. Vidal, T.G. Crainic, M. Gendreau, C. Prins, Heuristics for multi-attribute vehicle routing problems: a survey and synthesis. Eur. J. Oper. Res. 231(1), 1–21 (2013)Google Scholar
  66. 66.
    X. Wang, B. Golden, E. Wasil, The min-max multi-depot vehicle routing problem: heuristics and computational results. J. Oper. Res. Soc. 66(9), 1430–1441 (2015)Google Scholar
  67. 67.
    D.H. Wolpert, W.G. Macready, No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)Google Scholar
  68. 68.
    Y. Xiong, B. Golden, E. Wasil, A one-parameter genetic algorithm for the minimum labeling spanning tree problem. IEEE Trans. Evol. Comput. 9(1), 55–60 (2005)Google Scholar
  69. 69.
    J. Xu, J. Kelly, A network flow-based tabu search heuristic for the vehicle routing problem. Transp. Sci. 30(4), 379–393 (1996)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Ross School of BusinessUniversity of MichiganAnn ArborUSA
  2. 2.R. H. Smith School of BusinessUniversity of MarylandCollege ParkUSA
  3. 3.Simons Institute for the Theory of ComputingUC BerkeleyUSA
  4. 4.Engineering Systems and DesignSingapore University of Technology and DesignSingaporeSingapore

Personalised recommendations