Journal of Heuristics

, Volume 1, Issue 1, pp 9–32 | Cite as

Designing and reporting on computational experiments with heuristic methods

  • Richard S. Barr
  • Bruce L. Golden
  • James P. Kelly
  • Mauricio G. C. Resende
  • William R. StewartJr.


This article discusses the design of computational experiments to test heuristic methods and provides reporting guidelines for such experimentation. The goal is to promote thoughtful, well-planned, and extensive testing of heuristics, full disclosure of experimental conditions, and integrity in and reproducibility of the reported results.

Key words

heuristics algorithms experimental design computational testing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aarts, E., van Laarhoven, P., Lenstra, J., and Ulder, N. (1994). A computational study of local search algorithms for job shop scheduling.ORSA Journal on Computing, 6(2), 118–125.zbMATHGoogle Scholar
  2. Ahuja, R., Magnanti, T., and Orlin, J. (1993).Network Flows: Theory, Algorithms, and Applications, Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  3. Amini, M., and Barr, R. (1990). Network reoptimization algorithms: A statistically designed comparison.ORSA Journal on Computing, 5(4), 395–409.Google Scholar
  4. Arthur, J., and Frendewey, J. (1988). Generating traveling salemen problems with known optimal tours.Journal of the Operational Research Society, 39(2), 153–159.CrossRefzbMATHGoogle Scholar
  5. Arthur, J., and Frendewey, J. (1994). An algorithm for generating minimum cost network flow problems with specific structure and known optimal solutions.Networks, 24(8), 445–454.MathSciNetzbMATHGoogle Scholar
  6. Barr, R., and Hickman, B. (1993). Reporting computational experiments with parallel algorithms: Issues, measures, and experts' opinions.ORSA Journal on Computing, 5(1), 2–18.zbMATHGoogle Scholar
  7. Barr, R., and Hickman, B. (1994). Parallelization strategies for the network simplex algorithm.Operations Research, 42(1), 65–80.zbMATHGoogle Scholar
  8. Barton, R., and Ivey, Jr., J. (1996). Nelder-mead simplex modifications for simulation optimization. Tech. rep., Department of Industrial and Systems Engineering, Pennsylvania State University, University Park, PA. To appear inManagement Science.Google Scholar
  9. Battiti, R., and Tecchiolli, G. (1994). Simulated annealing and tabu search in the long run: A comparison on gap tasks.Computers and Mathematics with Applications, 28(6), 1–8.CrossRefzbMATHGoogle Scholar
  10. Bland, R., Cheriyan, J., Jensen, D., and Ladányi, L. (1993). An empirical study of min cost flow algorithms. In D. Johnson, and C. McGeoch (Eds.),Network Flows and Matching: First DIMACS Implementation Challenge, Vol. 12 ofDIMACS Series in Discrete Mathematics and Theoretical Computer Science (pp. 119–156): Providence, RI: American Mathematical Society.Google Scholar
  11. Box, G., and Draper, N. (1969).Evolutionary Operation, A Statistical Method for Process Improvement. New York: John Wiley.Google Scholar
  12. Bratley, P., Fox, B., and Schrage, L. (1983).A Guide to Simulation. New York: Springer-VerlagzbMATHGoogle Scholar
  13. Cornuejols, G., Sridharan, R., and Thizy, J. (1991). A comparison of heuristics and relaxations for the capacititated plant location problem.European Journal of Operational Research, 50, 280–297.CrossRefzbMATHGoogle Scholar
  14. Crowder, H., Dembo, R., and Mulvey, J. (1980). On reporting computational experiments with mathematical software.ACM Transactions on Mathematical Software, 5, 193–203.CrossRefGoogle Scholar
  15. Dyer, M., and Frieze, A. (1985). A simple heuristic for thep-centre problem.Operations Research Letters, 3(6), 285–288.CrossRefMathSciNetzbMATHGoogle Scholar
  16. Feo, T., and Resende, M. (1995). Greedy randomized adaptive search procedures.Journal of Global Optimization, 6, 109–133.CrossRefMathSciNetzbMATHGoogle Scholar
  17. Fisher, M. (1980). Worst-case analysis of heuristic algorithms.Management Science, 26(1), 1–17.zbMATHMathSciNetGoogle Scholar
  18. Floudas, C., and Pardalos, P. (1990).Collection of Test Problems for Constrained Global Optimization Algorithms, Vol. 455 ofLecture Notes in Computer Science. Springer-Verlag.Google Scholar
  19. Garey, M., and Johnson, D. (1979).Computers and Intractability: A Guide to the Theory of NP-Completeness. San Francisco: Freeman.zbMATHGoogle Scholar
  20. Gendreau, M., Hertz, A., and Laporte, G. (1994). A tabu search heuristic for the vehicle routing problem.Management Science, 40(10), 1276–1290.zbMATHGoogle Scholar
  21. Gilsinn, J., Hoffman, K., Jackson, R., Leyendecker, E., Saunders, P., and Shier, D. (1977). Methodology and analysis for comparing discrete linearl 1 approximation codes.Communications in Statistics, 136, 399–413.Google Scholar
  22. Glover, F. (1989). Tabu search-part I.ORSA Journal on Computing, 1(3), 190–206.zbMATHGoogle Scholar
  23. Glover, F., Karney, D., Klingman, D., and Napier, A. (1974). A computational study on start procedures, basis change criteria, and solution algorithms for transportation problems.Management Science, 20, 793–813.MathSciNetzbMATHGoogle Scholar
  24. Golden, B., Assad, A., Wasil, E., and Baker, E. (1986). Experimentation in optimization.European Journal of Operational Research, 27, 1–16.MathSciNetzbMATHGoogle Scholar
  25. Golden, B., and Stewart, W. (1985). Empirical analysis of heuristics. In E. Lawler, J. Lenstra A. Rinnooy Kan, and D. Shmoys (Eds.),The Travelling Salesman Problem, a Guided Tour of Combinatorial Optimization (pp. 207–249). Chichester (U.K.): Wiley.Google Scholar
  26. Greenberg, H. (1990). Computational testing: Why, how and how much?ORSA Journal on Computing, 2, 7–11.Google Scholar
  27. Held, M., and Karp, R. (1970). The travelling-salesman problem and minimum spanning trees.Operations Research, 18, 1138–1162.MathSciNetzbMATHGoogle Scholar
  28. Held, M., and Karp, R. (1971). The travelling-salesman problem and minimum spanning trees: Part ii.Mathematical Programming, 1, 6–25.CrossRefMathSciNetzbMATHGoogle Scholar
  29. Hochbaum, D., and Shmoys, D. (1985). A best possible heuristic for thek-center problem.Mathematics of Operations Research, 10(2), 180–184.MathSciNetzbMATHGoogle Scholar
  30. Holland, J. (1975),Adaptation in Natural and Artificial Systems. Ann Arbor: University of Michigan Press.Google Scholar
  31. Hooker, J. (1994). Needed: An empirical science of algorithms.Operations Research, 42(2), 201–212.zbMATHGoogle Scholar
  32. Hooker, J. (1995). Testing heuristics: We have it all wrong.Journal of Heuristics, 1(1), 33–42.zbMATHMathSciNetGoogle Scholar
  33. Hopfield, J., and Tank, D. (1985). Neural computation of decisions in optimization problems.Biological Cybernetics, 52, 141.MathSciNetzbMATHGoogle Scholar
  34. Jackson, R., Boggs, P., Nash, S., and Powell, S. (1990). Report of the ad hoc committee to revise the guidelines for reporting computational experiments in mathematical programming.Mathematical Programing, 49, 413–425.CrossRefMathSciNetGoogle Scholar
  35. Jackson, R., and Mulvey, J. (1978). A critical review of comparisons of mathematical programming algorithms and software (1953–1977).J. Research of the National Bureau of Standards, 83, 563–584.MathSciNetzbMATHGoogle Scholar
  36. Johnson, D. (1990). Local optimization and the traveling salesman problem. InProceedings of the 17th Colloquium on Automata, Languages and programming, Lecture Notes in Computer Science 443 (pp. 446–461). Berlin: Springer-Verlag.Google Scholar
  37. Johnson, D., Bentley, J., McGeoch, L., and Rothberg, E. (1995). Near-optimal solutions to very lage traveling salesman problems. Tech. rep., monograph, in preparation.Google Scholar
  38. Johnson, D., and Papadimitrious, C. (1985). Performance guarantees for heuristics. In E. Lawler, J. Lenstra, A. Rinnooy Kan, and D. Shmoys (Eds.),The Travelling Salesman Problem, A Guided Tour of Combinatorial Optimization (pp. 145–180). Chichester (U.K.): Wiley.Google Scholar
  39. Kelly, J., Golden, B., and Assad, A. (1992). Cell suppression: Disclosure protection for sensitive tabular data.Networks, 22(4), 397–417.zbMATHGoogle Scholar
  40. Kelly, J., Golden, B., and Assad, A. (1993). Large-scale controlled rounding using tabu search and strategic oscillation.Annals of Operations Research, 41, 69–84.CrossRefzbMATHGoogle Scholar
  41. Klingman, D., and Mote, J. (1987). Computational analysis of large-scale pure networks. Presented at the Joint National Meeting of ORSA/TIMS, New Orleans.Google Scholar
  42. Klingman, D., Napier, A., and Stutz, J. (1974). Netgen: A program for generating large scale capacitated assignment, transportation, and minimum cost flow network problems.Management Science, 20, 814–821.MathSciNetzbMATHGoogle Scholar
  43. Knox, J. (1994). Tabu search performance on the symmetric travelling salesman problem.Computers & Operations Research, 21(8), 867–876.zbMATHCrossRefGoogle Scholar
  44. Lawler, E., Lenstra, J., Kan, A. Rinnooy, and Shmoys, D. (1985).The Travelling Salesman Problem, A Guided Tour of Combinatorial Optimization. Chiehester (U.K.): Wiley.Google Scholar
  45. Lin, B., and Rardin, R. (1977). Development of a parametric generating procedure for integer programming test problems.Journal of the ACM, 24, 465–472.CrossRefMathSciNetzbMATHGoogle Scholar
  46. Lin, B., and Rardin, R. (1979). Controlled experimental design for statistical comparison of integer programming algorithms.Management Science, 25(12), 33–43.MathSciNetCrossRefGoogle Scholar
  47. Lin, S., and Kernighan, B. (1973). An effective heuristic algorithm for the traveling-salesman problem.Operations Research, 21(2), 498–516.MathSciNetzbMATHGoogle Scholar
  48. Martello, S., and Toth, P. (1990).Knapsack Problems. Chichester (U.K.): Wiley.zbMATHGoogle Scholar
  49. Mason, R., Gunst, R., and Hess, J. (1989).Statistical Design and Analysis of Experiments. New York: Wiley.Google Scholar
  50. McGeoch, C. (1986).Experimental Analysis of Algorithms. Ph.D. thesis, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA.Google Scholar
  51. McGeoch, C. (1995). Toward an experimental method for algorithm simulation.INFORMS Journal on Computing, to appear.Google Scholar
  52. McGeoch, C. (1992). Analyzing algorithms by simulation: Variance reduction techniques and simulation speedups.ACM Computing Surveys, 24(5), 195–212.CrossRefGoogle Scholar
  53. Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., and Teller, E. (1953). Equation of state calculation by fast computing machines.Journal of Chemical Physics, 21, 1087–1091.CrossRefGoogle Scholar
  54. Montgomery, D. (1984).Design and Analysis of Experiments. New York: Wiley.Google Scholar
  55. Mulvey, J. (1982).Evaluating Mathematical Programming Techniques. Berlin: Springer-Verlag.zbMATHGoogle Scholar
  56. Nance, R., Moose, Jr., R., and Foutz, R. (1987). A statistical technique for comparing heuristics: An example from capacity assignment strategies in computer network design.Communications of the ACM, 30(5), 430–442.CrossRefGoogle Scholar
  57. Nelder, J., and Mead, R. (1965). A simplex method for function minimization.Computer Journal, 7, 308–313.zbMATHGoogle Scholar
  58. Nygard, K., Juell, P., and Kadaba, N. (1990). Neural networks for selecting vehicle routing heuristics.ORSA Journal on Computing, 2(4), 353–364.zbMATHGoogle Scholar
  59. O'Neill, R. (1982). A comparison of real-world linear programs and their randomly generated analogs. In J. Mulvey (Ed.),Evaluating Mathematical Programming Techniques (pp. 44–59). Berlin: Springer-Verlag.Google Scholar
  60. Rardin, R., and Lin, B. (1982). Test problems for computational experiments—issues and techniques. In J. Mulvey (Ed.),Evaluating Mathematical Programming Techniques (pp. 8–15). Berlin: Springer-Verlag.Google Scholar
  61. Reeves, C. (1993a). Evaluation of heuristic performance. In C. Reeves, (Ed.),Modern Heuristic Techniques for Combinatorial Problems. New York: Wiley.Google Scholar
  62. Reeves, C. (1993b).Evaluation of Heuristic Performance. New York: Wiley.Google Scholar
  63. Reinelt, G. (1991). TSPLIB—a travelling salesman problem library.ORSA Journal on Computing, 3(4), 376–384.zbMATHGoogle Scholar
  64. Resende, M., and Ribeiro, C. (1995). A GRASP for graph planarization. Tech. rep., AT&T Bell Laboratories, Murray Hill, NJ.Google Scholar
  65. Rothfarb, B., Frank, H., Rosebaum, D., Steiglitz, K., and Kleitman, D. (1970). Optimal design of offshore natural gas pipeline systems.Operations Research, 18, 992–1020.Google Scholar
  66. Stewart, W. (1987). An accelerated branch exchange heuristic for the traveling salesman problem.Networks, 17, 423–437.zbMATHMathSciNetGoogle Scholar
  67. Stewart, W., Kelly, J., and Laguna, M. (1995). Solving vehicle routing problems using generalized assignments and tabu search. Tech. rep., Graduate School of Business, College of William and Mary, Williamsburg, VA.Google Scholar
  68. Taguchi, G., and Wu, Y.-I. (1979).Introduction to Off-Live Quality Control. Central Japan Quality Control Association, Meieki Nakamura-ku Magaya, Japan.Google Scholar
  69. Tufte, E. (1983).The Visual Display of Quantitative Information. Cheshire, CT: Graphics Press.Google Scholar
  70. Zanakis, S., Evans, J., and Vazacopoulos, A. (1989). Heuristic methods and applications: A categorized survey.European Journal of Operational Research, 43, 88–110.CrossRefMathSciNetzbMATHGoogle Scholar

Copyright information

© Kluwer Academic Publishers 1995

Authors and Affiliations

  • Richard S. Barr
    • 1
  • Bruce L. Golden
    • 2
  • James P. Kelly
    • 3
  • Mauricio G. C. Resende
    • 4
  • William R. StewartJr.
    • 5
  1. 1.Department of Computer Science and EngineeringSouthern Methodist UniversityDallas
  2. 2.College of Business and ManagementUniversity of MarylandCollege Park
  3. 3.College of Business and AdministrationUniversity of Colorado at BoulderBoulder
  4. 4.Mathematical Sciences Research CenterAT&T Bell LaboratoriesMurray Hill
  5. 5.School of Business AdministrationThe College of William and MaryWilliamsburg

Personalised recommendations