Journal of the Operational Research Society

, Volume 67, Issue 4, pp 676–689 | Cite as

Good Laboratory Practice for optimization research

  • Graham Kendall
  • Ruibin Bai
  • Jacek Błazewicz
  • Patrick De Causmaecker
  • Michel Gendreau
  • Robert John
  • Jiawei Li
  • Barry McCollum
  • Erwin Pesch
  • Rong Qu
  • Nasser Sabar
  • Greet Vanden Berghe
  • Angelina Yee
General Paper


Good Laboratory Practice has been a part of non-clinical research for over 40 years. Optimization Research, despite having many papers discussing standards being published over the same period of time, has yet to embrace standards that underpin its research. In this paper we argue the need to adopt standards in optimization research. Building on previous papers, many of which have suggested that the optimization research community should adopt certain standards, we suggest a concrete set of recommendations that the community should adopt. We also discuss how the proposals in this paper could be progressed.


optimization operations research reproducibility 


  1. Adenso-Diaz B and Laguna M (2006). Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54 (1): 99–114.CrossRefGoogle Scholar
  2. Ahuja RV and Orlin J (1996). Use of representative operation counts in computational testing of algorithms. INFORMS Journal on Computing 8 (3): 318–330.CrossRefGoogle Scholar
  3. Anon (2013). Announcement: Reducing our irreproducibility. Nature 496 (7446): 398.Google Scholar
  4. Archetti C and Speranza MG (2008). The split delivery vehicle routing problem: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challanges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 103–122.Google Scholar
  5. Baker M (2012). Independent labs to verify high-profile papers: Nature News, 14 August 2012, Chicago doi:10.1038/nature.2012.11176.Google Scholar
  6. Balinski ML (1978). On the reporting of computational experiments. Mathematical Programming 15 (1): 315.CrossRefGoogle Scholar
  7. Barr RS, Golden BL, Kelly JP, Resende MGC and Stewart Jr WR (1995). Designing and reporting on computational experiments with heuristic methods. Journal of Heuristics 1 (1): 9–32.CrossRefGoogle Scholar
  8. Begley CG and Ellis LM (2012). Drug development: Raise standards for preclinical cancer research. Nature 483 (7391): 531–533.CrossRefGoogle Scholar
  9. Bellmore M and Nemhauser GL (1968). Travelling salesman problem: A survey. Operations Research 16 (3): 538–558.CrossRefGoogle Scholar
  10. Błazewicz J, Domschke W and Pesch E (1996). The job shop scheduling problem: Conventional and new solution techniques. European Journal of Operational Research 93 (1): 1–33.CrossRefGoogle Scholar
  11. Błazewicz J, Ecker KH, Pesch E, Schmidt G and Weglarz J (2007). Handbook on Scheduling. Springer–Verlag: Berlin, Heidelberg.Google Scholar
  12. Błazewicz J, Formanowicz P and Kasprzak M (2005). Selected combinatorial problems of computational biology. European Journal of Operational Research 161 (3): 585–597.CrossRefGoogle Scholar
  13. Boylan JE, Goodwin P, Mohammadipour M and Syntetos AA (2015). Reproducibility in forecasting research. International Journal of Forecasting 31 (1): 79–90.CrossRefGoogle Scholar
  14. Braysy O, Dullaert W and Gendreau M (2004). Evolutionary algorithms for the vehicle routing problem with time windows. Journal of Heuristics 10 (6): 587–611.CrossRefGoogle Scholar
  15. Braysy O and Gendreau M (2005a). Vehicle routing problem with time windows, part I: Route construction and local search algorithms. Transportation Science 39 (1): 104–118.CrossRefGoogle Scholar
  16. Braysy O and Gendreau M (2005b). Vehicle routing problem with time windows, part II: Metaheuristics. Transportation Science 39 (1): 119–139.CrossRefGoogle Scholar
  17. Brown L (1999). Technical and Military Imperatives: A Radar History of World War 2. CRC Press: Bristol, UK.CrossRefGoogle Scholar
  18. Brucker P, Sotskov YN and Werner F (2007). Complexity of shop-scheduling problems with fixed number of jobs: A survey. Mathematical Methods of Operations Research 65 (3): 461–481.CrossRefGoogle Scholar
  19. Burkard RE, Deineko VG, Van Dal R, Van der Veen JAA and Woeginger GJ (1998). Well-solvable special cases of the traveling salesman problem: A survey. SIAM Review 40 (3): 496–546.CrossRefGoogle Scholar
  20. Burke EK et al (2013). Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64 (12): 1695–1724.CrossRefGoogle Scholar
  21. Cattrysse DG and Van Wassenhove LN (1992). A survey of algorithms for the generalized assignment problem. European Journal of Operational Research 60 (3): 260–272.CrossRefGoogle Scholar
  22. Cheng RW, Gen M and Tsujimura Y (1999a). A tutorial survey of jobshop scheduling problems using genetic algorithms, part II: Hybrid genetic search strategies. Computers & Industrial Engineering 36 (2): 343–364.CrossRefGoogle Scholar
  23. Cheng RW, Gen M and Tsujimura Y (1999b). A tutorial survey of jobshop scheduling problems using genetic algorithms: Part II. Hybrid genetic search strategies. Computers & Industrial Engineering 37 (1–2): 51–55.CrossRefGoogle Scholar
  24. Cook WJ (2011). In Pursuit of the Traveling Salesman: Mathematics at the Limits of Computation. Princeton University Press: USA.Google Scholar
  25. Cordeau JF, Gendreau M, Laporte G, Potvin JY and Semet F (2002). A guide to vehicle routing heuristics. Journal of the Operational Research Society 53 (5): 512–522.CrossRefGoogle Scholar
  26. Coy SP, Golden BL, Runger GC and Wasil EA (2001). Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics 7 (1): 77–97.CrossRefGoogle Scholar
  27. Crowder H, Dembo RS and Mulvey JM (1979). On reporting computational experiments with mathematical software. ACM Transactions on Mathematical Software 5 (2): 193–203.CrossRefGoogle Scholar
  28. Crowder HP, Dembo RS and Mulvey JM (1978). Reporting computational experiments in mathematical programming. Mathematical Programming 15 (1): 316–329.CrossRefGoogle Scholar
  29. Derrac J, Garcίa S, Molina D and Herrera F (2011). A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation 1 (1): 3–18.CrossRefGoogle Scholar
  30. Doerner KF and Schmid V (2010). Survey: Matheuristics for rich vehicle routing problems. In: Blesa MJ, Blum C, Raidl G, Roli A and Sampels M (eds). Hybrid Metaheuristics, Lecture Notes in Computer Science. Vol. 6373, 7th International Workshop on Hybrid Metaheuristics, Vienna, Austria, Springer: Berlin Heidelberg, pp 206–221, Oct 01-02, 2010.Google Scholar
  31. Dongarra JJ (1992). Performance of various computers using standard linear equations software. ACM SIGARCH Computer Architecture News 20 (3): 22–44.CrossRefGoogle Scholar
  32. Easley RW, Madden CS and Dunn MG (2000). Conducting marketing science: The role of replication in the research process. Journal of Business Research 48 (1): 83–92.CrossRefGoogle Scholar
  33. Easton K, Nemhauser G and Trick M (2001). The traveling tournament problem description and benchmarks, chap. Principles and Practice of Constraint Programming CP 2001: 7th International Conference, CP 2001 Paphos, Cyprus, 26 November–1 December, 2001 Proceedings. Lecture Notes in Computer Science 2239, Springer, Berlin Heidelberg, pp 580–584.Google Scholar
  34. Ernst AT, Jiang H, Krishnamoorthy M and Sier D (2004). Staff scheduling and rostering: A review of applications, methods and models. European Journal of Operational Research 153 (1): 3–27.CrossRefGoogle Scholar
  35. Evanschitzky H and Armstrong JS (2010). Replications of forecasting research. International Journal of Forecasting 26 (1): 4–8.CrossRefGoogle Scholar
  36. Evanschitzky H, Baumgarth C, Hubbard R and Armstrong JS (2007). Replication research’s disturbing trend. Journal of Business Research 60 (4): 411–415.CrossRefGoogle Scholar
  37. Fildes R (1979). Quantitative forecasting the state of the art: Extrapolative models. Journal of the Operational Research Society 30 (8): 691–710.Google Scholar
  38. Fildes R (1985). Quantitative forecasting the state of the art: Econometric models. Journal of the Operational Research Society 30 (7): 549–580.Google Scholar
  39. Fildes R, Nikolopoulos K, Crone SF and Syntetos AA (2008). Forecasting and operational research: A review. Journal of the Operational Research Society 59 (9): 1150–1172.CrossRefGoogle Scholar
  40. Garcίa S, Fernández A, Luengo J and Herrera F (2010). Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences 180 (10): 2044–2064.CrossRefGoogle Scholar
  41. Gass SI and Assad AA (2006). An Annotated Timeline of Operations Research: An Informal History. Springer: New York.Google Scholar
  42. Gendreau M, Potvin J-Y, Braysy O, Hasle G and Lokketangen A (2008). Metaheuristics for the vehicle routing problem and its extensions: A categorized bibliography. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 143–169.Google Scholar
  43. Ghobbar AA and Friend CH (2003). Evaluation of forecasting methods for intermittent parts demand in the field of aviation: A predictive model. Computers & Operations Research 30 (14): 2097–2114.CrossRefGoogle Scholar
  44. Golden BL, Assad AA, Wasil WA and Baker E (1986). Experimentation in optimization. European Journal of Operational Research 27 (1): 1–16.CrossRefGoogle Scholar
  45. Golden BL and Stewart WR (1985). The Traveling Salesman Problem. chap. Empirical Analysis of Heuristics John Wiley & Sons: Chichester, UK, pp 207–249.Google Scholar
  46. Greenberg HJ (1990). Computational testing: Why, how and how much. ORSA Journal on Computing 2 (1): 94–97.CrossRefGoogle Scholar
  47. Hellekalek P (1998). Good random number generators are (not so) easy to find. Mathematics and Computers in Simulation 46 (5–6): 485–505.CrossRefGoogle Scholar
  48. Hoffman A, Mannos M, Sokolowsky D and Wiegmann N (1953). Computational experience in solving linear programs. Journal of the Society for Industrial and Applied Mathematics 1 (1): 17–33.CrossRefGoogle Scholar
  49. Hooker JN (1994). Needed: An empirical science of algorithms. Operations Research 42 (2): 201–212.CrossRefGoogle Scholar
  50. Hooker JN (1995). Testing heuristics—We have it all wrong. Journal of Heuristics 1 (1): 33–42.CrossRefGoogle Scholar
  51. Hooker JN (2007). Good and bad futures for constraint programming (and operations research). Constraint Programming Letters 1: 21–32.Google Scholar
  52. Hubbard R and Armstrong JS (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11 (3): 233–248.CrossRefGoogle Scholar
  53. Hubbard R and Vetter DE (1996). An empirical comparison of published replication research in accounting, economics, finance, management, and marketing. Journal of Business Research 35 (2): 153–164.CrossRefGoogle Scholar
  54. Ignizio JP (1971). On the establishment of standards for comparing algorithm performance. Interfaces 2 (1): 8–11.CrossRefGoogle Scholar
  55. Ince DC, Hatton L and Graham-Cumming J (2012). The case for open computer programs. Nature 482 (7386): 485–488.CrossRefGoogle Scholar
  56. Ioannidis JA (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association 294 (2): 218–228.CrossRefGoogle Scholar
  57. Jackson RHF, Boggs PT, Nash SG and Powell S (1991). Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Mathematical Programming 49 (1–3): 413–425.Google Scholar
  58. Jackson RHF and Mulvey JM (1978). A critical review of methods for comparing mathematical programming algorithms and software (1953–1977). Journal of Research of the National Bureau of Standards 83 (6): 563–584.CrossRefGoogle Scholar
  59. Jaillet P and Wagner MR (2008). Online vehicle routing problems: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 221–237.Google Scholar
  60. Jourdan L, Basseur M and Talbi E-G (2009). Hybridizing exact methods and metaheuristics: A taxonomy. European Journal of Operational Research 199 (3): 620–629.CrossRefGoogle Scholar
  61. Kendall G, Knust S, Ribeiro CC and Urrutia SS (2010). Scheduling in sports: An annotated bibliography. Computers & Operations Research 37 (1): 1–19.CrossRefGoogle Scholar
  62. Kiran AS and Smith ML (1984). Simulation studies in job shop scheduling—1. A survey. Computers & Industrial Engineering 8 (2): 87–93.CrossRefGoogle Scholar
  63. Kirkpatrick S, Gelatt Jr CD and Vecchi M (1983). Optimization by simulated annealing. Science 220 (4598): 671–680.CrossRefGoogle Scholar
  64. Laporte G (2009). Fifty years of bvehicle routing. Transportation Science 43 (4): 408–416.CrossRefGoogle Scholar
  65. Lee C-Y, Bard J, Pinedo M and Wilhelm WE (1993). Guidelines for reporting computational resuts in IIE transactions. IIE Transactions 25 (6): 121–123.CrossRefGoogle Scholar
  66. Lin EYH (1998). A bibliographical survey on some well-known non-standard knapsack problems. INFOR 36 (4): 274–317.Google Scholar
  67. Loiola EM, de Abreu NMM, Boaventura-Netto PO, Hahn P and Querido T (2007). A survey for the quadratic assignment problem. European Journal of Operational Research 176 (2): 657–690.CrossRefGoogle Scholar
  68. Lukasiak P, Błazewicz J and Milostan M (2010). Some operations research methods for analyzing protein sequences and structures. Annals of Operations Research 175 (1): 9–35.CrossRefGoogle Scholar
  69. Lust T and Teghem J (2010). The multiobjective traveling salesman problem: A survey and a new approach. In: Coello CAC, Dhaenens C and Jourdan L (eds). Studies in Computational Intelligence. Vol. 272, Springer-Verlag: Berlin, pp 119–141.Google Scholar
  70. Lust T and Teghem J (2012). The multiobjective multidimensional knapsack problem: A survey and a new approach. International Transactions in Operational Research 19 (4): 495–520.CrossRefGoogle Scholar
  71. Marshall E (1983). The murky world of toxicity testing. Science 220 (4602): 1130–1132.CrossRefGoogle Scholar
  72. McCollum B et al (2010). Setting the research agenda in automated timetabling: The second international timetabling competition. INFORMS Journal on Computing 22 (1): 120–130.CrossRefGoogle Scholar
  73. McGeoch CC (1996). Toward an experimental method for algorithm simulation. INFORMS Journal on Computing 8 (1): 1–15.CrossRefGoogle Scholar
  74. Miller HE (1976). Personnel scheduling in public systems—A survey. Socio-Economic Planning Sciences 10 (6): 241–249.CrossRefGoogle Scholar
  75. Miller DM and Williams D (2003). Shrinkage estimators of time series seasonal factors and their effect on forecasting accuracy. International Journal of Forecasting 19 (4): 669–684.CrossRefGoogle Scholar
  76. Pentico DW (2007). Assignment problems: A golden anniversary survey. European Journal of Operational Research 176 (2): 774–793.CrossRefGoogle Scholar
  77. Pisinger D (2007). The quadratic knapsack problem—A survey. Discrete Applied Mathematics 155 (5): 623–648.CrossRefGoogle Scholar
  78. Prinz F, Schlange T and Asadullah K (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10 (7391): 10–11.Google Scholar
  79. Rardin RL and Uzsoy R (2001). Experimental evaluation of heuristic optimization algorithms: A tutorial. Journal of Heuristics 7 (3): 261–304.CrossRefGoogle Scholar
  80. Rasmussen RV and Trick MA (2008). Round robin scheduling—A survey. European Journal of Operational Research 188 (3): 617–636.CrossRefGoogle Scholar
  81. Russell JF (2013). If a job is worth doing, it is worth doing twice. Nature 496 (7443): 7.CrossRefGoogle Scholar
  82. Salkin HM and Kluyver CAD (1975). Knapsack problem—Survey. Naval Research Logistics 22 (1): 127–144.CrossRefGoogle Scholar
  83. Sӧrensen K (2015). Metaheuristics—The metaphor exposed. International Transactions in Operational Research 22 (1): 3–18.CrossRefGoogle Scholar
  84. Sӧrensen K and Glover F (2013). Encyclopedia of Operations Research and Management Science. chap. Metaheuristics 3rd edn. Springer: New York.Google Scholar
  85. Taillard ÉD, Waelti P and Zuber J (2008). Few statistical tests for proportions comparison. European Journal of Operational Research 185 (3): 1336–1350.CrossRefGoogle Scholar
  86. Talbi E-G (2009). Metaheuristics: From Design to Implementation. Wiley: Hoboken, NJ.CrossRefGoogle Scholar
  87. Trapero JR, Kourentzes N and Fildes R (2015). Identification of sales forecasting models. Journal of the Operational Research Society 66 (2): 299–307.CrossRefGoogle Scholar
  88. Vaux DL (2012). Research methods: Know when your numbers are significant. Nature 492 (7428): 180–181.Google Scholar
  89. Vidal T, Crainic TG, Gendreau M and Prins C (2013). Heuristics for multiattribute vehicle routing problems: A survey and synthesis. European Journal of Operational Research 231 (1): 1–21.CrossRefGoogle Scholar
  90. Vines TH et al (2013). The availability of research data declines rapidly with article age. Current Biology 24 (1): 94–97.CrossRefGoogle Scholar
  91. Wilbaut C, Hanafi S and Salhi S (2008). A survey of effective heuristics and their application to a variety of knapsack problems. IMA Journal of Management Mathematics 19 (3): 227–244.CrossRefGoogle Scholar
  92. Zanakis SH, Evans JR and Vazacopoulos AA (1989). Heuristic methods and applications: A categorized survey. European Journal of Operational Research 43 (1): 88–110.CrossRefGoogle Scholar

Copyright information

© Operational Research Society Ltd. 2015

Authors and Affiliations

  • Graham Kendall
    • 1
    • 2
  • Ruibin Bai
    • 3
  • Jacek Błazewicz
    • 4
  • Patrick De Causmaecker
    • 5
  • Michel Gendreau
    • 6
  • Robert John
    • 1
  • Jiawei Li
    • 1
  • Barry McCollum
    • 7
  • Erwin Pesch
    • 8
  • Rong Qu
    • 1
  • Nasser Sabar
    • 2
  • Greet Vanden Berghe
    • 9
  • Angelina Yee
    • 2
  1. 1.University of NottinghamNottinghamUK
  2. 2.University of Nottingham Malaysia CampusSemenyihMalaysia
  3. 3.University of Nottingham NingboNingboChina
  4. 4.Poznan University of TechnologyPoznanPoland
  5. 5.KU Leuven, Campus KulakKortrijkBelgium
  6. 6.University of MontrealMontrealCanada
  7. 7.Queen’s University BelfastBelfastUK
  8. 8.Universität SiegenSiegenGermany
  9. 9.KU Leuven, Technology Campus GentGentBelgium

Personalised recommendations