Skip to main content
SpringerLink
Log in

Good Laboratory Practice for optimization research

  • General Paper
  • Published:
Journal of the Operational Research Society

Abstract

Good Laboratory Practice has been a part of non-clinical research for over 40 years. Optimization Research, despite having many papers discussing standards being published over the same period of time, has yet to embrace standards that underpin its research. In this paper we argue the need to adopt standards in optimization research. Building on previous papers, many of which have suggested that the optimization research community should adopt certain standards, we suggest a concrete set of recommendations that the community should adopt. We also discuss how the proposals in this paper could be progressed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. The full series of articles are freely available at go.nature.com/huhbyr

  2. http://www.mhra.gov.uk/Howweregulate/Medicines/Inspectionandstandards/GoodLaboratoryPractice/Structure/index.htm, last accessed 5 November 2013.

References

  • Adenso-Diaz B and Laguna M (2006). Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54 (1): 99–114.

    Article  Google Scholar 

  • Ahuja RV and Orlin J (1996). Use of representative operation counts in computational testing of algorithms. INFORMS Journal on Computing 8 (3): 318–330.

    Article  Google Scholar 

  • Anon (2013). Announcement: Reducing our irreproducibility. Nature 496 (7446): 398.

  • Archetti C and Speranza MG (2008). The split delivery vehicle routing problem: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challanges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 103–122.

  • Baker M (2012). Independent labs to verify high-profile papers: Nature News, 14 August 2012, Chicago doi:10.1038/nature.2012.11176.

  • Balinski ML (1978). On the reporting of computational experiments. Mathematical Programming 15 (1): 315.

    Article  Google Scholar 

  • Barr RS, Golden BL, Kelly JP, Resende MGC and Stewart Jr WR (1995). Designing and reporting on computational experiments with heuristic methods. Journal of Heuristics 1 (1): 9–32.

    Article  Google Scholar 

  • Begley CG and Ellis LM (2012). Drug development: Raise standards for preclinical cancer research. Nature 483 (7391): 531–533.

    Article  Google Scholar 

  • Bellmore M and Nemhauser GL (1968). Travelling salesman problem: A survey. Operations Research 16 (3): 538–558.

    Article  Google Scholar 

  • Błazewicz J, Domschke W and Pesch E (1996). The job shop scheduling problem: Conventional and new solution techniques. European Journal of Operational Research 93 (1): 1–33.

    Article  Google Scholar 

  • Błazewicz J, Ecker KH, Pesch E, Schmidt G and Weglarz J (2007). Handbook on Scheduling. Springer–Verlag: Berlin, Heidelberg.

    Google Scholar 

  • Błazewicz J, Formanowicz P and Kasprzak M (2005). Selected combinatorial problems of computational biology. European Journal of Operational Research 161 (3): 585–597.

    Article  Google Scholar 

  • Boylan JE, Goodwin P, Mohammadipour M and Syntetos AA (2015). Reproducibility in forecasting research. International Journal of Forecasting 31 (1): 79–90.

    Article  Google Scholar 

  • Braysy O, Dullaert W and Gendreau M (2004). Evolutionary algorithms for the vehicle routing problem with time windows. Journal of Heuristics 10 (6): 587–611.

    Article  Google Scholar 

  • Braysy O and Gendreau M (2005a). Vehicle routing problem with time windows, part I: Route construction and local search algorithms. Transportation Science 39 (1): 104–118.

    Article  Google Scholar 

  • Braysy O and Gendreau M (2005b). Vehicle routing problem with time windows, part II: Metaheuristics. Transportation Science 39 (1): 119–139.

    Article  Google Scholar 

  • Brown L (1999). Technical and Military Imperatives: A Radar History of World War 2. CRC Press: Bristol, UK.

    Book  Google Scholar 

  • Brucker P, Sotskov YN and Werner F (2007). Complexity of shop-scheduling problems with fixed number of jobs: A survey. Mathematical Methods of Operations Research 65 (3): 461–481.

    Article  Google Scholar 

  • Burkard RE, Deineko VG, Van Dal R, Van der Veen JAA and Woeginger GJ (1998). Well-solvable special cases of the traveling salesman problem: A survey. SIAM Review 40 (3): 496–546.

    Article  Google Scholar 

  • Burke EK et al (2013). Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64 (12): 1695–1724.

    Article  Google Scholar 

  • Cattrysse DG and Van Wassenhove LN (1992). A survey of algorithms for the generalized assignment problem. European Journal of Operational Research 60 (3): 260–272.

    Article  Google Scholar 

  • Cheng RW, Gen M and Tsujimura Y (1999a). A tutorial survey of jobshop scheduling problems using genetic algorithms, part II: Hybrid genetic search strategies. Computers & Industrial Engineering 36 (2): 343–364.

    Article  Google Scholar 

  • Cheng RW, Gen M and Tsujimura Y (1999b). A tutorial survey of jobshop scheduling problems using genetic algorithms: Part II. Hybrid genetic search strategies. Computers & Industrial Engineering 37 (1–2): 51–55.

    Article  Google Scholar 

  • Cook WJ (2011). In Pursuit of the Traveling Salesman: Mathematics at the Limits of Computation. Princeton University Press: USA.

    Google Scholar 

  • Cordeau JF, Gendreau M, Laporte G, Potvin JY and Semet F (2002). A guide to vehicle routing heuristics. Journal of the Operational Research Society 53 (5): 512–522.

    Article  Google Scholar 

  • Coy SP, Golden BL, Runger GC and Wasil EA (2001). Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics 7 (1): 77–97.

    Article  Google Scholar 

  • Crowder H, Dembo RS and Mulvey JM (1979). On reporting computational experiments with mathematical software. ACM Transactions on Mathematical Software 5 (2): 193–203.

    Article  Google Scholar 

  • Crowder HP, Dembo RS and Mulvey JM (1978). Reporting computational experiments in mathematical programming. Mathematical Programming 15 (1): 316–329.

    Article  Google Scholar 

  • Derrac J, Garcίa S, Molina D and Herrera F (2011). A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation 1 (1): 3–18.

    Article  Google Scholar 

  • Doerner KF and Schmid V (2010). Survey: Matheuristics for rich vehicle routing problems. In: Blesa MJ, Blum C, Raidl G, Roli A and Sampels M (eds). Hybrid Metaheuristics, Lecture Notes in Computer Science. Vol. 6373, 7th International Workshop on Hybrid Metaheuristics, Vienna, Austria, Springer: Berlin Heidelberg, pp 206–221, Oct 01-02, 2010.

  • Dongarra JJ (1992). Performance of various computers using standard linear equations software. ACM SIGARCH Computer Architecture News 20 (3): 22–44.

    Article  Google Scholar 

  • Easley RW, Madden CS and Dunn MG (2000). Conducting marketing science: The role of replication in the research process. Journal of Business Research 48 (1): 83–92.

    Article  Google Scholar 

  • Easton K, Nemhauser G and Trick M (2001). The traveling tournament problem description and benchmarks, chap. Principles and Practice of Constraint Programming CP 2001: 7th International Conference, CP 2001 Paphos, Cyprus, 26 November–1 December, 2001 Proceedings. Lecture Notes in Computer Science 2239, Springer, Berlin Heidelberg, pp 580–584.

  • Ernst AT, Jiang H, Krishnamoorthy M and Sier D (2004). Staff scheduling and rostering: A review of applications, methods and models. European Journal of Operational Research 153 (1): 3–27.

    Article  Google Scholar 

  • Evanschitzky H and Armstrong JS (2010). Replications of forecasting research. International Journal of Forecasting 26 (1): 4–8.

    Article  Google Scholar 

  • Evanschitzky H, Baumgarth C, Hubbard R and Armstrong JS (2007). Replication research’s disturbing trend. Journal of Business Research 60 (4): 411–415.

    Article  Google Scholar 

  • Fildes R (1979). Quantitative forecasting the state of the art: Extrapolative models. Journal of the Operational Research Society 30 (8): 691–710.

    Google Scholar 

  • Fildes R (1985). Quantitative forecasting the state of the art: Econometric models. Journal of the Operational Research Society 30 (7): 549–580.

    Google Scholar 

  • Fildes R, Nikolopoulos K, Crone SF and Syntetos AA (2008). Forecasting and operational research: A review. Journal of the Operational Research Society 59 (9): 1150–1172.

    Article  Google Scholar 

  • Garcίa S, Fernández A, Luengo J and Herrera F (2010). Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences 180 (10): 2044–2064.

    Article  Google Scholar 

  • Gass SI and Assad AA (2006). An Annotated Timeline of Operations Research: An Informal History. Springer: New York.

    Google Scholar 

  • Gendreau M, Potvin J-Y, Braysy O, Hasle G and Lokketangen A (2008). Metaheuristics for the vehicle routing problem and its extensions: A categorized bibliography. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 143–169.

  • Ghobbar AA and Friend CH (2003). Evaluation of forecasting methods for intermittent parts demand in the field of aviation: A predictive model. Computers & Operations Research 30 (14): 2097–2114.

    Article  Google Scholar 

  • Golden BL, Assad AA, Wasil WA and Baker E (1986). Experimentation in optimization. European Journal of Operational Research 27 (1): 1–16.

    Article  Google Scholar 

  • Golden BL and Stewart WR (1985). The Traveling Salesman Problem. chap. Empirical Analysis of Heuristics John Wiley & Sons: Chichester, UK, pp 207–249.

    Google Scholar 

  • Greenberg HJ (1990). Computational testing: Why, how and how much. ORSA Journal on Computing 2 (1): 94–97.

    Article  Google Scholar 

  • Hellekalek P (1998). Good random number generators are (not so) easy to find. Mathematics and Computers in Simulation 46 (5–6): 485–505.

    Article  Google Scholar 

  • Hoffman A, Mannos M, Sokolowsky D and Wiegmann N (1953). Computational experience in solving linear programs. Journal of the Society for Industrial and Applied Mathematics 1 (1): 17–33.

    Article  Google Scholar 

  • Hooker JN (1994). Needed: An empirical science of algorithms. Operations Research 42 (2): 201–212.

    Article  Google Scholar 

  • Hooker JN (1995). Testing heuristics—We have it all wrong. Journal of Heuristics 1 (1): 33–42.

    Article  Google Scholar 

  • Hooker JN (2007). Good and bad futures for constraint programming (and operations research). Constraint Programming Letters 1: 21–32.

    Google Scholar 

  • Hubbard R and Armstrong JS (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11 (3): 233–248.

    Article  Google Scholar 

  • Hubbard R and Vetter DE (1996). An empirical comparison of published replication research in accounting, economics, finance, management, and marketing. Journal of Business Research 35 (2): 153–164.

    Article  Google Scholar 

  • Ignizio JP (1971). On the establishment of standards for comparing algorithm performance. Interfaces 2 (1): 8–11.

    Article  Google Scholar 

  • Ince DC, Hatton L and Graham-Cumming J (2012). The case for open computer programs. Nature 482 (7386): 485–488.

    Article  Google Scholar 

  • Ioannidis JA (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association 294 (2): 218–228.

    Article  Google Scholar 

  • Jackson RHF, Boggs PT, Nash SG and Powell S (1991). Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Mathematical Programming 49 (1–3): 413–425.

    Google Scholar 

  • Jackson RHF and Mulvey JM (1978). A critical review of methods for comparing mathematical programming algorithms and software (1953–1977). Journal of Research of the National Bureau of Standards 83 (6): 563–584.

    Article  Google Scholar 

  • Jaillet P and Wagner MR (2008). Online vehicle routing problems: A survey. In: Golden B, Raghavan S and Wasil E (eds). Vehicle Routing Problem: Latest Advances and New Challenges, Operations Research Computer Science Interfaces. Vol. 43, Springer, New York, pp 221–237.

  • Jourdan L, Basseur M and Talbi E-G (2009). Hybridizing exact methods and metaheuristics: A taxonomy. European Journal of Operational Research 199 (3): 620–629.

    Article  Google Scholar 

  • Kendall G, Knust S, Ribeiro CC and Urrutia SS (2010). Scheduling in sports: An annotated bibliography. Computers & Operations Research 37 (1): 1–19.

    Article  Google Scholar 

  • Kiran AS and Smith ML (1984). Simulation studies in job shop scheduling—1. A survey. Computers & Industrial Engineering 8 (2): 87–93.

    Article  Google Scholar 

  • Kirkpatrick S, Gelatt Jr CD and Vecchi M (1983). Optimization by simulated annealing. Science 220 (4598): 671–680.

    Article  Google Scholar 

  • Laporte G (2009). Fifty years of bvehicle routing. Transportation Science 43 (4): 408–416.

    Article  Google Scholar 

  • Lee C-Y, Bard J, Pinedo M and Wilhelm WE (1993). Guidelines for reporting computational resuts in IIE transactions. IIE Transactions 25 (6): 121–123.

    Article  Google Scholar 

  • Lin EYH (1998). A bibliographical survey on some well-known non-standard knapsack problems. INFOR 36 (4): 274–317.

    Google Scholar 

  • Loiola EM, de Abreu NMM, Boaventura-Netto PO, Hahn P and Querido T (2007). A survey for the quadratic assignment problem. European Journal of Operational Research 176 (2): 657–690.

    Article  Google Scholar 

  • Lukasiak P, Błazewicz J and Milostan M (2010). Some operations research methods for analyzing protein sequences and structures. Annals of Operations Research 175 (1): 9–35.

    Article  Google Scholar 

  • Lust T and Teghem J (2010). The multiobjective traveling salesman problem: A survey and a new approach. In: Coello CAC, Dhaenens C and Jourdan L (eds). Studies in Computational Intelligence. Vol. 272, Springer-Verlag: Berlin, pp 119–141.

    Google Scholar 

  • Lust T and Teghem J (2012). The multiobjective multidimensional knapsack problem: A survey and a new approach. International Transactions in Operational Research 19 (4): 495–520.

    Article  Google Scholar 

  • Marshall E (1983). The murky world of toxicity testing. Science 220 (4602): 1130–1132.

    Article  Google Scholar 

  • McCollum B et al (2010). Setting the research agenda in automated timetabling: The second international timetabling competition. INFORMS Journal on Computing 22 (1): 120–130.

    Article  Google Scholar 

  • McGeoch CC (1996). Toward an experimental method for algorithm simulation. INFORMS Journal on Computing 8 (1): 1–15.

    Article  Google Scholar 

  • Miller HE (1976). Personnel scheduling in public systems—A survey. Socio-Economic Planning Sciences 10 (6): 241–249.

    Article  Google Scholar 

  • Miller DM and Williams D (2003). Shrinkage estimators of time series seasonal factors and their effect on forecasting accuracy. International Journal of Forecasting 19 (4): 669–684.

    Article  Google Scholar 

  • Pentico DW (2007). Assignment problems: A golden anniversary survey. European Journal of Operational Research 176 (2): 774–793.

    Article  Google Scholar 

  • Pisinger D (2007). The quadratic knapsack problem—A survey. Discrete Applied Mathematics 155 (5): 623–648.

    Article  Google Scholar 

  • Prinz F, Schlange T and Asadullah K (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10 (7391): 10–11.

    Google Scholar 

  • Rardin RL and Uzsoy R (2001). Experimental evaluation of heuristic optimization algorithms: A tutorial. Journal of Heuristics 7 (3): 261–304.

    Article  Google Scholar 

  • Rasmussen RV and Trick MA (2008). Round robin scheduling—A survey. European Journal of Operational Research 188 (3): 617–636.

    Article  Google Scholar 

  • Russell JF (2013). If a job is worth doing, it is worth doing twice. Nature 496 (7443): 7.

    Article  Google Scholar 

  • Salkin HM and Kluyver CAD (1975). Knapsack problem—Survey. Naval Research Logistics 22 (1): 127–144.

    Article  Google Scholar 

  • Sӧrensen K (2015). Metaheuristics—The metaphor exposed. International Transactions in Operational Research 22 (1): 3–18.

    Article  Google Scholar 

  • Sӧrensen K and Glover F (2013). Encyclopedia of Operations Research and Management Science. chap. Metaheuristics 3rd edn. Springer: New York.

    Google Scholar 

  • Taillard ÉD, Waelti P and Zuber J (2008). Few statistical tests for proportions comparison. European Journal of Operational Research 185 (3): 1336–1350.

    Article  Google Scholar 

  • Talbi E-G (2009). Metaheuristics: From Design to Implementation. Wiley: Hoboken, NJ.

    Book  Google Scholar 

  • Trapero JR, Kourentzes N and Fildes R (2015). Identification of sales forecasting models. Journal of the Operational Research Society 66 (2): 299–307.

    Article  Google Scholar 

  • Vaux DL (2012). Research methods: Know when your numbers are significant. Nature 492 (7428): 180–181.

    Google Scholar 

  • Vidal T, Crainic TG, Gendreau M and Prins C (2013). Heuristics for multiattribute vehicle routing problems: A survey and synthesis. European Journal of Operational Research 231 (1): 1–21.

    Article  Google Scholar 

  • Vines TH et al (2013). The availability of research data declines rapidly with article age. Current Biology 24 (1): 94–97.

    Article  Google Scholar 

  • Wilbaut C, Hanafi S and Salhi S (2008). A survey of effective heuristics and their application to a variety of knapsack problems. IMA Journal of Management Mathematics 19 (3): 227–244.

    Article  Google Scholar 

  • Zanakis SH, Evans JR and Vazacopoulos AA (1989). Heuristic methods and applications: A categorized survey. European Journal of Operational Research 43 (1): 88–110.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Nottingham, Nottingham, UK

    Graham Kendall, Robert John, Jiawei Li & Rong Qu

  2. University of Nottingham Malaysia Campus, Semenyih, Malaysia

    Graham Kendall, Nasser Sabar & Angelina Yee

  3. University of Nottingham Ningbo, Ningbo, China

    Ruibin Bai

  4. Poznan University of Technology, Poznan, Poland

    Jacek Błazewicz

  5. KU Leuven, Campus Kulak, Kortrijk, Belgium

    Patrick De Causmaecker

  6. University of Montreal, Montreal, Canada

    Michel Gendreau

  7. Queen’s University Belfast, Belfast, UK

    Barry McCollum

  8. Universität Siegen, Siegen, Germany

    Erwin Pesch

  9. KU Leuven, Technology Campus Gent, Gent, Belgium

    Greet Vanden Berghe

Authors
  1. Graham Kendall
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruibin Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacek Błazewicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrick De Causmaecker
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michel Gendreau
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Robert John
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiawei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Barry McCollum
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Erwin Pesch
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Rong Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Nasser Sabar
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Greet Vanden Berghe
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Angelina Yee
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kendall, G., Bai, R., Błazewicz, J. et al. Good Laboratory Practice for optimization research. J Oper Res Soc 67, 676–689 (2016). https://doi.org/10.1057/jors.2015.77

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jors.2015.77

Keywords

Search

Navigation