Abstract
Optimisation algorithms are commonly compared on benchmarks to get insight into performance differences. However, it is not clear how closely benchmarks match the properties of real-world problems because these properties are largely unknown. This work investigates the properties of real-world problems through a questionnaire to enable the design of future benchmark problems that more closely resemble those found in the real world. The results, while not representative as they are based on only 45 responses, indicate that many problems possess at least one of the following properties: they are constrained, deterministic, have only continuous variables, require substantial computation times for both the objectives and the constraints, or allow a limited number of evaluations. Properties like known optimal solutions and analytical gradients are rarely available, limiting the options in guiding the optimisation process. These are all important aspects to consider when designing realistic benchmark problems. At the same time, the design of realistic benchmarks is difficult, because objective functions are often reported to be black-box and many problem properties are unknown. To further improve the understanding of real-world problems, readers working on a real-world optimisation problem are encouraged to fill out the questionnaire: https://tinyurl.com/opt-survey.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
N. Beume, B. Naujoks, M.T.M. Emmerich, SMS-EMOA: multiobjective selection based on dominated hypervolume. Eur. J. Oper. Res. 181(3), 1653–1669 (2007)
D. Brockhoff, D. Horn, J. Knowles, I. Loshchilov, T. Tušar, I. Voutchkov, Publicly available real-world optimization problems and free simulators (2016). https://lcworkshops.github.io/samco/benchmarking/. Accessed 15 April 2020
M.R. Bussieck, A.S. Drud, A. Meeraus, Minlplib-a collection of test models for mixed-integer nonlinear programming. INFORMS J. Comput. 15(1), 114–119 (2003)
C.A.C. Coello, N.C. Cortés, Solving multiobjective optimization problems using an artificial immune system. Genetic Prog. Evolvable Mach. 6(2), 163–190 (2005)
S.J. Daniels, A.A. Rahat, R.M. Everson, G.R. Tabor, J.E. Fieldsend, A suite of computationally expensive shape optimisation problems using computational fluid dynamics, in Parallel Problem Solving from Nature (PPSN) (Springer, 2018), pp. 296–307
S. Das, P.N. Suganthan, Problem definitions and evaluation criteria for CEC 2011 competition on testing evolutionary algorithms on real world optimization problems. Technical report, Jadavpur University, Kolkata, India (2010)
C. Doerr, J. Dreo, P. Kerschke, Making a case for (hyper-) parameter tuning as benchmark problems, in Genetic and Evolutionary Computation Conference (GECCO) Companion (ACM Press, 2019), pp. 1755–1764
K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, F. Hutter, HPOBench: a collection of reproducible multi-fidelity benchmark problems for HPO, in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021). https://openreview.net/forum?id=1k4rJYEwda
E.J. Hughes, Radar waveform optimisation as a many-objective application benchmark, in Evolutionary Multi-criterion Optimization (EMO) (Springer, 2007), pp. 700–714
H. Ishibuchi, L. He, K. Shang, Regular Pareto front shape is not realistic, in Congress on Evolutionary Computation (CEC) (IEEE Press, 2019), pp. 2034–2041
Y. Jin, Surrogate-assisted evolutionary computation: recent advances and future challenges. Swarm Evolut. Comput. 1(2), 61–70 (2011)
JSEC and JAXA. The 1st Evolutionary Computation Competition - Car Structure Design Optimization (2017). http://is-csse-muroran.sakura.ne.jp/ec2017/EC2017compe.html (in Japanese). Accessed 20 April 2020
P. Kerschke, M. Gallagher, M. Preuss, O. Teytaud, The Machine Learning and Data Analysis (MLDA) Problem Set, v1.0 (2019). https://www.wi.uni-muenster.de/sites/wi/files/users/kerschke/gecco2019/gecco2019_umlop_mlda.pdf. Accessed 15 April 2020
B.A. Kitchenham, S.L. Pfleeger, Principles of survey research part 2: designing a survey. SIGSOFT Softw. Eng. Notes 27(1), 18–20 (2002)
T. Kohira, H. Kemmotsu, O. Akira, T. Tatsukawa, Proposal of benchmark problem based on real-world car structure design optimization, in Genetic and Evolutionary Computation Conference (GECCO) (ACM Press, 2018), pp. 183–184
T. Liao, K. Socha, M.A.M. de Oca, T. Stützle, M. Dorigo, Ant colony optimization for mixed-variable optimization problems. IEEE Trans. Evolut. Comput. 18(4), 503–518 (2013)
K. McClymont, E. Keedwell, Benchmark multi-objective optimisation test problems with mixed encodings, in Congress on Evolutionary Computation (CEC) (IEEE Press, 2011), pp. 2131–2138
O. Mersmann, B. Bischl, H. Trautmann, M. Preuss, C. Weihs, G. Rudolph, Exploratory landscape analysis, in Genetic and Evolutionary Computation Conference (GECCO) (ACM Press, 2011), pp. 829–836
A. Oyama, H. Fukumoto, T. Tatsukawa, Proposal of an optimization benchmark problem based on lunar lander landing site selection problem, in 5th Optimization in Space Engineering Workshop (2019)
S. Polyakovskiy, M.R. Bonyadi, M. Wagner, Z. Michalewicz, F. Neumann, A comprehensive benchmark set and heuristics for the traveling thief problem, inGenetic and Evolutionary Computation Conference (GECCO) (ACM Press, 2014), pp. 477–484
K.L. Sadowski, D. Thierens, P.A. Bosman, Gambit: a parameterless model-based evolutionary algorithm for mixed-integer problems. Evolut. Comput. 26(1), 117–143 (2018)
R. Tanabe, H. Ishibuchi, An easy-to-use real-world multi-objective optimization problem suite. Appl. Soft Comput. 89:106078, 2020. https://github.com/ryojitanabe/reproblems. Accessed 15 April 2020
R. Tanabe, A. Oyama. A note on constrained multi-objective optimization benchmark problems, in Congress on Evolutionary Computation (CEC) (IEEE Press, 2017), pp. 1127–1134
The Benchmarking Network. Benchmarking Network Homepage (2019). https://sites.google.com/view/benchmarking-network. Accessed 13 Sep 2020
The Japanese Society for Evolutionary Computation (JSEC) and Japan Aerospace Exploration Agency (JAXA). The 2nd Evolutionary Computation Competition - Lunar lander Landing Site Selection (2018). http://www.jpnsec.org/files/competition2018/EC-Symposium-2018-Competition-English.html. Accessed 20 April 2020
The Japanese Society of Evolutionary Computation (JSEC). The 3rd Evolutionary Computation Competition - Wind Turbine Design Optimization (2019). http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html. Accessed 1 Sep 2020
A. Tiwari, P. Noriega Hoyos, W. Hutabarat, C. Turner, N. Ince, X.-P. Gan, N. Prajapat, Survey on the use of computational optimisation in UK engineering companies. CIRP J. Manuf. Sci. Technol. 9, 57–68 (2015)
T. Tušar, D. Brockhoff, N. Hansen, Mixed-integer benchmark problems for single- and bi-objective optimization, in Genetic and Evolutionary Computation Conference (GECCO) (ACM Press, 2019), pp. 718–726
K. van der Blom, T. Deist, M. Marchi, B. Naujoks, Y. Nojima, A. Oyama, T. Tušar, V. Volz, MACODA Working Group: Survey on Real-World Optimization Problems (2020). https://sites.google.com/view/macoda-rwp. Accessed 15 April 2020
K. van der Blom, T.M. Deist, T. Tušar, M. Marchi, Y. Nojima, A. Oyama, V. Volz, B. Naujoks, Towards realistic optimization benchmarks: A questionnaire on the properties of real-world problems, in Genetic and Evolutionary Computation Conference (GECCO) Companion (ACM Press, 2020), pp. 293–294
D.A. Van Veldhuizen, G.B. Lamont, On measuring multiobjective evolutionary algorithm performance, in Congress on Evolutionary Computation (CEC), vol. 1 (IEEE Press, 2000), pp. 204–211
V. Volz, B. Naujoks, P. Kerschke, T. Tušar, Single- and multi-objective game-benchmark for evolutionary algorithms, in Genetic and Evolutionary Computation Conference (GECCO) (ACM Press, 2019), pp. 647–655
H. Wang, C. He, Y. Tian, Y. Jin, Competition on Online Data-driven Multi-objective Optimization (2019). https://handingwang.github.io/DDMOP/. Accessed 15 April 2020
J.-P. Watson, L. Barbulescu, A. Howe, D. Whitley, Algorithm performance and problem structure for flow-shop scheduling, in National Conference on Artificial Intelligence (AAAI) (1999)
D.R. White, J. McDermott, M. Castelli, L. Manzoni, B.W. Goldman, G. Kronberger, W. Jaśkowski, U.-M. O’Reilly, S. Luke, Better GP benchmarks: community survey results and proposals. Gen. Progr. Evolvable Mach. 14(1), 3–29 (2013)
D. Whitley, S. Rana, J. Dzubera, K.E. Mathias, Evaluating evolutionary algorithms. Artif. Intell. 85, 245–276 (1995)
Acknowledgements
The questionnaire was first proposed at the Lorentz Center MACODA (Many Criteria Optimization and Decision Analysis) workshop as a group effort by J. Fieldsend, J. Forde, H. Ishibuchi, E. Marescaux, M. Miyakawa, R. Purshouse, J. Richter, D. Thierens, C. Touré, and the authors of this paper. We thank the working group participants for their valuable contributions. We also wish to thank R. Allmendiger, J. Alza-Santos, M. M. Awad, M. Balvert, L. Bliek, A. Bouter, B. Breiderhoff, K. Chiba, C. Doerr, M. Ehrgott, M. Erascu, D. Gaudrie, M. Interciso, M. Kanazaki, J. Knowles, T. Kohira, P. Z. Korondi, O. Krause, W. B. Langdon, M. van der Meer, N. Namura, M. Ohki, Y. Ohta, J. Rohmer, M. Schlueter, N. Urquhart, A. Zamuda and all anonymous contributors for their time and effort to fill in the questionnaire. Timo M. Deist is funded by the Open Technology Programme (project No. 15586), financed by the Dutch Research Council (NWO), Elekta, and Xomnia. This project is co-funded by the public-private partnership allowance for top consortia for knowledge and innovation (TKIs) from the Ministry of Economic Affairs. Boris Naujoks acknowledges the European Commission’s H2020 programme, H2020-MSCA-ITN-2016 UTOPIAE (grant agreement No. 722734), and the DAAD (German Academic Exchange Service), Project-ID: 57515062. Tea Tušar acknowledges financial support from the Slovenian Research Agency (projects No. Z2-8177 and BI-DE/20-21-019 and program No. P2-0209) and the European Commission’s Horizon 2020 research and innovation program (grant agreement No. 692286).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this chapter
Cite this chapter
van der Blom, K. et al. (2023). Identifying Properties of Real-World Optimisation Problems Through a Questionnaire. In: Brockhoff, D., Emmerich, M., Naujoks, B., Purshouse, R. (eds) Many-Criteria Optimization and Decision Analysis. Natural Computing Series. Springer, Cham. https://doi.org/10.1007/978-3-031-25263-1_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-25263-1_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25262-4
Online ISBN: 978-3-031-25263-1
eBook Packages: Computer ScienceComputer Science (R0)