Advertisement

Automatic Configuration of Multi-objective Optimizers and Multi-objective Configuration

  • Leonardo C. T. Bezerra
  • Manuel López-Ibáñez
  • Thomas StützleEmail author
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 833)

Abstract

Heuristic optimizers are an important tool in academia and industry, and their performance-optimizing configuration requires a significant amount of expertise. As the proper configuration of algorithms is a crucial aspect in the engineering of heuristic algorithms, a significant research effort has been dedicated over the last years towards moving this step to the computer and, thus, make it automatic. These research efforts go way beyond tuning only numerical parameters of already fully defined algorithms, but exploit automatic configuration as a means for automatic algorithm design. In this chapter, we review two main aspects where the research on automatic configuration and multi-objective optimization intersect. The first is the automatic configuration of multi-objective optimizers, where we discuss means and specific approaches. In addition, we detail a case study that shows how these approaches can be used to design new, high-performing multi-objective evolutionary algorithms. The second aspect is the research on multi-objective configuration, that is, the possibility of using multiple performance metrics for the evaluation of algorithm configurations. We highlight some few examples in this direction.

Notes

Acknowledgements

This work received support from the COMEX project within the Interuniversity Attraction Poles Programme of the Belgian Science Policy Office. Thomas Stützle acknowledges support from the Belgian F.R.S.-FNRS, of which he is a research director.

References

  1. 1.
    Abbass, H.A.: The self-adaptive Pareto differential evolution algorithm. In: Proceedings of the 2002 Congress on Evolutionary Computation (CEC 2002), pp. 831–836. IEEE Press, NJ (2002)Google Scholar
  2. 2.
    Abbass, H.A., Sarker, R., Newton, C.: PDE: A Pareto-frontier differential evolution approach for multi-objective optimization problems. In: Proceedings of the 2001 Congress on Evolutionary Computation (CEC 2001), pp. 971–978. IEEE Press, NJ (2001)Google Scholar
  3. 3.
    Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Gent, I.P. (ed.) Principles and Practice of Constraint Programming, CP 2009. Lecture Notes in Computer Science, vol. 5732, pp. 142–157. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  4. 4.
    Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-race algorithm: Sampling design and iterative refinement. In: Bartz-Beielstein, T., Blesa, M.J., Blum, C., Naujoks, B., Roli, A., Rudolph, G., Sampels, M. (eds.) Hybrid Metaheuristics. Lecture Notes in Computer Science, vol. 4771, pp. 108–122. Springer, Heidelberg (2007)Google Scholar
  5. 5.
    Bartz-Beielstein, T., Lasarczyk, C., Preuss, M.: The sequential parameter optimization toolbox. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 337–360. Springer, Berlin (2010)CrossRefGoogle Scholar
  6. 6.
    Bezerra, L.C.T.: A component-wise approach to multi-objective evolutionary algorithms: from flexible frameworks to automatic design. Ph.D. thesis, IRIDIA, École polytechnique, Université Libre de Bruxelles, Belgium (2016)Google Scholar
  7. 7.
    Bezerra, L.C.T., López-Ibáñez, M., Stützle, T.: Automatic generation of multi-objective ACO algorithms for the biobjective knapsack. In: Dorigo, M., et al. (eds.) Swarm Intelligence, 8th International Conference, ANTS 2012. Lecture Notes in Computer Science, vol. 7461, pp. 37–48. Springer, Heidelberg (2012)Google Scholar
  8. 8.
    Bezerra, L.C.T., López-Ibáñez, M., Stützle, T.: Automatic component-wise design of multi-objective evolutionary algorithms (2015). http://iridia.ulb.ac.be/supp/IridiaSupp2014-010/
  9. 9.
    Bezerra, L.C.T., López-Ibáñez, M., Stützle, T.: Automatic component-wise design of multi-objective evolutionary algorithms. IEEE Trans. Evol. Comput. 20(3), 403–417 (2016)CrossRefGoogle Scholar
  10. 10.
    Birattari, M.: The problem of tuning metaheuristics as seen from a machine learning perspective. Ph.D. thesis, IRIDIA, École polytechnique, Université Libre de Bruxelles, Belgium (2004)Google Scholar
  11. 11.
    Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-race and iterated F-race: An overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 311–336. Springer, Berlin (2010)CrossRefGoogle Scholar
  12. 12.
    Blot, A., Hoos, H.H., Jourdan, L., Kessaci-Marmion, M.E., Trautmann, H.: MO-ParamILS: A multi-objective automatic algorithm configuration framework. In: Festa, P., Sellmann, M., Vanschoren, J. (eds.) Learning and Intelligent Optimization, 10th International Conference, LION 10. Lecture Notes in Computer Science, vol. 10079, pp. 32–47. Springer, Cham (2016)Google Scholar
  13. 13.
    Blot, A., Pernet, A., Jourdan, L., Kessaci-Marmion, M.E., Hoos, H.H.: Automatically configuring multi-objective local search using multi-objective optimisation. In: Trautmann, H., Rudolph, G., Klamroth, K., Schütze, O., Wiecek, M.M., Jin, Y., Grimme, C. (eds.) Evolutionary Multi-criterion Optimization, EMO 2017. Lecture Notes in Computer Science, pp. 61–76. Springer International Publishing, Cham (2017)Google Scholar
  14. 14.
    Box, G.E.P., Draper, N.R.: Response Surfaces, Mixtures, and Ridge Analyses. Wiley, New York (2007)Google Scholar
  15. 15.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  16. 16.
    Burke, E.K., Hyde, M.R., Kendall, G.: Grammatical evolution of local search heuristics. IEEE Trans. Evol. Comput. 16(7), 406–417 (2012)CrossRefGoogle Scholar
  17. 17.
    Dang Thi Thanh, N., De Causmaecker, P.: Motivations for the development of a multi-objective algorithm configurator. In: Vitoriano, B., Pinson, E., Valente, F. (eds.) ICORES 2014 - Proceedings of the 3rd International Conference on Operations Research and Enterprise Systems, pp. 328–333. SciTePress (2014)Google Scholar
  18. 18.
    Deb, K., Thiele, L., Laumanns, M., Zitzler, E.: Scalable test problems for evolutionary multiobjective optimization. In: Abraham, A., Jain, L., Goldberg, R. (eds.) Evolutionary Multiobjective Optimization, Advanced Information and Knowledge Processing, pp. 105–145. Springer, London, UK (2005)Google Scholar
  19. 19.
    Dorigo, M., Gambardella, L.M.: Ant colony system: A cooperative learning approach to the traveling salesman problem. IEEE Trans. Evol. Comput. 1(1), 53–66 (1997)CrossRefGoogle Scholar
  20. 20.
    Dréo, J.: Using performance fronts for parameter setting of stochastic metaheuristics. In: Rothlauf, F. (ed.) GECCO (Companion), pp. 2197–2200. ACM Press, New York (2009)Google Scholar
  21. 21.
    Dubois-Lacoste, J., López-Ibáñez, M., Stützle, T.: Automatic configuration of state-of-the-art multi-objective optimizers using the TP\(+\)PLS framework. In: Krasnogor, N., Lanzi, P.L. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2011, pp. 2019–2026. ACM Press, New York (2011)Google Scholar
  22. 22.
    Dubois-Lacoste, J., López-Ibáñez, M., Stützle, T.: Combining two search paradigms for multi-objective optimization: Two-Phase and Pareto local search. In: Talbi, E.G. (ed.) Hybrid Metaheuristics. Studies in Computational Intelligence, vol. 434, pp. 97–117. Springer, Berlin (2013)CrossRefGoogle Scholar
  23. 23.
    Eiben, A.E., Smit, S.K.: Parameter tuning for configuring and analyzing evolutionary algorithms. Swarm Evol. Comput. 1(1), 19–31 (2011)CrossRefGoogle Scholar
  24. 24.
    Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems (NIPS 28), pp. 2962–2970 (2015)Google Scholar
  25. 25.
    Fukunaga, A.S.: Evolving local search heuristics for SAT using genetic programming. In: Deb, K. et al. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2004, Part II. Lecture Notes in Computer Science, vol. 3103, pp. 483–494. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  26. 26.
    Fukunaga, A.S.: Automated discovery of local search heuristics for satisfiability testing. Evol. Comput. 16(1), 31–61 (2008)CrossRefGoogle Scholar
  27. 27.
    Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, USA (1989)Google Scholar
  28. 28.
    Hoos, H.H.: Automated algorithm configuration and parameter tuning. In: Hamadi, Y., Monfroy, E., Saubion, F. (eds.) Autonomous Search, pp. 37–71. Springer, Berlin (2012)CrossRefGoogle Scholar
  29. 29.
    Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)CrossRefGoogle Scholar
  30. 30.
    Huband, S., Hingston, P., Barone, L., While, L.: A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans. Evol. Comput. 10(5), 477–506 (2006)CrossRefGoogle Scholar
  31. 31.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. In: Lodi, A., Milano, M., Toth, P. (eds.) 7th International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, CPAIOR 2010. Lecture Notes in Computer Science, vol. 6140, pp. 186–202. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  32. 32.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello Coello, C.A. (ed.) 5th International Conference on Learning and Intelligent Optimization, LION 5. Lecture Notes in Computer Science, vol. 6683, pp. 507–523. Springer, Heidelberg (2011)Google Scholar
  33. 33.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: An automatic algorithm configuration framework. J. Artif. Intell. Res. 36, 267–306 (2009)CrossRefGoogle Scholar
  34. 34.
    Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: Methods and evaluation. Artif. Intell. 206, 79–111 (2014)MathSciNetCrossRefGoogle Scholar
  35. 35.
  36. 36.
    Ishibuchi, H., Masuda, H., Tanigaki, Y., Nojima, Y.: Modified distance calculation in generational distance and inverted generational distance. In: Gaspar-Cunha, A., Antunes, C.H., Coello Coello, C.A. (eds.) Evolutionary Multi-criterion Optimization, EMO 2015 Part I. Lecture Notes in Computer Science, vol. 9018, pp. 110–125. Springer, Heidelberg (2015)Google Scholar
  37. 37.
    Jiang, S., Ong, Y.S., Zhang, J., Feng, L.: Consistencies and contradictions of performance metrics in multiobjective optimization. IEEE Trans. Cybern. 44(12), 2391–2404 (2014)CrossRefGoogle Scholar
  38. 38.
    KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: Automatically building local search SAT solvers from components. In: Boutilier, C. (ed.) Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 517–524. AAAI Press, CA (2009)Google Scholar
  39. 39.
    Knowles, J.D., Thiele, L., Zitzler, E.: A tutorial on the performance assessment of stochastic multiobjective optimizers. TIK-Report 214, Computer Engineering and Networks Laboratory (TIK), Swiss Federal Institute of Technology (ETH), Zürich, Switzerland (2006). Revised versionGoogle Scholar
  40. 40.
    Kotthoff, L., Thornton, C., Hoos, H.H., Hutter, F., Leyton-Brown, K.: Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res. 17, 1–5 (2016)Google Scholar
  41. 41.
    Kukkonen, S., Lampinen, J.: GDE3: the third evolution step of generalized differential evolution. In: Proceedings of the 2005 Congress on Evolutionary Computation (CEC 2005), pp. 443–450. IEEE Press, NJ (2005)Google Scholar
  42. 42.
    Liao, T., Stützle, T., Montes de Oca, M.A., Dorigo, M.: A unified ant colony optimization algorithm for continuous optimization. Eur. J. Oper. Res. 234(3), 597–609 (2014)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Liefooghe, A., Jourdan, L., Talbi, E.G.: A software framework based on a conceptual unified model for evolutionary multiobjective optimization: ParadisEO-MOEO. Eur. J. Oper. Res. 209(2), 104–112 (2011)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Lindauer, M.T., Hoos, H.H., Hutter, F., Schaub, T.: AutoFolio: An automatically configured algorithm selector. J. Artif. Intell. Res. 53, 745–778 (2015)CrossRefGoogle Scholar
  45. 45.
    López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Stützle, T., Birattari, M.: The irace package: Iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)MathSciNetCrossRefGoogle Scholar
  46. 46.
    López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Birattari, M.: The irace package, iterated race for automatic algorithm configuration. Technical report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium (2011)Google Scholar
  47. 47.
    López-Ibáñez, M., Stützle, T.: An analysis of algorithmic components for multiobjective ant colony optimization: a case study on the biobjective TSP. In: Collet, P., Monmarché, N., Legrand, P., Schoenauer, M., Lutton, E. (eds.) Artificial Evolution: 9th International Conference, Evolution Artificielle, EA, 2009. Lecture Notes in Computer Science, vol. 5975, pp. 134–145. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  48. 48.
    López-Ibáñez, M., Stützle, T.: Automatic configuration of multi-objective ACO algorithms. In: Dorigo, M., et al. (eds.) 7th International Conference on Swarm Intelligence, ANTS 2010. Lecture Notes in Computer Science, vol. 6234, pp. 95–106. Springer, Heidelberg (2010)Google Scholar
  49. 49.
    López-Ibáñez, M., Stützle, T.: The impact of design choices of multi-objective ant colony optimization algorithms on performance: An experimental study on the biobjective TSP. In: Pelikan, M., Branke, J. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2010, pp. 71–78. ACM Press, New York (2010)Google Scholar
  50. 50.
    López-Ibáñez, M., Stützle, T.: The automatic design of multi-objective ant colony optimization algorithms. IEEE Trans. Evol. Comput. 16(6), 861–875 (2012)CrossRefGoogle Scholar
  51. 51.
    Lourenço, H.R., Martin, O., Stützle, T.: Iterated local search. In: Glover, F., Kochenberger, G. (eds.) Handbook of Metaheuristics, pp. 321–353. Kluwer Academic Publishers, Norwell, MA (2002)Google Scholar
  52. 52.
    Madavan, N.K.: Multiobjective optimization using a Pareto differential evolution approach. In: Proceedings of the 2002 World Congress on Computational Intelligence (WCCI 2002), pp. 1145–1150. IEEE Press, NJ (2002)Google Scholar
  53. 53.
    Marmion, M.E., Mascia, F., López-Ibáñez, M., Stützle, T.: Automatic design of hybrid stochastic local search algorithms. In: Blesa, M.J., Blum, C., Festa, P., Roli, A., Sampels, M. (eds.) Hybrid Metaheuristics. Lecture Notes in Computer Science, vol. 7919, pp. 144–158. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  54. 54.
    Mascia, F., López-Ibáñez, M., Dubois-Lacoste, J., Marmion, M.E., Stützle, T.: Algorithm comparison by automatically configurable stochastic local search frameworks: A case study using flow-shop scheduling problems. In: Blesa, M.J., Blum, C., Voß, S. (eds.) Hybrid Metaheuristics. Lecture Notes in Computer Science, vol. 8457, pp. 30–44. Springer, Heidelberg (2014)Google Scholar
  55. 55.
    Mascia, F., López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T.: From grammars to parameters: Automatic iterated greedy design for the permutation flow-shop problem with weighted tardiness. In: Pardalos, P.M., Nicosia, G. (eds.) 7th International Conference on Learning and Intelligent Optimization, LION 7. Lecture Notes in Computer Science, vol. 7997, pp. 321–334. Springer, Heidelberg (2013)Google Scholar
  56. 56.
    Mascia, F., López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T.: Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools. Comput. Oper. Res. 51, 190–199 (2014)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Mendoza, H., Klein, A., Feurer, M., Springenberg, J.T., Hutter, F.: Towards automatically-tuned neural networks. In: Workshop on Automatic Machine Learning, pp. 58–65 (2016)Google Scholar
  58. 58.
    Minella, G., Ruiz, R., Ciavotta, M.: A review and evaluation of multiobjective algorithms for the flowshop scheduling problem. INFORMS J. Comput. 20(3), 451–471 (2008)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Miranda, P., Silva, R.M., Prudêncio, R.B.: I/S-Race: An iterative multi-objective racing algorithm for the SVM parameter selection problem. In: 22st European Symposium on Artificial Neural Networks, Computational Intelligence And Machine Learning, Bruges, April 23-24-25, 2014, pp. 573–578. ESANN (2015)Google Scholar
  60. 60.
    Paquete, L., Chiarandini, M., Stützle, T.: Pareto local optimum sets in the biobjective traveling salesman problem: An experimental study. In: Gandibleux, X., Sevaux, M., Sörensen, K., T’Kindt, V. (eds.) Metaheuristics for Multiobjective Optimisation. Lecture Notes in Economics and Mathematical Systems, vol. 535, pp. 177–200. Springer, Berlin (2004)zbMATHGoogle Scholar
  61. 61.
    Paquete, L., Stützle, T.: A two-phase local search for the biobjective traveling salesman problem. In: Fonseca, C.M., Fleming, P.J., Zitzler, E., Deb, K., Thiele, L. (eds.) Evolutionary Multi-criterion Optimization, EMO 2003. Lecture Notes in Computer Science, vol. 2632, pp. 479–493. Springer, Heidelberg (2003)Google Scholar
  62. 62.
    Pérez Cáceres, L., López-Ibáñez, M., Hoos, H.H., Stützle, T.: An experimental study of adaptive capping in irace. In: Battiti, R., Kvasov, D.E., Sergeyev, Y.D. (eds.) 11th International Conference on Learning and Intelligent Optimization, LION 11. Lecture Notes in Computer Science, vol. 10556, pp. 235–250. Springer, Cham (2017)Google Scholar
  63. 63.
    Price, K., Storn, R.M., Lampinen, J.A.: Differential Evolution: A Practical Approach to Global Optimization. Springer, New York (2005)Google Scholar
  64. 64.
    Robič, T., Filipič, B.: DEMO: Differential evolution for multiobjective optimization. In: Coello Coello, C.A., Aguirre, A.H., Zitzler, E. (eds.) Evolutionary Multi-criterion Optimization, EMO 2005. Lecture Notes in Computer Science, vol. 3410, pp. 520–533. Springer, Heidelberg (2005)Google Scholar
  65. 65.
    Schütze, O., Esquivel, X., Lara, A., Coello Coello, C.A.: Using the averaged Hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE Trans. Evol. Comput. 16(4), 504–522 (2012)CrossRefGoogle Scholar
  66. 66.
    Stützle, T.: ACOTSP: A software package of various ant colony optimization algorithms applied to the symmetric traveling salesman problem (2002). http://www.aco-metaheuristic.org/aco-code/
  67. 67.
    Stützle, T., Hoos, H.H.: \({\cal{M}}{\cal{A}}{\cal{X}}\)-\({\cal{M}}{\cal{I}}{\cal{N}}\) Ant System. Futur. Gener. Comput. Syst. 16(8), 889–914 (2000)CrossRefGoogle Scholar
  68. 68.
    Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In: Dhillon, I.S., Koren, Y., Ghani, R., Senator, T.E., Bradley, P., Parekh, R., He, J., Grossman, R.L., Uthurusamy, R. (eds.) The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 847–855. ACM Press, New York (2013)Google Scholar
  69. 69.
    Tušar, T., Filipič, B.: Differential evolution versus genetic algorithms in multiobjective optimization. In: Obayashi, S., et al. (eds.) Evolutionary Multi-criterion Optimization, EMO 2007. Lecture Notes in Computer Science, vol. 4403, pp. 257–271. Springer, Heidelberg (2007)Google Scholar
  70. 70.
    Van Veldhuizen, D.A., Lamont, G.B.: Multiobjective evolutionary algorithms: Analyzing the state-of-the-art. Evol. Comput. 8(2), 125–147 (2000)CrossRefGoogle Scholar
  71. 71.
    Wessing, S., Beume, N., Rudolph, G., Naujoks, B.: Parameter tuning boosts performance of variation operators in multiobjective optimization. In: Schaefer, R., Cotta, C., Kolodziej, J., Rudolph, G. (eds.) Parallel Problem Solving from Nature, PPSN XI. Lecture Notes in Computer Science, vol. 6238, pp. 728–737. Springer, Heidelberg (2010)Google Scholar
  72. 72.
    Zhang, T., Georgiopoulos, M., Anagnostopoulos, G.C.: S-Race: A multi-objective racing algorithm. In: Blum, C., Alba, E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2013, pp. 1565–1572. ACM Press, New York (2013)Google Scholar
  73. 73.
    Zhang, T., Georgiopoulos, M., Anagnostopoulos, G.C.: SPRINT: Multi-objective model racing. In: Silva, S., Esparcia-Alcázar, A.I. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2015, pp. 1383–1390. ACM Press, New York (2015)Google Scholar
  74. 74.
    Zhang, T., Georgiopoulos, M., Anagnostopoulos, G.C.: Multi-objective model selection via racing. IEEE Trans. Cybern. 46(8), 1863–1876 (2016)CrossRefGoogle Scholar
  75. 75.
    Zitzler, E., Thiele, L., Bader, J.: On set-based multiobjective optimization. IEEE Trans. Evol. Comput. 14(1), 58–79 (2010)CrossRefGoogle Scholar
  76. 76.
    Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C.M., Grunert da Fonseca, V.: Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans. Evol. Comput. 7(2), 117–132 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Leonardo C. T. Bezerra
    • 1
  • Manuel López-Ibáñez
    • 2
  • Thomas Stützle
    • 3
    Email author
  1. 1.Instituto Metrópole Digital (IMD)Universidade Federal do Rio Grande do Norte (UFRN)NatalBrazil
  2. 2.Alliance Manchester Business SchoolUniversity of ManchesterManchesterUK
  3. 3.IRIDIAUniversité Libre de Bruxelles (ULB)BrusselsBelgium

Personalised recommendations