Multi-objective Performance Measurement: Alternatives to PAR10 and Expected Running Time

  • Jakob BossekEmail author
  • Heike Trautmann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11353)


A multiobjective perspective onto common performance measures such as the PAR10 score or the expected runtime of single-objective stochastic solvers is presented by directly investigating the tradeoff between the fraction of failed runs and the average runtime. Multi-objective indicators operating in the bi-objective space allow for an overall performance comparison on a set of instances paving the way for instance-based automated algorithm selection techniques.


Algorithm selection Performance measurement 



The authors acknowledge support from the European Research Center for Information Systems (ERCIS) and the DAAD PPP project No. 57314626.


  1. 1.
    Bischl, B. et al.: ASlib: a benchmark library for algorithm selection. Artif. Intell. J. 237, 41–58 (2016).
  2. 2.
    Blot, A., Hoos, H., Jourdan, L., Marmion, M., Trautmann, H.: In: Joaquin, V. et al. (ed.) MO-ParamILS: A multi-objective automatic algorithm configuration framework, pp. 32–47. Springer International Publishing, Ischia (2016)Google Scholar
  3. 3.
    Coello Coello, C., Lamont, G.B., van Veldhuizen, D.: Evolutionary Algorithms for Solving Multi-objective Problems. Springer, Berlin (2007)zbMATHGoogle Scholar
  4. 4.
    Hansen, N., Auger, A., Finck, S., Ros, R.: Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup. Technical Report RR-6828, INRIA (2009).
  5. 5.
    Kerschke, P., Kotthoff, L., Bossek, J., Hoos, H.H., Trautmann, H.: Leveraging TSP solver complementarity through machine learning. Evol. Comput. 0(0), 1–24 (2017)., pMID: 28836836
  6. 6.
    Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Trans. Evol. Comput. 3(4), 257–271 (1999)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Information Systems and StatisticsUniversity of MünsterMünsterGermany

Personalised recommendations