Experimental Methods for the Analysis of Optimization Algorithms

  • Thomas Bartz-Beielstein
  • Marco Chiarandini
  • Luís Paquete
  • Mike Preuss

Table of contents

  1. Front Matter
    Pages i-xxii
  2. Thomas Bartz-Beielstein, Marco Chiarandini, Luís Paquete, Mike Preuss
    Pages 1-13
  3. Overview

    1. Front Matter
      Pages 15-15
    2. Thomas Bartz-Beielstein, Mike Preuss
      Pages 17-49
    3. Markus Chimani, Karsten Klein
      Pages 131-158
  4. Characterizing Algorithm Performance

    1. Front Matter
      Pages 159-159
    2. Matteo Gagliolo, Catherine Legrand
      Pages 161-184
    3. Manuel López-Ibáñez, Luís Paquete, Thomas Stützle
      Pages 209-222
  5. Algorithm Configuration and Tuning

    1. Front Matter
      Pages 223-223
    2. Marco Chiarandini, Yuri Goegebeur
      Pages 225-264
    3. Enda Ridge, Daniel Kudenko
      Pages 265-286
    4. Selmar K. Smit, Agoston E. Eiben
      Pages 287-310
    5. Mauro Birattari, Zhi Yuan, Prasanna Balaprakash, Thomas Stützle
      Pages 311-336
    6. Thomas Bartz-Beielstein, Christian Lasarczyk, Mike Preuss
      Pages 337-362
    7. Frank Hutter, Thomas Bartz-Beielstein, Holger H. Hoos, Kevin Leyton-Brown, Kevin P. Murphy
      Pages 363-414
  6. Back Matter
    Pages 415-457

About this book


In operations research and computer science it is common practice to evaluate the performance of optimization algorithms on the basis of computational results, and the experimental approach should follow accepted principles that guarantee the reliability and reproducibility of results. However, computational experiments differ from those in other sciences, and the last decade has seen considerable methodological research devoted to understanding the particular features of such experiments and assessing the related statistical methods.

This book consists of methodological contributions on different scenarios of experimental analysis. The first part overviews the main issues in the experimental analysis of algorithms, and discusses the experimental cycle of algorithm development; the second part treats the characterization by means of statistical distributions of algorithm performance in terms of solution quality, runtime and other measures; and the third part collects advanced methods from experimental design for configuring and tuning algorithms on a specific class of instances with the goal of using the least amount of experimentation. The contributor list includes leading scientists in algorithm design, statistical design, optimization and heuristics, and most chapters provide theoretical background and are enriched with case studies.

This book is written for researchers and practitioners in operations research and computer science who wish to improve the experimental assessment of optimization algorithms and, consequently, their design.


Algorithm engineering Algorithms Combinatorial optimization Evolutionary algorithms Experiment designs Experimental analysis Extreme value theory F-race Heuristics Inferential statistics Local search Multiobjective optimization Optimization Scheduling Sequential parameter optimization (SPO)

Editors and affiliations

  • Thomas Bartz-Beielstein
    • 1
  • Marco Chiarandini
    • 2
  • Luís Paquete
    • 3
  • Mike Preuss
    • 4
  1. 1.Institute of Computer Science, Faculty of Computer ScienceCologne University of Applied SciencesGummersbachGermany
  2. 2., Department of MathematicsUniversity of Southern DenmarkOdenseDenmark
  3. 3.CISUC, Department of Informatics EngineeringUniversity of CoimbraCoimbraPortugal
  4. 4.Algorithm EngineeringTU DortmundDortmundGermany

Bibliographic information