How to Create Generalizable Results


Basically, this chapter tries to find answers for the following fundamental questions in experimental research.
  1. (Q-1)

    How can problem instances be generated?

  2. (Q-2)

    How can experimental results be generalized?


The chapter is structured as follows. Section 56.2 introduces real-world and artificial optimization problems. Algorithms are described in Sect. 56.3. Objective functions and statistical models are introduced in Sect. 56.4; these models take problem and algorithm features into consideration. Section 56.5 presents case studies that illustrate our methodology. The chapter closes with a summary and an outlook.


Covariance Recombination 

analysis of variance


computational intelligence


evolutionary algorithm


evolution strategy


foreign exchange


independent, identically distributed


multiple algorithms, multiple problems


multiple algorithms and multiple problem instances


multiple algorithms and one single problem


max-set of Gaussian landscape generator




restricted maximum likelihood estimator


one single algorithm and multiple problems


one single algorithm and one single problem


sequential parameter optimization toolbox


United States Environmental Protection Agency


  1. [56.1]
    P.J. Brockwell, R.A. Davis: Introduction to Time Series and Forecasting (Springer, New York 2002)CrossRefMATHGoogle Scholar
  2. [56.2]
    D.C. Montgomery: Design and Analysis of Experiments (Wiley, New York 2001)Google Scholar
  3. [56.3]
    M. Chiarandini, Y. Goegebeur: Mixed models for the analysis of optimization algorithms. In: Experimental Methods for the Analysis of Optimization Algorithms, ed. by T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss (Springer, Berlin Heidelberg 2010) pp. 225–264, preliminary version available as Tech. Rep. DMF-2009-07-001 at the The Danish Mathematical SocietyCrossRefGoogle Scholar
  4. [56.4]
    C.C. McGeoch: Toward an experimental method for algorithm simulation, INFORMS J. Comput. 8(1), 1–15 (1996)MathSciNetCrossRefMATHGoogle Scholar
  5. [56.5]
    M. Birattari: On the Estimation of the Expected Performance of a Metaheuristic on a Class of Instances 2004), Tech. Rep. (IRIDIA, Bruxelles 2004)Google Scholar
  6. [56.6]
    M. Gallagher, B. Yuan: A general-purpose tunable landscape generator, IEEE Trans. Evol. Comput. 10(5), 590–603 (2006)CrossRefGoogle Scholar
  7. [56.7]
    I. Castiñeiras, M.D. Cauwer, B. O'Sullivan: Weibull-based benchmarks for bin packing, Lect. Notes Comput. Sci. 7514, 207–222 (2012)CrossRefGoogle Scholar
  8. [56.8]
    G.E.P. Box, G.M. Jenkins, G.C. Reinsel: Time Series Analysis, Forecasting and Control (Holden-Day, San Francisco 1976)MATHGoogle Scholar
  9. [56.9]
    T. Bartz-Beielstein: Beyond Particular Problem Instances: How to Create Meaningful and Generalizable Results, Tech. Rep. TR 03/2012 (Cologne University of Applied Sciences, 2012)Google Scholar
  10. [56.10]
    T. Bartz-Beielstein, M. Friese, B. Naujoks, M. Zaefferer: SPOT applied to non-stochastic optimization problems – An experimental study, Genet. Evol. Comput. Conf. (GECCO 2012) (ACM, Philadelphia 2012)Google Scholar
  11. [56.11]
    M. Zaefferer: Optimization and Empirical Analysis of an Event Detection Software for Water Quality Monitoring, Master Thesis (University of Applied Sciences, Cologne 2012)Google Scholar
  12. [56.12]
    O. Flasch, T. Bartz-Beielstein, D. Bicker, W. Kantschik, C. von Strachwitz: Results of the GECCO 2011 Industrial Challenge: Optimizing Foreign Exchange Trading Strategies. CIOP Tech. Rep. 10/11, Res. Center CIOP (Cologne University of Applied Science, Cologne 2011) Google Scholar
  13. [56.13]
    E. Vladislavleva: Model–based Problem Solving through Symbolic Regression via Pareto Genetic Programming, Ph.D. Thesis (Tilburg University, Tilburg 2008)Google Scholar
  14. [56.14]
    H.-G. Beyer, H.-P. Schwefel: Evolution strategies – A comprehensive introduction, Nat. Comput. 1, 3–52 (2002)MathSciNetCrossRefMATHGoogle Scholar
  15. [56.15]
    H.-P. Schwefel: Numerical Optimization of Computer Models (Wiley, Chichester 1981)MATHGoogle Scholar
  16. [56.16]
    T. Bartz-Beielstein: Experimental Research in Evolutionary Computation – The New Experimentalism, Natural Computing Series (Springer, Berlin, Heidelberg, New York 2006)MATHGoogle Scholar
  17. [56.17]
    T. Bartz-Beielstein, M. Preuss: Automatic and interactive tuning of algorithms, Proc. 13th Annu. Conf. Companion Genet. Evol. Comput. (GECCO) (ACM, New York 2011) pp. 1361–1380Google Scholar
  18. [56.18]
    J. Pinheiro, D. Bates: Mixed-Effects Models in S and S-PLUS (Springer, New York 2000)CrossRefMATHGoogle Scholar
  19. [56.19]
    T. Bartz-Beielstein, M. Zaefferer: A Gentle Introduction to Sequential Parameter Optimization, Tech. Rep. TR 01/2012 CIplus Bd. 1/2012 (Cologne University of Applied Sciences, Cologne 2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Faculty of Computer Science and Engineering ScienceCologne University of Applied SciencesGummersbachGermany

Personalised recommendations