A Framework for the Empirical Analysis of Genetic Programming System Performance

  • Oliver FlaschEmail author
  • Thomas Bartz-Beielstein
Part of the Genetic and Evolutionary Computation book series (GEVO)


This chapter introduces a framework for statistically sound, reproducible empirical research in Genetic Programming (GP). It provides tools to understand GP algorithms and heuristics and their interaction with problems of varying difficulty. Following an approach where scientific claims are broken down to testable statistical hypotheses and GP runs are treated as experiments, the framework helps to achieve statistically verified results of high reproducibility.

Key words

Genetic programming Symbolic regression Design of experiments Sequential parameter optimization Reproducible research Multi-objective optimization 



This work was supported by the Bundesministerium für Bildung und Forschung (BMBF) under the grant FIWA (AiF FKZ 17N2309). Many thanks to Boris Naujoks, Tobias Brandt, and Jörg Stork for valuable ideas and suggestions.


  1. Bartz-Beielstein T (2006) Experimental Research in Evolutionary Computation—The New Experimentalism. Natural Computing Series, Springer, Berlin, Heidelberg, New YorkzbMATHGoogle Scholar
  2. Bartz-Beielstein T (2010) SPOT: An R package for automatic and interactive tuning of optimization algorithms by sequential parameter optimization. CIOP Technical Report 05/10, Research Center CIOP (Computational Intelligence, Optimization and Data Mining), Cologne University of Applied Science, URL
  3. Deb K, Agrawal S, Pratab A, Meyarivan T (2000) A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II. In: Schoenauer M, Deb K, Rudolph G, Yao X, Lutton E, Merelo JJ, Schwefel HP (eds) Proceedings of the Parallel Problem Solving from Nature VI, Springer, Berlin, Heidelberg, New York, pp 849–858CrossRefGoogle Scholar
  4. Flasch O, Mersmann O, Bartz-Beielstein T (2010) RGP: An open source genetic programming system for the R environment. In: Pelikan M, Branke J (eds) Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings, Portland, Oregon, ACM, pp 2071–2072Google Scholar
  5. Kieseppa IA (1997) Akaike information criterion, curve-fitting and the philosophical problem of simplicity. British Journal for the Philosophy of Science 48:21–48MathSciNetCrossRefGoogle Scholar
  6. Korns MF (2011) Accuracy in symbolic regression. In: Riolo R, Vladislavleva E, Moore JH (eds) Genetic Programming Theory and Practice IX, Genetic and Evolutionary Computation, Springer, Ann Arbor, USA, chap 8, pp 129–151, DOI doi: 10.1007/978-1-4614-1770-5-8
  7. Kotanchek M, Smits G, Vladislavleva E (2006) Pursuing the pareto paradigm tournaments, algorithm variations & ordinal optimization. In: Riolo RL, Soule T, Worzel B (eds) Genetic Programming Theory and Practice IV, Genetic and Evolutionary Computation, vol 5, Springer, Ann Arbor, chap 12, pp 167–186Google Scholar
  8. Kotanchek M, Smits G, Vladislavleva E (2007) Trustable symoblic regression models. In: Riolo RL, Soule T, Worzel e Bill (eds) Genetic Programming Theory and Practice V, pp 203–222Google Scholar
  9. Koza JR (1992) A genetic approach to the truck backer upper problem and the inter-twined spiral problem. In: Proceedings of IJCNN International Joint Conference on Neural Networks, IEEE Press, vol IV, pp 310–318, URL
  10. Parkes AJ, Walser JP (1996) Tuning local search for satisfiability testing. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI’96), pp 356–362, URL
  11. Poli R (2004) TinyGP. See TinyGP GECCO 2004 competition at
  12. R Development Core Team (2011) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, URL, ISBN 3-900051-07-0
  13. Schmidt MD, Lipson H (2010) Age-fitness pareto optimization. In: Proceedings of the 12th annual conference on Genetic and evolutionary computation, ACM, New York, NY, USA, GECCO ’10, pp 543–544Google Scholar
  14. Smits G, Vladislavleva E (2006) Ordinal pareto genetic programming. In: Yen GG, et al (eds) Proceedings of the 2006 IEEE Congress on Evolutionary Computation, IEEE Press, Vancouver, BC, Canada, pp 3114–3120, URL
  15. Vladislavleva E (2008) Model–based problem solving through symbolic regression via Pareto genetic programming. PhD thesis, Tilburg UniversityGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Cologne University of Applied SciencesCologneGermany

Personalised recommendations