Evolutionary Intelligence

, Volume 5, Issue 4, pp 213–230 | Cite as

Exploratory analysis of an on-line evolutionary algorithm in simulated robots

Research Paper


In traditional evolutionary robotics, robot controllers are evolved in a separate design phase preceding actual deployment; we call this off-line evolution. Alternatively, robot controllers can evolve while the robots perform their proper tasks, during the actual operational phase; we call this on-line evolution. In this paper we describe three principal categories of on-line evolution for developing robot controllers (encapsulated, distributed, and hybrid), present an evolutionary algorithm belonging to the first category (the (μ + 1) on-line algorithm), and perform an extensive study of its behaviour. In particular, we use the Bonesa parameter tuning method to explore its parameter space. This delivers near-optimal settings for our algorithm in a number of tasks and, even more importantly, it offers profound insights into the impact of our algorithm’s parameters and features. Our experimental analysis of (μ + 1) on-line shows that it seems preferable to try many alternative solutions and spend little effort on refining possibly faulty assessments; that there is no single combination of parameters that performs well on all problem instances and that the most influential parameter of this algorithm—and therefore the prime candidate for a control scheme—is the evaluation length τ.


Evolutionary robotics On-line evolution Scientific testing Parameter tuning Bonesa 



The authors gratefully acknowledge D.J. Christensen’s providing the code on which we based our own locomotion experiments. Giorgos Karafotias was very helpful in setting up the other experiments. The authors thank the reviewers for their extensive and insightful comments; this has helped us produce a much better paper.


  1. 1.
    Bäck T (1996) Evolutionary algorithms in theory and practice. Oxford University Press, Oxford, UKMATHGoogle Scholar
  2. 2.
    Baldassarre G, Nolfi S, Parisi D (2002) Evolving mobile robots able to display collective behaviours. Artif Life 9:255–267CrossRefGoogle Scholar
  3. 3.
    Bartz-Beielstein T, Lasarczyk C, Preuss M (2005) Sequential parameter optimization. In: Proceedings of the 2005 IEEE congress on evolutionary computation IEEE congress on evolutionary computation, vol 1. IEEE Press, Edinburgh, UK, pp 773–780. doi: 10.1109/CEC.2005.1554761
  4. 4.
    Beyer HG (2000) Evolutionary algorithms in noisy environments: theoretical issues and guidelines for practice. Comput Methods Appl Mech Eng 186(2–4):239–267. doi: 10.1016/S0045-7825(99)00386-2, http://www.sciencedirect.com/science/article/B6V29-40CRYKF-7/2/d4ea05ab01cc73ab36dbdf737c752439
  5. 5.
    Branke J, Schmidt C, Schmec H (2001) Efficient fitness estimation in noisy environments. In: Proceedings of genetic and evolutionary computation, pp 243–250Google Scholar
  6. 6.
    Bredeche N, Haasdijk E, Eiben A (2009) On-line, on-board evolution of robot controllers. In: Collet P, Monmarché N, Legrand P, Schoenauer M, Lutton E (eds) Artificial evolution, Springer, Lecture Notes in Computer Science, pp 110–121. http://few.vu.nl/∼ehaasdi/papers/EA2009selfadapt1robot.pdf
  7. 7.
    Brooks RA (1992) Artificial life and real robots. In: Varela F, Bourgine P (eds) Toward a practice of autonomous systems: proceedings of the 1st European conference on artificial life, MIT Press, Cambridge, MA, pp 3–10. http://people.csail.mit.edu/brooks/papers/real-robots.pdf
  8. 8.
    Christensen DJ, Spröwitz A, Ijspeert AJ (2010) Distributed online learning of central pattern generators in modular robots. In: Doncieux S, Girard B, Guillot A, Hallam J, Meyer JA, Mouret JB (eds) From animals to animats 11, Springer Berlin, Lecture Notes in Computer Science, vol 6226. pp 402–412Google Scholar
  9. 9.
    Eiben A, Smith J (2003) Introduction to evolutionary computing. Springer, BerlinMATHGoogle Scholar
  10. 10.
    Eiben AE, Smit SK (2011) Parameter tuning for configuring and analyzing evolutionary algorithms. Swarm Evol Comput 1(1):19–31CrossRefGoogle Scholar
  11. 11.
    El-Beltagy M, Nair P, Keane A (1999) Metamodeling techniques for evolutionary optimization of computationally expensive problems: Promises and limitations. In: Banzhaf W, Daida J, Eiben A, Garzon M, Honavar V, Jakiela M, Smith R (eds). Proceedings of the genetic and evolutionary computation conference (GECCO-1999). Morgan Kaufmann, San Francisco, pp 196–203Google Scholar
  12. 12.
    Floreano D, Schoeni N, Caprari G, Blynel J (2002) Evolutionary bits’n’spikes. In: Standish RK, Bedau MA, Abbass HA (eds) Artificial Life VIII : Proceedings of the eighth international conference on artificial life. MIT Press, Cambridge, MA, pp 335–344Google Scholar
  13. 13.
    Haasdijk E, Eiben AE, Karafotias G (2010) On-line evolution of robot controllers by an encapsulated evolution strategy. In: Proceedings of the 2010 IEEE congress on evolutionary computation, IEEE computational intelligence society, IEEE Press, Barcelona. http://www.few.vu.nl/∼ehaasdi/papers/cec-2010-MuPlusOne.pdf
  14. 14.
    Haasdijk E, ul Qayyum AA, Eiben A (2011) Racing to improve on-line, on-board evolutionary robotics. In: Krasnogor N (ed) Proceedings of the genetic and evolutionary computation conference (GECCO-2011), ACM, Dublin, Ireland, pp 187–194. http://www.cs.vu.nl/ci/pubs/2011/MuPlusOneAndRacing.pdf
  15. 15.
    Haralick R, Shapiro L (1992) Computer and robot vision, chap 7, vol 1, Addison-Wesley Publishing Company, Reading, MAGoogle Scholar
  16. 16.
    Haroun Mahdavi S, Bentley PJ (2006) Innately adaptive robotics through embodied evolution. Auton Robots 20(2):149–163. doi: 10.1007/s10514-006-5941-6 Google Scholar
  17. 17.
    Hart W, Belew R (1991) Optimizing an arbitrary function is hard for the genetic algorithm. In: Belew RK, Booker LB (eds) Proceedings of the fourth international conference on genetic algorithms (ICGA ’91), Morgan Kaufmann Publishers, Inc., San Mateo CA, pp 190–195. URL citeseer.ist.psu.edu/hart91optimizing.htmlGoogle Scholar
  18. 18.
    Hoeffding W (1963) Probability inequalities for sums of bounded random variables. J Am Stat Assoc 58(301):13–30MathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Hooker J (1995) Testing heuristics: we have it all wrong. J Heuristics 1:33–42. doi: 10.1007/BF02430364 Google Scholar
  20. 20.
    Jelasity M, Montresor A, Babaoglu O (2005) Gossip-based aggregation in large dynamic networks. ACM Trans Comput Syst 23:219–252. doi: 10.1145/1082469.1082470 Google Scholar
  21. 21.
    Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9(1):3–12CrossRefGoogle Scholar
  22. 22.
    Johnson DS (2002) A theoretician’s guide to the experimental analysis of algorithms. In: Goldwasser MH, Johnson DS, McGeoch CC (eds) Data structures, near neighbor searches, and methodology: fifth and sixth DIMACS implementation challenges. American Mathematical Society, Providence, RI, pp 215–250Google Scholar
  23. 23.
    Jones T (1995) Crossover, macromutation, and population-based search. In: Eshelman L (ed) Proceedings of the 6th international conference on genetic algorithms. Morgan Kaufmann, San Francisco, pp 73–80Google Scholar
  24. 24.
    Karafotias G, Haasdijk E, Eiben A (2011) An algorithm for distributed on-line, on-board evolutionary robotics. In: Krasnogor N (ed) pp 171–178. http://www.cs.vu.nl/ci/pubs/2011/Encapsulated-vs-Distributed.pdf
  25. 25.
    Krasnogor N (ed) (2011) Proceedings of the genetic and evolutionary computation conference (GECCO-2011), ACM, Dublin, IrelandGoogle Scholar
  26. 26.
    Martinoli A (1999) Swarm intelligence in autonomous collective robotics from tools to the analysis and synthesis of distributed control strategies. PhD thesis, Ecole Polytechnique Fédérale de LausanneGoogle Scholar
  27. 27.
    Nannen V, Eiben A (2007) Relevance estimation and value calibration of evolutionary algorithm parameters. In: Proceedings of the international joint conference on atrificial intelligence (IJCAI’07), international joint conferences on artificial intelligence. AAAI Press, Menlo Park, CA, pp 975–980Google Scholar
  28. 28.
    Nehmzow U (2002) Physically embedded genetic algorithm learning in multi-robot scenarios: The pega algorithm. In: Prince C, Demiris Y, Marom Y, Kozima H, Balkenius C (eds) Proceedings of the second international workshop on epigenetic robotics: modeling cognitive development in robotic systems, LUCS, Edinburgh, UK, no. 94 in Lund University Cognitive StudiesGoogle Scholar
  29. 29.
    Nelson AL, Barlow GJ, Doitsidis L (2009) Fitness functions in evolutionary robotics: a survey and analysis. Robot Auton Syst 57(4):345–370. doi: 10.1016/j.robot.2008.09.009, http://www.sciencedirect.com/science/article/B6V16-4TTMJV3-1/2/2549524d8e0f3982730659e49ad3fa75 Google Scholar
  30. 30.
    Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence, and technology of self-organizing machines. MIT Press, Cambridge, MAGoogle Scholar
  31. 31.
    Nordin P, Banzhaf W (1997) An on-line method to evolve behavior and to control a miniature robot in real time with genetic programming. Adapt Behav 5:107–140CrossRefGoogle Scholar
  32. 32.
    Ostermeier A, Gawelczyk A, Hansen N (1994) A derandomized approach to self adaptation of evolution strategies. Evol Comput 2(4):369–380. doi: 10.1162/evco.1994.2.4.369 Google Scholar
  33. 33.
    Pedersen MEH et al (2008) Parameter tuning versus adaptation: proof of principle study on differential evolution. Tech. rep., Hvass Laboratories. http://citeseer.ist.psu.edu/viewdoc/download?doi=
  34. 34.
    Schwefel HP (1995) Evolution and optimum seeking. Wiley, New YorkGoogle Scholar
  35. 35.
    Simões EDV, Dimond KR (2001) Embedding a distributed evolutionary system into population of autonomous mobile robots. In: Proceedings of the 2001 IEEE systems, man, and cybernetics conference. http://citeseer.ist.psu.edu/simoes01embedding.html
  36. 36.
    Smit S, Eiben A (2010) Parameter tuning of evolutionary algorithms: Generalist vs. specialist. In: Chio CD et al. (eds) Applications of evolutionary computation, Springer, Lecture Notes in Computer Science, vol 6024, pp 542–551. http://www.cs.vu.nl/∼gusz/papers/2010-EVOSTAR-GeneralistSpecialist.pdf
  37. 37.
    Smit S, Eiben A (2011) Multi-problem parameter tuning using bonesa. In: Hao JK, Legrand P, Collet P, Monmarché N, Lutton E, Schoenauer M (eds) Proceedings of artificial evolution, 10th international conference, evolution artificielle (EA 2011), Springer, Lecture Notes in Computer Science, pp 222–233Google Scholar
  38. 38.
    Tuci E, Quinn M, Harvey I (2002) Evolving fixed-weight networks for learning robots. In: CEC ’02: Proceedings of the evolutionary computation on 2002. CEC ’02. Proceedings of the 2002 congress, IEEE Computer Society, pp 1970–1975Google Scholar
  39. 39.
    Walker JH, Garrett SM, Wilson MS (2006) The balance between initial training and lifelong adaptation in evolving robot controllers. IEEE Trans Syst Man Cybern Part B 36(2):423–432CrossRefGoogle Scholar
  40. 40.
    Watson RA, Ficici SG, Pollack JB (2002) Embodied evolution: Distributing an evolutionary algorithm in a population of robots. Robot Auton Syst 39(1):1–18. http://eprints.ecs.soton.ac.uk/10620/
  41. 41.
    Wischmann S, Stamm K, Wörgötter F (2007) Embodied evolution and learning: the neglected timing of maturation. In: Almeidae Costa F (ed) Advances in artificial life: 9th European conference on artificial life, lecture notes in artificial intelligence, vol 4648. Springer, Lisbon, Portugal, pp 284–293Google Scholar
  42. 42.
    Zitzler E, Laumanns M, Thiele L (2001) SPEA2: Improving the strength pareto evolutionary algorithm. Tech. Rep. 103, Computer Engineering and Networks Laboratory (TIK), Swiss Federal Institute of Technology (ETH) Zürich, Gloriastrasse 35, CH-8092 Zürich, SwitzerlandGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Vrije UniversiteitAmsterdamThe Netherlands
  2. 2.TNOThe HagueThe Netherlands

Personalised recommendations