Advertisement

Evolutionary Intelligence

, Volume 1, Issue 1, pp 47–62 | Cite as

Neuroevolution: from architectures to learning

  • Dario Floreano
  • Peter Dürr
  • Claudio Mattiussi
Review Article

Abstract

Artificial neural networks (ANNs) are applied to many real-world problems, ranging from pattern classification to robot control. In order to design a neural network for a particular task, the choice of an architecture (including the choice of a neuron model), and the choice of a learning algorithm have to be addressed. Evolutionary search methods can provide an automatic solution to these problems. New insights in both neuroscience and evolutionary biology have led to the development of increasingly powerful neuroevolution techniques over the last decade. This paper gives an overview of the most prominent methods for evolving ANNs with a special focus on recent advances in the synthesis of learning architectures.

Keywords

Neural networks Evolution Learning 

Notes

Acknowledgements

This work was supported by the Swiss National Science Foundation, grant no. 200021-112060. Thanks to Daniel Marbach for the illustrations and the two anonymous reviewers for their helpful comments.

References

  1. 1.
    Ackley DH, Littman ML (1992) Interactions between learning and evolution. In: Langton C, Farmer J, Rasmussen S, Taylor C (eds) Artificial Life II: Proceedings volume of Santa Fe conference, vol XI. Addison Wesley, Redwood City, pp 487–510Google Scholar
  2. 2.
    Bailey CH, Giustetto M, Huang Y.-Y, Hawkins RD, Kandel ER (2000) Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nat Rev Neurosci 1(1):11–20CrossRefGoogle Scholar
  3. 3.
    Baldwin JM (1896) A new factor in evolution. Am Nat 30:441–451CrossRefGoogle Scholar
  4. 4.
    Banzhaf W, Nordin P, Keller RE, Francone FD (1998) Genetic programming—an introduction. In: On the automatic evolution of computer programs and its applications. Morgan Kaufmann, San FranciscoGoogle Scholar
  5. 5.
    Barto AG (1995) Adaptive critic and the basal ganglia. In: Houk JC, Davis JL, Beiser DG (eds) Models of information processing in the basal ganglia. MIT Press, Cambridge, pp 215–232Google Scholar
  6. 6.
    Baxter J (1992) The evolution of learning algorithms for artificial neural networks. In: Green D, Bossomaier T (eds) Complex Systems. IOS PressGoogle Scholar
  7. 7.
    Beer RD, Gallagher JC (1992) Evolving dynamical neural networks for adaptive behavior. Adapt Behav 1:91–122CrossRefGoogle Scholar
  8. 8.
    Belew RK, McInerney J, Schraudolph NN (1992) Evolving networks: using the genetic algorithm with connectionistic learning. In: Langton CG, Taylor C, Farmer JD, Rasmussen S (eds) Proceedings of the 2nd Conference on Artificial Life. Addison-Wesley, Reading, pp 511–548Google Scholar
  9. 9.
    Blynel J, Floreano D (2003) Exploring the T-maze: evolving learning-like robot behaviors using CTRNNs. In: Raidl Ge AE (ed) 2nd European workshop on evolutionary robotics (EvoRob’2003)Google Scholar
  10. 10.
    Bongard J (2002) Evolving modular genetic regulatory networks. In: Proceedings of the 2002 congress on evolutionary computation 2002, CEC ’02, vol 2, pp 1872–1877Google Scholar
  11. 11.
    Chalmers DJ (1990) The evolution of learning: an experiment in genetic connectionism. In: Touretzky DS, Elman JL, Sejnowski T, Hinton GE (eds) Proceedings of the 1990 connectionist models summer school. Morgan Kaufmann, San Mateo, pp 81–90Google Scholar
  12. 12.
    Chandra A, Yao X (2006) Ensemble learning using multi-objective evolutionary algorithms. J Math Model Algorithms 5(4):417–445zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Chellapilla K, Fogel D (2001) Evolving an expert checkers playing program without using humanexpertise. IEEE Trans Evol Comput 5(4):422–428CrossRefGoogle Scholar
  14. 14.
    Dasdan A, Oflazer K (1993) Genetic synthesis of unsupervised learning algorithms. In: Proceedings of the 2nd Turkish symposium on artificial intelligence and ANNs. Department of Computer Engineering and Information Science, Bilkent University, AnkaraGoogle Scholar
  15. 15.
    DiPaolo E (2003) Evolving spike-timing-dependent plasticity for single-trial learning in robots. Phil Trans R Soc Lond A 361:2299–2319CrossRefMathSciNetGoogle Scholar
  16. 16.
    Dürr P, Mattiussi C, Floreano D (2006) Neuroevolution with Analog Genetic Encoding. In: Parallel problem solving from nature—PPSN iX, vol 9. Springer, Berlin, pp 671–680Google Scholar
  17. 17.
    Federici D (2005) Evolving developing spiking neural networks. In: Proceedings of CEC 2005 IEEE congress on evolutionary computationGoogle Scholar
  18. 18.
    Fellous J-M, Linster C (1998) Computational models of neuromodulation. Neural Comput 10(4):771–805CrossRefGoogle Scholar
  19. 19.
    Floreano D, Mattiussi C (2001) Evolution of spiking neural controllers for autonomous vision-based robots. In: Gomi T (ed) Evolutionary robotics. From intelligent robotics to artificial life. Springer, TokyoGoogle Scholar
  20. 20.
    Floreano D, Mondada F (1996) Evolution of plastic neurocontrollers for situated agents. In: Maes P, Matarić M, Meyer J, Pollack J, Roitblat H, Wilson S (eds) From animals to animats IV: proceedings of the 4th international conference on simulation of adaptive behavior. MIT Press-Bradford Books, Cambridge, pp 402–410Google Scholar
  21. 21.
    Floreano D, Urzelai J (2000) Evolutionary robots with online self-organization and behavioral fitness. Neural Netw 13:431–443CrossRefGoogle Scholar
  22. 22.
    Floreano D, Urzelai J (2001) Evolution of plastic control networks. Autonom Robots 11(3):311–317zbMATHCrossRefGoogle Scholar
  23. 23.
    Fontanari JF, Meir R (1991) Evolving a learning algorithm for the binary perceptron. Network 2:353–359CrossRefGoogle Scholar
  24. 24.
    Funahashi K, Nakamura Y (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Netw 6(6):801–806CrossRefGoogle Scholar
  25. 25.
    Geard NL, Wiles J (2003) Structure and dynamics of a gene network model incorporating small RNAs. In: Proceedings of 2003 congress on evolutionary computation, pp 199–206Google Scholar
  26. 26.
    Gerstner W (1999) Spiking neurons. In: Maass W, Bishop CM (eds) Pulsed neural networks. MIT Press-Bradford Books, CambridgeGoogle Scholar
  27. 27.
    Gomez F, Miikkulainen R (1997) Incremental evolution of complex general behavior. Adapt Behav 5(3–4):317–342CrossRefGoogle Scholar
  28. 28.
    Gruau F (1995) Automatic definition of modular neural networks. Adapt Behav 3(2):151–183CrossRefGoogle Scholar
  29. 29.
    Gruau, F, Whitley, D, and Pyeatt, L (1996) A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza JR, Goldberg DE, Fogel DB, Riolo RL (eds) Genetic programming 1996: proceedings of the first annual conference. MIT Press, Stanford University, pp 81–89Google Scholar
  30. 30.
    Hansen N, Ostermeier A (2001) Completely derandomized self-adaptation in evolution strategies. Evol Comput 9(2):159–195CrossRefGoogle Scholar
  31. 31.
    Haykin, S (1999) Neural networks. a comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle RiverzbMATHGoogle Scholar
  32. 32.
    Hebb DO (1949) The organisation of behavior. Wiley, New YorkGoogle Scholar
  33. 33.
    Hinton GE, Nowlan SJ (1987) How learning can guide evolution. Complex Syst 1:495–502zbMATHGoogle Scholar
  34. 34.
    Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol (Lond) 108:500–544Google Scholar
  35. 35.
    Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12 (3):353–380CrossRefGoogle Scholar
  36. 36.
    Husbands P, Harvey I, Cliff D, Miller G (1994) The use of genetic algorithms for the development of sensorimotor control systems. In: Gaussier P, Nicoud J-D (eds) From perceptin to action. IEEE Press, Los AlamitosGoogle Scholar
  37. 37.
    Husbands P, Smith T, Jakobi N, O’Shea M (1998) Better living through chemistry: evolving gasnets for robot control. Connect Sci 10:185–210CrossRefGoogle Scholar
  38. 38.
    Igel, C (2003) Neuroevolution for reinforcement learning using evolution strategies. In: Sarker R, et al (eds) Congress on evolutionary computation, vol 4. IEEE Press, New York, pp 2588–2595Google Scholar
  39. 39.
    Katz PS (1999) What are we talking about? Modes of neuronal communication. In: Katz P (eds) Beyond neurotransmission: neuromodulation and its importance for information processing, chap 1. Oxford University Press, Oxford, pp 1–28Google Scholar
  40. 40.
    Kitano H (1990) Designing neural networks by genetic algorithms using graph generation system. Complex Syst J 4:461–476zbMATHGoogle Scholar
  41. 41.
    Korkin M, Nawa NE, de Garis H (1998) A ’spike interval information coding’ representation for ATR’s CAM-brain machine (CBM) In: Proceedings of the 2nd international conference on evolvable systems: from biology to hardware (ICES’98). Springer, HeidelbergGoogle Scholar
  42. 42.
    Koza JR (1994) Genetic programming II: automatic discovery of reusable programs. MIT Press, CambridgezbMATHGoogle Scholar
  43. 43.
    Magg S, Philippides A (2006) Gasnets and CTRNNs : a comparison in terms of evolvability. In: From animals to animats 9: proceedings of the 9th international conference on simulation of adaptive behavior. Springer, Heidelberg, pp 461–472Google Scholar
  44. 44.
    Mattiussi C, Floreano D (2004) Evolution of analog networks using local string alignment on highly reorganizable genomes. In: Zebulum RS et al (eds) NASA/DoD conference on evolvable hardware (EH’2004), pp 30–37Google Scholar
  45. 45.
    Mattiussi C, Dürr P, Floreano D (2007a) Center of mass encoding: a self-adaptive representation with adjustable redundancy for real-valued parameters. In: GECCO 2007. ACM Press, New York, pp 1304–1311CrossRefGoogle Scholar
  46. 46.
    Mattiussi C, Marbach D, Dürr P, Floreano D (2007b) The age of analog networks. AI Magazine (in press)Google Scholar
  47. 47.
    Mayley G (1996) Landscapes, learning costs and genetic assimilation. Evol Comput 4(3):213–234Google Scholar
  48. 48.
    McHale G, Husbands P (2004) Gasnets and other evolvable neural networks applied to bipedal locomotion. In: Schaal S (ed) Proceedings from animals to animats 8: proceedings of the 8th international conference on simulation of adaptive behaviour (SAB’2004). MIT Press, Cambridge, pp 163–172Google Scholar
  49. 49.
    Mizutani E, Dreyfus SE (1998) Totally model-free reinforcement learning by actor-critic elman networks in non-markovian domains. In: Proceedings of the IEEE world congress on computational intelligence. IEEE Press, New YorkGoogle Scholar
  50. 50.
    Montague P, Dayan P, Sejnowski T (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16(5):1936–1947Google Scholar
  51. 51.
    Montana D, Davis L (1989) Training feed forward neural networks using genetic algorithms. In: Proceedings of the 11th international joint conference on artificial intelligence. Morgan Kaufmann, San Mateo, pp 529–538Google Scholar
  52. 52.
    Moriarty DE, Miikkulainen R (1996) Efficient reinforcement learning through symbiotic evolution. Machine Learn 22:11–32Google Scholar
  53. 53.
    Moscato P (1989) On evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. In: Technical report C3P 826, PasadenaGoogle Scholar
  54. 54.
    Niv Y, Joel D, Meilijson I, Ruppin E (2002) Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adapt Behav 10(1):5–24CrossRefGoogle Scholar
  55. 55.
    Nolfi S, Floreano D (1999) Learning and evolution. Auton Robots 7(1):89–113CrossRefGoogle Scholar
  56. 56.
    Nolfi S, Parisi D (1996) Learning to adapt to changing environments in evolving neural networks. Adapt Behav 5(1):75–98CrossRefGoogle Scholar
  57. 57.
    Nolfi S, Miglino O, Parisi D (1994) Phenotypic plasticity in evolving neural networks. In: Gaussier P, Nicoud J-D (eds) From perception to action. IEEE Press, Los AlamitosGoogle Scholar
  58. 58.
    Pfeifer R, Scheier C (1999) Understanding Intelligence. MIT Press, CambridgeGoogle Scholar
  59. 59.
    Purves D (1994) Neural activity in the growth of the brain. Cambridge University Press, CambridgeGoogle Scholar
  60. 60.
    Quartz S, Sejnowski TJ (1997) The neural basis of cognitive development: a constructivist manifesto. Behav Brain Sci 4:537–555CrossRefGoogle Scholar
  61. 61.
    Radcliffe NJ (1991) Form an analysis and random respectful recombination. In: Belew RK, Booker LB (eds) Proceedings of the 4th international conference on genetic algorithms. Morgan Kaufmann, San MateoGoogle Scholar
  62. 62.
    Rechenberg I (1973) Evolutionsstrategie—Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Fommann-Holzboog, StuttgartGoogle Scholar
  63. 63.
    Reil T (1999) Dynamics of gene expression in an artificial genome—implications for biological and artificial ontogeny. In: Proceedings of the 5th European conference on artificial life, pp 457–466Google Scholar
  64. 64.
    Reil T (2003) On growth, form and computers. In: Artificial genomes as models of gene regulation. Academic Press, London, pp 256–277Google Scholar
  65. 65.
    Reisinger J, Miikkulainen R (2007) Acquiring evolvability through adaptive representations. In: Proceedings of genetic and evolutionary computation conference (GECCO 2007)Google Scholar
  66. 66.
    Reisinger J, Bahçeci E, Karpov I, Miikkulainen R (2007) Coevolving strategies for general game playing. In: Proceedings of the IEEE symposium on computational intelligence and games (CIG-2007)Google Scholar
  67. 67.
    Rieke F, Warland D, van Steveninck R, Bialek W (1997) Spikes. Exploring the neural code. MIT Press, CambridgeGoogle Scholar
  68. 68.
    Rumelhart DE, Hinton GE, Williams RJ (1986a) Learning representations by back-propagation of errors. Nature 323:533–536CrossRefGoogle Scholar
  69. 69.
    Rumelhart DE, McClelland J, the PDP Research Group (1986b) Parallel distributed processing: explorations in the microstructure of cognition. Foundations, vol 1. MIT Press-Bradford Books, CambridgeGoogle Scholar
  70. 70.
    Saggie K, Keinan A, Ruppin E (2004) Spikes that count: rethinking spikiness in neurally embedded systems. Neurocomputing 58-60:303–311CrossRefGoogle Scholar
  71. 71.
    Sasaki T, Tokoro M (1997) Adaptation toward changing environments: Why Darwinian in nature?. In: Husbands P, Harvey I (eds) Proceedings of the 4th European conference on artificial life. MIT Press, CambridgeGoogle Scholar
  72. 72.
    Schaffer JD, Whitley D, Eshelman LJ (1992) Combinations of genetic algorithms and neural networks: a survey of the state of the art. In: Whitley D, Schaffer JD (eds) Proceedings of an international workshop on the combinations of genetic algorithms and neural networks (COGANN-92). IEEE Press, New YorkGoogle Scholar
  73. 73.
    Schraudolph NN, Belew RK (1992) Dynamic parameter encoding for genetic algorithms. Machine Learn 9:9–21Google Scholar
  74. 74.
    Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593–1599CrossRefGoogle Scholar
  75. 75.
    Shapiro J (2005) A 21st century view of evolution: genome system architecture, repetitive DNA, and natural genetic engineering. Gene 345(1):91–100CrossRefGoogle Scholar
  76. 76.
    Siddiqi A, Lucas S (1998) A comparison of matrix rewriting versus direct encoding for evolving neural networks. In: Proceedings of the 1998 IEEE international conference on evolutionary computation. Piscataway, NJ, pp 392–397Google Scholar
  77. 77.
    Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586CrossRefGoogle Scholar
  78. 78.
    Soltoggio A, Duerr P, Mattiussi C, Floreano D (2007) Evolving neuromodulatory topologies for reinforcement learning-like problems. In: Angeline P, Michaelewicz M, Schonauer G, Yao X, Zalzala Z (eds) Proceedings of the 2007 congress on evolutionary computation. IEEE Press, New YorkGoogle Scholar
  79. 79.
    Stanley K, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127CrossRefGoogle Scholar
  80. 80.
    Stanley KO, Miikkulainen R (2004) Competitive coevolution through evolutionary complexification. J Artif Intell Res 21:63–100Google Scholar
  81. 81.
    Stanley K, Kohl N, Sherony R, Miikkulainen R (2005a) Neuroevolution of an automobile crash warning system. In: Proceedings of genetic and evolutionary computation conference (GECCO 2005)Google Scholar
  82. 82.
    Stanley KO, Cornelius R, Miikkulainen R, D’Silva T, Gold A (2005b) Real-time learning in the nero video game. In: Proceedings of the artificial intelligence and interactive digital entertainment conference (AIIDE 2005) demo papersGoogle Scholar
  83. 83.
    Sutton RS (1988) Learning to predict by the method of temporal difference. Machine Learn 3:9–44Google Scholar
  84. 84.
    Sutton RS, Barto AG (1998) Reinforcement learning. an introduction. MIT Press, CambridgeGoogle Scholar
  85. 85.
    Trianni V, Ampatzis C, Christensen A, Tuci E, Dorigo M, Nolfi S (2007) From solitary to collective behaviours: decision making and cooperation. In: Advances in artificial life, proceedings of ECAL 2007. Lecture Notes in Artificial Intelligence, vol LNAI 4648. Springer, Berlin, pp 575–584Google Scholar
  86. 86.
    Tuci E, Quinn M, Harvey I (2002) An evolutionary ecological approach to the study of learning behavior using a robot-based model. Adapt Behav 10(3–4):201–221CrossRefGoogle Scholar
  87. 87.
    Urzelai J, Floreano D (2001) Evolution of adaptive synapses: robots with fast adaptive behavior in new environments. Evol Comput 9:495–524CrossRefGoogle Scholar
  88. 88.
    Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14:347–361CrossRefGoogle Scholar
  89. 89.
    Widrow B, Hoff ME (1960) Adaptive switching circuits. In: Proceedings of the 1960 IRE WESCON convention, vol IV, New York. IRE. Reprinted in Anderson and Rosenfeld, 1988, pp 96–104Google Scholar
  90. 90.
    Yamauchi BM, Beer RD (1994) Sequential behavior and learning in evolved dynamical neural networks. Adapt Behav 2(3):219–246CrossRefGoogle Scholar
  91. 91.
    Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  • Dario Floreano
    • 1
  • Peter Dürr
    • 1
  • Claudio Mattiussi
    • 1
  1. 1.Ecole Polytechnique Fédérale de Lausanne Laboratory of Intelligent SystemsLausanneSwitzerland

Personalised recommendations