Skip to main content
SpringerLink
Log in

Neuroevolution: from architectures to learning

  • Review Article
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

Artificial neural networks (ANNs) are applied to many real-world problems, ranging from pattern classification to robot control. In order to design a neural network for a particular task, the choice of an architecture (including the choice of a neuron model), and the choice of a learning algorithm have to be addressed. Evolutionary search methods can provide an automatic solution to these problems. New insights in both neuroscience and evolutionary biology have led to the development of increasingly powerful neuroevolution techniques over the last decade. This paper gives an overview of the most prominent methods for evolving ANNs with a special focus on recent advances in the synthesis of learning architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. Analog networks are collections of dynamical devices interconnected by links of varying strength. For example, genetic regulatory networks, metabolic networks, neural networks, or electronic circuits can be seen as analog networks.

  2. Algorithms which combine evolutionary search with some kinds of local search are sometimes called memetic algorithms [53].

  3. The two spaces are correlated if genotypes which are close in the evolutionary space correspond to phenotypes which are also close in the phenotype space.

  4. An alternative approach to this are neural learning classifier systems. For example, Hurst and Bull [35] addressed the control of a simulated robot in a maze task. They used a population of neural networks acting as ‘rules’ controlling the robot. As evolution favored rules that led to succesful behavior, the set of rules adapted to the requirements of the task.

References

  1. Ackley DH, Littman ML (1992) Interactions between learning and evolution. In: Langton C, Farmer J, Rasmussen S, Taylor C (eds) Artificial Life II: Proceedings volume of Santa Fe conference, vol XI. Addison Wesley, Redwood City, pp 487–510

  2. Bailey CH, Giustetto M, Huang Y.-Y, Hawkins RD, Kandel ER (2000) Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nat Rev Neurosci 1(1):11–20

    Article  Google Scholar 

  3. Baldwin JM (1896) A new factor in evolution. Am Nat 30:441–451

    Article  Google Scholar 

  4. Banzhaf W, Nordin P, Keller RE, Francone FD (1998) Genetic programming—an introduction. In: On the automatic evolution of computer programs and its applications. Morgan Kaufmann, San Francisco

  5. Barto AG (1995) Adaptive critic and the basal ganglia. In: Houk JC, Davis JL, Beiser DG (eds) Models of information processing in the basal ganglia. MIT Press, Cambridge, pp 215–232

  6. Baxter J (1992) The evolution of learning algorithms for artificial neural networks. In: Green D, Bossomaier T (eds) Complex Systems. IOS Press

  7. Beer RD, Gallagher JC (1992) Evolving dynamical neural networks for adaptive behavior. Adapt Behav 1:91–122

    Article  Google Scholar 

  8. Belew RK, McInerney J, Schraudolph NN (1992) Evolving networks: using the genetic algorithm with connectionistic learning. In: Langton CG, Taylor C, Farmer JD, Rasmussen S (eds) Proceedings of the 2nd Conference on Artificial Life. Addison-Wesley, Reading, pp 511–548

  9. Blynel J, Floreano D (2003) Exploring the T-maze: evolving learning-like robot behaviors using CTRNNs. In: Raidl Ge AE (ed) 2nd European workshop on evolutionary robotics (EvoRob’2003)

  10. Bongard J (2002) Evolving modular genetic regulatory networks. In: Proceedings of the 2002 congress on evolutionary computation 2002, CEC ’02, vol 2, pp 1872–1877

  11. Chalmers DJ (1990) The evolution of learning: an experiment in genetic connectionism. In: Touretzky DS, Elman JL, Sejnowski T, Hinton GE (eds) Proceedings of the 1990 connectionist models summer school. Morgan Kaufmann, San Mateo, pp 81–90

  12. Chandra A, Yao X (2006) Ensemble learning using multi-objective evolutionary algorithms. J Math Model Algorithms 5(4):417–445

    Article  MATH  MathSciNet  Google Scholar 

  13. Chellapilla K, Fogel D (2001) Evolving an expert checkers playing program without using humanexpertise. IEEE Trans Evol Comput 5(4):422–428

    Article  Google Scholar 

  14. Dasdan A, Oflazer K (1993) Genetic synthesis of unsupervised learning algorithms. In: Proceedings of the 2nd Turkish symposium on artificial intelligence and ANNs. Department of Computer Engineering and Information Science, Bilkent University, Ankara

  15. DiPaolo E (2003) Evolving spike-timing-dependent plasticity for single-trial learning in robots. Phil Trans R Soc Lond A 361:2299–2319

    Article  MathSciNet  Google Scholar 

  16. Dürr P, Mattiussi C, Floreano D (2006) Neuroevolution with Analog Genetic Encoding. In: Parallel problem solving from nature—PPSN iX, vol 9. Springer, Berlin, pp 671–680

  17. Federici D (2005) Evolving developing spiking neural networks. In: Proceedings of CEC 2005 IEEE congress on evolutionary computation

  18. Fellous J-M, Linster C (1998) Computational models of neuromodulation. Neural Comput 10(4):771–805

    Article  Google Scholar 

  19. Floreano D, Mattiussi C (2001) Evolution of spiking neural controllers for autonomous vision-based robots. In: Gomi T (ed) Evolutionary robotics. From intelligent robotics to artificial life. Springer, Tokyo

    Google Scholar 

  20. Floreano D, Mondada F (1996) Evolution of plastic neurocontrollers for situated agents. In: Maes P, Matarić M, Meyer J, Pollack J, Roitblat H, Wilson S (eds) From animals to animats IV: proceedings of the 4th international conference on simulation of adaptive behavior. MIT Press-Bradford Books, Cambridge, pp 402–410

  21. Floreano D, Urzelai J (2000) Evolutionary robots with online self-organization and behavioral fitness. Neural Netw 13:431–443

    Article  Google Scholar 

  22. Floreano D, Urzelai J (2001) Evolution of plastic control networks. Autonom Robots 11(3):311–317

    Article  MATH  Google Scholar 

  23. Fontanari JF, Meir R (1991) Evolving a learning algorithm for the binary perceptron. Network 2:353–359

    Article  Google Scholar 

  24. Funahashi K, Nakamura Y (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Netw 6(6):801–806

    Article  Google Scholar 

  25. Geard NL, Wiles J (2003) Structure and dynamics of a gene network model incorporating small RNAs. In: Proceedings of 2003 congress on evolutionary computation, pp 199–206

  26. Gerstner W (1999) Spiking neurons. In: Maass W, Bishop CM (eds) Pulsed neural networks. MIT Press-Bradford Books, Cambridge

  27. Gomez F, Miikkulainen R (1997) Incremental evolution of complex general behavior. Adapt Behav 5(3–4):317–342

    Article  Google Scholar 

  28. Gruau F (1995) Automatic definition of modular neural networks. Adapt Behav 3(2):151–183

    Article  Google Scholar 

  29. Gruau, F, Whitley, D, and Pyeatt, L (1996) A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza JR, Goldberg DE, Fogel DB, Riolo RL (eds) Genetic programming 1996: proceedings of the first annual conference. MIT Press, Stanford University, pp 81–89

  30. Hansen N, Ostermeier A (2001) Completely derandomized self-adaptation in evolution strategies. Evol Comput 9(2):159–195

    Article  Google Scholar 

  31. Haykin, S (1999) Neural networks. a comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle River

    MATH  Google Scholar 

  32. Hebb DO (1949) The organisation of behavior. Wiley, New York

    Google Scholar 

  33. Hinton GE, Nowlan SJ (1987) How learning can guide evolution. Complex Syst 1:495–502

    MATH  Google Scholar 

  34. Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol (Lond) 108:500–544

    Google Scholar 

  35. Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12 (3):353–380

    Article  Google Scholar 

  36. Husbands P, Harvey I, Cliff D, Miller G (1994) The use of genetic algorithms for the development of sensorimotor control systems. In: Gaussier P, Nicoud J-D (eds) From perceptin to action. IEEE Press, Los Alamitos

    Google Scholar 

  37. Husbands P, Smith T, Jakobi N, O’Shea M (1998) Better living through chemistry: evolving gasnets for robot control. Connect Sci 10:185–210

    Article  Google Scholar 

  38. Igel, C (2003) Neuroevolution for reinforcement learning using evolution strategies. In: Sarker R, et al (eds) Congress on evolutionary computation, vol 4. IEEE Press, New York, pp 2588–2595

    Google Scholar 

  39. Katz PS (1999) What are we talking about? Modes of neuronal communication. In: Katz P (eds) Beyond neurotransmission: neuromodulation and its importance for information processing, chap 1. Oxford University Press, Oxford, pp 1–28

    Google Scholar 

  40. Kitano H (1990) Designing neural networks by genetic algorithms using graph generation system. Complex Syst J 4:461–476

    MATH  Google Scholar 

  41. Korkin M, Nawa NE, de Garis H (1998) A ’spike interval information coding’ representation for ATR’s CAM-brain machine (CBM) In: Proceedings of the 2nd international conference on evolvable systems: from biology to hardware (ICES’98). Springer, Heidelberg

  42. Koza JR (1994) Genetic programming II: automatic discovery of reusable programs. MIT Press, Cambridge

    MATH  Google Scholar 

  43. Magg S, Philippides A (2006) Gasnets and CTRNNs : a comparison in terms of evolvability. In: From animals to animats 9: proceedings of the 9th international conference on simulation of adaptive behavior. Springer, Heidelberg, pp 461–472

  44. Mattiussi C, Floreano D (2004) Evolution of analog networks using local string alignment on highly reorganizable genomes. In: Zebulum RS et al (eds) NASA/DoD conference on evolvable hardware (EH’2004), pp 30–37

  45. Mattiussi C, Dürr P, Floreano D (2007a) Center of mass encoding: a self-adaptive representation with adjustable redundancy for real-valued parameters. In: GECCO 2007. ACM Press, New York, pp 1304–1311

    Chapter  Google Scholar 

  46. Mattiussi C, Marbach D, Dürr P, Floreano D (2007b) The age of analog networks. AI Magazine (in press)

  47. Mayley G (1996) Landscapes, learning costs and genetic assimilation. Evol Comput 4(3):213–234

    Google Scholar 

  48. McHale G, Husbands P (2004) Gasnets and other evolvable neural networks applied to bipedal locomotion. In: Schaal S (ed) Proceedings from animals to animats 8: proceedings of the 8th international conference on simulation of adaptive behaviour (SAB’2004). MIT Press, Cambridge, pp 163–172

    Google Scholar 

  49. Mizutani E, Dreyfus SE (1998) Totally model-free reinforcement learning by actor-critic elman networks in non-markovian domains. In: Proceedings of the IEEE world congress on computational intelligence. IEEE Press, New York

  50. Montague P, Dayan P, Sejnowski T (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16(5):1936–1947

    Google Scholar 

  51. Montana D, Davis L (1989) Training feed forward neural networks using genetic algorithms. In: Proceedings of the 11th international joint conference on artificial intelligence. Morgan Kaufmann, San Mateo, pp 529–538

  52. Moriarty DE, Miikkulainen R (1996) Efficient reinforcement learning through symbiotic evolution. Machine Learn 22:11–32

    Google Scholar 

  53. Moscato P (1989) On evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. In: Technical report C3P 826, Pasadena

  54. Niv Y, Joel D, Meilijson I, Ruppin E (2002) Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adapt Behav 10(1):5–24

    Article  Google Scholar 

  55. Nolfi S, Floreano D (1999) Learning and evolution. Auton Robots 7(1):89–113

    Article  Google Scholar 

  56. Nolfi S, Parisi D (1996) Learning to adapt to changing environments in evolving neural networks. Adapt Behav 5(1):75–98

    Article  Google Scholar 

  57. Nolfi S, Miglino O, Parisi D (1994) Phenotypic plasticity in evolving neural networks. In: Gaussier P, Nicoud J-D (eds) From perception to action. IEEE Press, Los Alamitos

    Google Scholar 

  58. Pfeifer R, Scheier C (1999) Understanding Intelligence. MIT Press, Cambridge

    Google Scholar 

  59. Purves D (1994) Neural activity in the growth of the brain. Cambridge University Press, Cambridge

    Google Scholar 

  60. Quartz S, Sejnowski TJ (1997) The neural basis of cognitive development: a constructivist manifesto. Behav Brain Sci 4:537–555

    Article  Google Scholar 

  61. Radcliffe NJ (1991) Form an analysis and random respectful recombination. In: Belew RK, Booker LB (eds) Proceedings of the 4th international conference on genetic algorithms. Morgan Kaufmann, San Mateo

  62. Rechenberg I (1973) Evolutionsstrategie—Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Fommann-Holzboog, Stuttgart

    Google Scholar 

  63. Reil T (1999) Dynamics of gene expression in an artificial genome—implications for biological and artificial ontogeny. In: Proceedings of the 5th European conference on artificial life, pp 457–466

  64. Reil T (2003) On growth, form and computers. In: Artificial genomes as models of gene regulation. Academic Press, London, pp 256–277

  65. Reisinger J, Miikkulainen R (2007) Acquiring evolvability through adaptive representations. In: Proceedings of genetic and evolutionary computation conference (GECCO 2007)

  66. Reisinger J, Bahçeci E, Karpov I, Miikkulainen R (2007) Coevolving strategies for general game playing. In: Proceedings of the IEEE symposium on computational intelligence and games (CIG-2007)

  67. Rieke F, Warland D, van Steveninck R, Bialek W (1997) Spikes. Exploring the neural code. MIT Press, Cambridge

    Google Scholar 

  68. Rumelhart DE, Hinton GE, Williams RJ (1986a) Learning representations by back-propagation of errors. Nature 323:533–536

    Article  Google Scholar 

  69. Rumelhart DE, McClelland J, the PDP Research Group (1986b) Parallel distributed processing: explorations in the microstructure of cognition. Foundations, vol 1. MIT Press-Bradford Books, Cambridge

  70. Saggie K, Keinan A, Ruppin E (2004) Spikes that count: rethinking spikiness in neurally embedded systems. Neurocomputing 58-60:303–311

    Article  Google Scholar 

  71. Sasaki T, Tokoro M (1997) Adaptation toward changing environments: Why Darwinian in nature?. In: Husbands P, Harvey I (eds) Proceedings of the 4th European conference on artificial life. MIT Press, Cambridge

  72. Schaffer JD, Whitley D, Eshelman LJ (1992) Combinations of genetic algorithms and neural networks: a survey of the state of the art. In: Whitley D, Schaffer JD (eds) Proceedings of an international workshop on the combinations of genetic algorithms and neural networks (COGANN-92). IEEE Press, New York

  73. Schraudolph NN, Belew RK (1992) Dynamic parameter encoding for genetic algorithms. Machine Learn 9:9–21

    Google Scholar 

  74. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593–1599

    Article  Google Scholar 

  75. Shapiro J (2005) A 21st century view of evolution: genome system architecture, repetitive DNA, and natural genetic engineering. Gene 345(1):91–100

    Article  Google Scholar 

  76. Siddiqi A, Lucas S (1998) A comparison of matrix rewriting versus direct encoding for evolving neural networks. In: Proceedings of the 1998 IEEE international conference on evolutionary computation. Piscataway, NJ, pp 392–397

  77. Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586

    Article  Google Scholar 

  78. Soltoggio A, Duerr P, Mattiussi C, Floreano D (2007) Evolving neuromodulatory topologies for reinforcement learning-like problems. In: Angeline P, Michaelewicz M, Schonauer G, Yao X, Zalzala Z (eds) Proceedings of the 2007 congress on evolutionary computation. IEEE Press, New York

  79. Stanley K, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127

    Article  Google Scholar 

  80. Stanley KO, Miikkulainen R (2004) Competitive coevolution through evolutionary complexification. J Artif Intell Res 21:63–100

    Google Scholar 

  81. Stanley K, Kohl N, Sherony R, Miikkulainen R (2005a) Neuroevolution of an automobile crash warning system. In: Proceedings of genetic and evolutionary computation conference (GECCO 2005)

  82. Stanley KO, Cornelius R, Miikkulainen R, D’Silva T, Gold A (2005b) Real-time learning in the nero video game. In: Proceedings of the artificial intelligence and interactive digital entertainment conference (AIIDE 2005) demo papers

  83. Sutton RS (1988) Learning to predict by the method of temporal difference. Machine Learn 3:9–44

    Google Scholar 

  84. Sutton RS, Barto AG (1998) Reinforcement learning. an introduction. MIT Press, Cambridge

    Google Scholar 

  85. Trianni V, Ampatzis C, Christensen A, Tuci E, Dorigo M, Nolfi S (2007) From solitary to collective behaviours: decision making and cooperation. In: Advances in artificial life, proceedings of ECAL 2007. Lecture Notes in Artificial Intelligence, vol LNAI 4648. Springer, Berlin, pp 575–584

  86. Tuci E, Quinn M, Harvey I (2002) An evolutionary ecological approach to the study of learning behavior using a robot-based model. Adapt Behav 10(3–4):201–221

    Article  Google Scholar 

  87. Urzelai J, Floreano D (2001) Evolution of adaptive synapses: robots with fast adaptive behavior in new environments. Evol Comput 9:495–524

    Article  Google Scholar 

  88. Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14:347–361

    Article  Google Scholar 

  89. Widrow B, Hoff ME (1960) Adaptive switching circuits. In: Proceedings of the 1960 IRE WESCON convention, vol IV, New York. IRE. Reprinted in Anderson and Rosenfeld, 1988, pp 96–104

  90. Yamauchi BM, Beer RD (1994) Sequential behavior and learning in evolved dynamical neural networks. Adapt Behav 2(3):219–246

    Article  Google Scholar 

  91. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Swiss National Science Foundation, grant no. 200021-112060. Thanks to Daniel Marbach for the illustrations and the two anonymous reviewers for their helpful comments.

Author information

Authors and Affiliations

  1. Ecole Polytechnique Fédérale de Lausanne Laboratory of Intelligent Systems, Station 11, 1015, Lausanne, Switzerland

    Dario Floreano, Peter Dürr & Claudio Mattiussi

Authors
  1. Dario Floreano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Dürr
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claudio Mattiussi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Dürr.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Floreano, D., Dürr, P. & Mattiussi, C. Neuroevolution: from architectures to learning. Evol. Intel. 1, 47–62 (2008). https://doi.org/10.1007/s12065-007-0002-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-007-0002-4

Keywords

Search

Navigation