Compositional pattern producing networks: A novel abstraction of development

Abstract

Natural DNA can encode complexity on an enormous scale. Researchers are attempting to achieve the same representational efficiency in computers by implementing developmental encodings, i.e. encodings that map the genotype to the phenotype through a process of growth from a small starting point to a mature form. A major challenge in in this effort is to find the right level of abstraction of biological development to capture its essential properties without introducing unnecessary inefficiencies. In this paper, a novel abstraction of natural development, called Compositional Pattern Producing Networks (CPPNs), is proposed. Unlike currently accepted abstractions such as iterative rewrite systems and cellular growth simulations, CPPNs map to the phenotype without local interaction, that is, each individual component of the phenotype is determined independently of every other component. Results produced with CPPNs through interactive evolution of two-dimensional images show that such an encoding can nevertheless produce structural motifs often attributed to more conventional developmental abstractions, suggesting that local interaction may not be essential to the desirable properties of natural encoding in the way that is usually assumed.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Notes

  1. 1.

    Potential confusion notwithstanding, the relationship between development and ANNs is not accidental; rather, it shows that the same kinds of abstractions underly diverse complex phenomena.

  2. 2.

    Even though speciation is not used in these experiments, it is still a crucial component of CPPN–NEAT for any experiment that is not interactive. Speciation in original NEAT has been shown to protect innovative topologies long enough to reach their potential [10]. Since CPPN–NEAT will be used in the future to evolve complex phenotypes without user interaction, speciation is still important and therefore described in Sect. 3.4.

  3. 3.

    Our research group is currently experimenting with four-dimensional CPPNs that produce two-dimensional connectivity patterns, with promising preliminary results.

References

  1. 1.

    Dellaert, F.: Toward a biologically defensible model of development. Master’s thesis, Case Western Reserve University, Clevekand, OH (1995)

  2. 2.

    Kandel, E.R., Schwartz, J.H., Jessell, T.M. (eds.) Principles of Neural Science. Elsevier, Amsterdam, 3rd edn (1991)

    Google Scholar 

  3. 3.

    Zigmond, M.J., Bloom, F.E., Landis, S.C., Roberts, J.L., Squire, L.R. (eds.) Fundamental Neuroscience. Academic Press, London (1999)

    Google Scholar 

  4. 4.

    Deloukas, P., Schuler, G.D., Gyapay, G., Beasley, E.M., et al.: A physical map of 30,000 human genes. Science 282, 744–746 (1998)

    Article  Google Scholar 

  5. 5.

    Angeline, P.J.: Morphogenic evolutionary computations: Introduction, issues and examples. In: McDonnell, J.R., Reynolds, R.G., Fogel, D.B. (eds.) Evolutionary Programming IV: The Fourth Annual Conference on Evolutionary Programming, pp. 387–401. MIT Press, Cambridge (1995)

    Google Scholar 

  6. 6.

    Bentley, P.J., Kumar, S.: The ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-1999), pp. 35–43. Kaufmann, San Francisco (1999)

  7. 7.

    Hornby, G.S., Pollack, J.B.: Creating high-level components with a generative representation for body-brain evolution. Artif. Life 8, 223–246 (2002)

    Article  Google Scholar 

  8. 8.

    Stanley, K.O., Miikkulainen, R.: A taxonomy for artificial embryogeny. Artif. Life 9, 93–130 (2003)

    Article  Google Scholar 

  9. 9.

    Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the NERO video game. IEEE T. Evolut. Comput. Special Issue Evolut. Comput. Games 9, 653–668 (2005)

    Google Scholar 

  10. 10.

    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Comput. 10, 99–127 (2002)

    Article  Google Scholar 

  11. 11.

    Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)

    Google Scholar 

  12. 12.

    Astor, J.S., Adami, C.: A developmental model for the evolution of artificial neural networks. Artif. Life 6, 189–218 (2000)

    Article  Google Scholar 

  13. 13.

    Boers, E.J.W., Kuiper, H.: Biological metaphors and the design of modular artificial neural networks. Master’s thesis, Departments of Computer Science and Experimental and Theoretical Psychology at Leiden University, The Netherlands (1992)

  14. 14.

    Bongard, J.C.: Evolving modular genetic regulatory networks. In: Proceedings of the 2002 Congress on Evolutionary Computation (2002)

  15. 15.

    Bongard, J.C., Pfeifer, R.: Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny. In: Spector, L. et al. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference. Kaufmann, San Francisco, pp. 829–836 (2001)

  16. 16.

    Dellaert, F., Beer, R.D.: A developmental model for the evolution of complete autonomous agents. In: Maes, P., Mataric, M.J., Meyer, J.-A., Pollack, J., Wilson, S.W. (eds.) From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge, MA (1996)

    Google Scholar 

  17. 17.

    Eggenberger, P.: Evolving morphologies of simulated 3D organisms based on differential gene expression. In: Husbands, P., Harvey, I. (eds.) Proceedings of the Fourth European Conference on Artificial Life, pp. 205–213. MIT Press, Cambridge, MA (1997)

    Google Scholar 

  18. 18.

    Federici, D.: Evolving a neurocontroller through a process of embryogeny. In: Schaal, S., Ijspeert, A.J., Billard, A., Vijayakumar, S., Hallam, J., Jean-Arcady (eds.) Proceedings of the Eighth International Conference on Simulation and Adaptive Behavior (SAB-2004), pp. 373–384. MIT Press, Cambridge, MA (2004a)

    Google Scholar 

  19. 19.

    Federici, D.: Using embryonic stages to increase the evolvability of development. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004) Workshop Program. Springer-Verlag, Berlin (2004b)

  20. 20.

    Fleischer, K., Barr, A.H.: A simulation testbed for the study of multicellular development: The multiple mechanisms of morphogenesis. In: Langton, C.G. (ed.) Artificial Life III, pp. 389–416. Addison-Wesley, Reading (1993)

    Google Scholar 

  21. 21.

    Gruau, F., Whitley, D., Pyeatt, L.: A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza, J.R., Goldberg, D.E., Fogel, D.B., Riolo, R.L., (eds.) Genetic Programming 1996: Proceedings of the First Annual Conference, pp. 81–89. Cambridge, MA: MIT Press (1996)

    Google Scholar 

  22. 22.

    Hornby, G.S., Pollack, J.B.: The advantages of generative grammatical encodings for physical design. In: Proceedings of the 2002 Congress on Evolutionary Computation (2001a)

  23. 23.

    Hornby, G.S., Pollack, J.B.: Body-brain co-evolution using L-systems as a generative encoding. In: Spector, L. et al. (ed.) Proceedings of the Genetic and Evolutionary Computation Conference. Kaufmann, San Francisco (2001b)

  24. 24.

    Jakobi, N.: Harnessing morphogenesis. In: Proceedings of Information Processing in Cells and Tissues, pp. 29–41. University of Liverpool (1995)

  25. 25.

    Kaneko, K., Furusawa, C.: Emergence of multicellular organisms with dynamic differentiation and spatial pattern. Artif. Life 4, 79–93 (1998)

    Article  Google Scholar 

  26. 26.

    Kauffman, S.A.: The Origins of Order. Oxford University Press, New York (1993)

    Google Scholar 

  27. 27.

    Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Syst. 4, 461–476 (1990)

    MATH  Google Scholar 

  28. 28.

    Komosinski, M., Rotaru-Varga, A.: Comparison of different genotype encodings for simulated 3D agents. Artif. Life 7, 395–418 (2001)

    Article  Google Scholar 

  29. 29.

    Lindenmayer, A.: Mathematical models for cellular interaction in development parts I and II. J. Theor. Biol. 18, 280–299; 300–315 (1968)

    Article  Google Scholar 

  30. 30.

    Lindenmayer, A.: Adding continuous components to L-systems. In: Rozenberg, G., Salomaa, A. (eds.) L Systems, Lecture Notes in Computer Science 15, pp. 53–68. Springer-Verlag, Heidelberg, Germany (1974)

  31. 31.

    Miller, J.F.: Evolving a self-repairing, self-regulating, French flag organism. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004). Springer Verlag, Berlin (2004)

  32. 32.

    Mjolsness, E., Sharp, D.H., Reinitz, J.: A connectionist model of development. J. Theor. Biol. 152, 429–453 (1991)

    Article  Google Scholar 

  33. 33.

    Nolfi, S., Parisi, D.: Growing neural networks. Technical Report PCIA-91- 15. Institute of Psychology, C.N.R., Rome (1991)

    Google Scholar 

  34. 34.

    Prusinkiewicz, P., Lindenmayer, A.: The Algorithmic Beauty of Plants. Springer-Verlag, Heidelberg, Germany (1990)

    Google Scholar 

  35. 35.

    Schnier, T.: Evolved representations and their use in computational creativity. PhD thesis, University of Sydney Department of Architectural and Design Science, Australia (1998)

  36. 36.

    Sims, K.: Evolving 3D morphology and behavior by competition. In: Brooks, R.A., Maes, P. (eds.) Proceedings of the Fourth International Workshop on the Synthesis and Simulation of Living Systems (Artificial Life IV), pp. 28–39. MIT Press, Cambridge, MA (1994)

  37. 37.

    Turing, A.: The chemical basis of morphogenesis. Phil. T. Roy. Soc. B 237, 37–72 (1952)

    Google Scholar 

  38. 38.

    Gilbert, C.D., Wiesel, T.N.: Receptive field dynamics in adult primary visual cortex. Nature 356, 150–152 (1992)

    Article  Google Scholar 

  39. 39.

    Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. J. Neurophysiol. 28, 229–289 (1965)

    Google Scholar 

  40. 40.

    Behravan, R., Bentley, P.J.: Exploring reaction-diffusion and pattern formation. In: Proceedings of the First Australian Conference on Artificial Life (ACAL 2003) (2003)

  41. 41.

    Cangelosi, A., Parisi, D., Nolfi, S.: Cell division and migration in a genotype for neural networks. Technical Report PCIA-93, Institute of Psychology, C.N.R. Rome (1993)

  42. 42.

    Belew, R.K., Kammeyer, T.E.: In: Forrest, S. (ed.) Proceedings of the Fifth International Conference on Genetic Algorithms. Kaufmann, San Francisco (1993)

  43. 43.

    Stanley, K.O., Reisinger, J., Miikkulainen, R.: Exploiting morphological conventions for genetic reuse. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004) Workshop Program. Springer Verlag, Berlin (2004)

  44. 44.

    Raff, R.A.: The Shape of Life: Genes, Development, and the Evolution of Animal Form. Chicago, The University of Chicago Press (1996)

    Google Scholar 

  45. 45.

    Amores, A., Force, A., Yan, Y.-L., Joly, L., Amemiya, C., Fritz, A., Ho, R.K., Langeland, J., Prince, V., Wang, Y.-L., Westerfield, M., Ekker, M., Postlethwait, J.H.: Zebrafish HOX clusters and vertebrate genome evolution. Science 282, 1711–1784 (1998)

    Article  Google Scholar 

  46. 46.

    Darnell, J.E., Doolittle, W.F.: Speculations on the early course of evolution. Proc. Natl. Acad. Sci. USA 83, 1271–1275 (1986)

    Article  Google Scholar 

  47. 47.

    Force, A., Lynch, M., Pickett, F.B., Amores, A., Lin Yan, Y., Postlethwait, J.: Preservation of duplicate genes by complementary, degenerative mutations. Genetics 151, 1531–1545 (1999)

    Google Scholar 

  48. 48.

    Martin, A.P.: Increasing genomic complexity by gene duplication and the origin of vertebrates. Am. Nat. 154, 111–128 (1999)

    Article  Google Scholar 

  49. 49.

    Watson, J.D., Hopkins, N.H., Roberts, J.W., Steitz, J.A., Weiner, A.M.: Molecular Biology of the Gene, 4th edn. The Benjamin Cummings Publishing Company, Inc, Menlo Park, CA (1987)

    Google Scholar 

  50. 50.

    Altenberg, L.: Evolving better representations through selective genome growth. In: Proceedings of the IEEE World Congress on Computational Intelligence, pp. 182–187. IEEE Press, Piscataway, NJ (1994)

  51. 51.

    Carroll, S.B.: Homeotic genes and the evolution of arthropods and chordates. Nature 376, 479–485 (1995)

    Article  Google Scholar 

  52. 52.

    Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)

    MATH  Article  MathSciNet  Google Scholar 

  53. 53.

    Meinhardt, H.: Models of Biological Pattern Formation. Academic Press, London (1982)

    Google Scholar 

  54. 54.

    Curtis, D., Apfeld, J., Lehmann, R.: Nanos is an evolutionarily conserved organizer of anterior-posterior polarity. Development 121, 1899–1910 (1995)

    Google Scholar 

  55. 55.

    Lall, S., Patel, N.: Conservation and divergence in molecular mechanisms of axis formation. Annu. Rev. Genet. 35, 407–447 (2001)

    Article  Google Scholar 

  56. 56.

    Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Networks 5, 54–65 (1993)

    Article  Google Scholar 

  57. 57.

    Opitz, D.W., Shavlik, J.W.: Connectionist theory refinement: Genetically searching the space of network topologies. J. Artif. Intell. Res. 6, 177–209 (1997)

    MATH  Google Scholar 

  58. 58.

    Pujol, J.C.F., Poli, R.: Evolution of the topology and the weights of neural networks using genetic programming with a dual representation. Technical Report CSRP-97-7, School of Computer Science, The University of Birmingham, Birmingham B15 2TT, UK (1997)

  59. 59.

    Yao, X., Liu, Y.: Towards designing artificial neural networks by evolution. Appl. Math. Comput. 91, 83–90 (1996)

    Article  Google Scholar 

  60. 60.

    Zhang, B.-T., Muhlenbein, H.: Evolving optimal neural networks using genetic algorithms with Occam’s razor. Complex Syst. 7, 199–220 (1993)

    Google Scholar 

  61. 61.

    Miller, J.F.P.T.: Cartesian genetic programming. In: Proceedings of the Third European Conference on Genetic Programming Published as Lecture Notes in Computer Science, vol. 1802, pp. 121–132 (2000)

  62. 62.

    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Slection. MIT Press, Cambridge, MA (1992)

    Google Scholar 

  63. 63.

    Gomez, F., Miikkulainen, R.: Solving non-Markovian control tasks with neuroevolution. In: Proceedings of the 16th International Joint Conference on Artificial Intelligence, pp. 1356–1361. Kaufmann, San Francisco (1999)

  64. 64.

    Saravanan, N., Fogel, D.B.: Evolving neural control systems. IEEE Expert 10, 23–27 (1995)

    Article  Google Scholar 

  65. 65.

    Yao, X.: Evolving artificial neural networks. Proc. IEEE 87, 1423–1447 (1999)

    Article  Google Scholar 

  66. 66.

    Radcliffe, N.J.L: Genetic set recombination and its application to neural network topology optimization. Neural Comput. Appl. 1, 67–90 (1993)

    MATH  Article  Google Scholar 

  67. 67.

    Roggen, D., Federici, D.: Multi-cellular development: Is there scalability and robustness to gain. In: Proceedings of the 8th International Conference on Parallel Problem Solving from Nature (PPSN VIII), pp. 391–400 (2004)

  68. 68.

    Reisinger, J., Stanley, K.O., Miikkulainen, R.: Towards an empirical measure of evolvability. In: Genetic and Evolutionary Computation Conference (GECCO2005) Workshop Program, pp. 257–264. ACM Press, Washington, D.C. (2005)

  69. 69.

    Parmee, I.C.: Improving problem definition through interactive evolutionary computation. J. Artif. Intell. Eng. Design Analy. Manuf. Special Issue: Human-computer Interact. Eng. 16, 185–202 (2002)

    Google Scholar 

  70. 70.

    Takagi, H.: Interactive evolutionary computation: Fusion of the capacities of EC optimization and human evaluation. Proc. IEEE 89, 1275–1296 (2001)

    Article  Google Scholar 

  71. 71.

    Baluja, S., Pomerleau, D., Jochem, T.: Towards automated artificial evolution for computer-generated images. Connect. Sci. 6, 325–354 (1994)

    Google Scholar 

  72. 72.

    Dawkins, R.: The evolution of evolvability. In: Langton, C.G. (ed.) Artificial Life, pp. 201–220. Addison-Wesley, Reading, MA (1989)

    Google Scholar 

  73. 73.

    Greenfield, G.R.: Evolving expressions and art by choice. Leonard 33, 93–99 (2000)

    Article  Google Scholar 

  74. 74.

    Lund, H.H., Pagliarini, L., Miglino, P.: Artistic design with GA and NN. In: Proceedings of the 1st Nordic Workshop on Genetic Algorithms and Their Applications (1NWGA), pp. 97–105 (1995)

  75. 75.

    McCormack, J.P.: Interactive evolution of L-system grammars for computer graphics modelling. In: Green, D.G., Bossomaier, T. (eds.) Complex Systems: From Biology to Computation, pp. 118–130. IOS Press, Amsterdam (1993)

    Google Scholar 

  76. 76.

    Sims, K.: Artificial evolution for computer graphics. In: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’91), pp. 319–328. ACM Press, New York, NY (1991)

  77. 77.

    Smith, J.R.: Designing biomorphs with an interactive genetic algorithm. In: Belew, R.K., Booker, L.B. (eds.) Proceedings of the 4th International Conference on Genetic Algorithms (ICGA-91), pp. 535–538. Morgan Kaufmann, San Mateo, CA (1991)

    Google Scholar 

  78. 78.

    Todd, S., Latham, W.: Evolutionary Art and Computers. Academic Press, London (1992)

    Google Scholar 

  79. 79.

    Dawkins, R.: The Blind Watchmaker. Longman, Essex, UK (1986)

    Google Scholar 

  80. 80.

    Nishino, H., Takagi, H., Cho, S.-B., Utsumiya, K.: A 3D modeling system for creative design. In: 15th International Conference on Information Networking, pp. 479–486 (2001)

  81. 81.

    Todd, S., Latham, W.: The Mutation and Growth of Art by Computers, chapter 9, pp. 221–250. Morgan Kaufmann (1999)

  82. 82.

    Husbands, P., Germy, G., McIlhagga, M., Ives, R.: Two applications of genetic algorithms to component design. Evolutionary Computing. LNCS 1143, 50–61 (1996)

    Google Scholar 

  83. 83.

    Andersen, T., Otter, C., Petschulat, C., Eoff, U., Menten, T., Davis, R., Crowley, B.: A biologically-derived approach to tissue modeling. In: Westwood, J. et al., (eds.) Technology and Informatics, pp. 15–21. IOS Press, Amsterdam (2005)

    Google Scholar 

  84. 84.

    Miller, J.F.: Evolving developmental programs for adaptation, morphogenesis, and self-repair. In: Banzhaf, W., Christaller, T., Dittrich, P., Kim, J.T., Ziegler, J. (eds.) Advances in Artificial Life. 7th European Conference on Artificial Life, vol. 2801 of Lecture Notes in Artificial Intelligence, pp. 256–265. Springer, Dortmund, Germany (2003)

    Google Scholar 

Download references

Acknowledgements

Special thanks to Mattias Fagerlund for creating the first NEAT-based genetic art program based on his DelphiNEAT implementation, to Holger Ferstl for creating the second NEAT-based genetic art program based on SharpNEAT, and to Colin Green for creating SharpNEAT. All the software and source code used in this paper, including DNGA, SNGA, DelphiNEAT, and SharpNEAT, is available through http://www.cs.ucf.edu/∼kstanley.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Kenneth O. Stanley.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Stanley, K.O. Compositional pattern producing networks: A novel abstraction of development. Genet Program Evolvable Mach 8, 131–162 (2007). https://doi.org/10.1007/s10710-007-9028-8

Download citation

Keywords

  • Evolutionary computation
  • Representation
  • Developmental encoding
  • Indirect encoding
  • Artificial embryogeny
  • Generative systems
  • Complexity