Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Compositional pattern producing networks: A novel abstraction of development

Abstract

Natural DNA can encode complexity on an enormous scale. Researchers are attempting to achieve the same representational efficiency in computers by implementing developmental encodings, i.e. encodings that map the genotype to the phenotype through a process of growth from a small starting point to a mature form. A major challenge in in this effort is to find the right level of abstraction of biological development to capture its essential properties without introducing unnecessary inefficiencies. In this paper, a novel abstraction of natural development, called Compositional Pattern Producing Networks (CPPNs), is proposed. Unlike currently accepted abstractions such as iterative rewrite systems and cellular growth simulations, CPPNs map to the phenotype without local interaction, that is, each individual component of the phenotype is determined independently of every other component. Results produced with CPPNs through interactive evolution of two-dimensional images show that such an encoding can nevertheless produce structural motifs often attributed to more conventional developmental abstractions, suggesting that local interaction may not be essential to the desirable properties of natural encoding in the way that is usually assumed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Notes

  1. 1.

    Potential confusion notwithstanding, the relationship between development and ANNs is not accidental; rather, it shows that the same kinds of abstractions underly diverse complex phenomena.

  2. 2.

    Even though speciation is not used in these experiments, it is still a crucial component of CPPN–NEAT for any experiment that is not interactive. Speciation in original NEAT has been shown to protect innovative topologies long enough to reach their potential [10]. Since CPPN–NEAT will be used in the future to evolve complex phenotypes without user interaction, speciation is still important and therefore described in Sect. 3.4.

  3. 3.

    Our research group is currently experimenting with four-dimensional CPPNs that produce two-dimensional connectivity patterns, with promising preliminary results.

References

  1. 1.

    Dellaert, F.: Toward a biologically defensible model of development. Master’s thesis, Case Western Reserve University, Clevekand, OH (1995)

  2. 2.

    Kandel, E.R., Schwartz, J.H., Jessell, T.M. (eds.) Principles of Neural Science. Elsevier, Amsterdam, 3rd edn (1991)

  3. 3.

    Zigmond, M.J., Bloom, F.E., Landis, S.C., Roberts, J.L., Squire, L.R. (eds.) Fundamental Neuroscience. Academic Press, London (1999)

  4. 4.

    Deloukas, P., Schuler, G.D., Gyapay, G., Beasley, E.M., et al.: A physical map of 30,000 human genes. Science 282, 744–746 (1998)

  5. 5.

    Angeline, P.J.: Morphogenic evolutionary computations: Introduction, issues and examples. In: McDonnell, J.R., Reynolds, R.G., Fogel, D.B. (eds.) Evolutionary Programming IV: The Fourth Annual Conference on Evolutionary Programming, pp. 387–401. MIT Press, Cambridge (1995)

  6. 6.

    Bentley, P.J., Kumar, S.: The ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-1999), pp. 35–43. Kaufmann, San Francisco (1999)

  7. 7.

    Hornby, G.S., Pollack, J.B.: Creating high-level components with a generative representation for body-brain evolution. Artif. Life 8, 223–246 (2002)

  8. 8.

    Stanley, K.O., Miikkulainen, R.: A taxonomy for artificial embryogeny. Artif. Life 9, 93–130 (2003)

  9. 9.

    Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the NERO video game. IEEE T. Evolut. Comput. Special Issue Evolut. Comput. Games 9, 653–668 (2005)

  10. 10.

    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Comput. 10, 99–127 (2002)

  11. 11.

    Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)

  12. 12.

    Astor, J.S., Adami, C.: A developmental model for the evolution of artificial neural networks. Artif. Life 6, 189–218 (2000)

  13. 13.

    Boers, E.J.W., Kuiper, H.: Biological metaphors and the design of modular artificial neural networks. Master’s thesis, Departments of Computer Science and Experimental and Theoretical Psychology at Leiden University, The Netherlands (1992)

  14. 14.

    Bongard, J.C.: Evolving modular genetic regulatory networks. In: Proceedings of the 2002 Congress on Evolutionary Computation (2002)

  15. 15.

    Bongard, J.C., Pfeifer, R.: Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny. In: Spector, L. et al. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference. Kaufmann, San Francisco, pp. 829–836 (2001)

  16. 16.

    Dellaert, F., Beer, R.D.: A developmental model for the evolution of complete autonomous agents. In: Maes, P., Mataric, M.J., Meyer, J.-A., Pollack, J., Wilson, S.W. (eds.) From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge, MA (1996)

  17. 17.

    Eggenberger, P.: Evolving morphologies of simulated 3D organisms based on differential gene expression. In: Husbands, P., Harvey, I. (eds.) Proceedings of the Fourth European Conference on Artificial Life, pp. 205–213. MIT Press, Cambridge, MA (1997)

  18. 18.

    Federici, D.: Evolving a neurocontroller through a process of embryogeny. In: Schaal, S., Ijspeert, A.J., Billard, A., Vijayakumar, S., Hallam, J., Jean-Arcady (eds.) Proceedings of the Eighth International Conference on Simulation and Adaptive Behavior (SAB-2004), pp. 373–384. MIT Press, Cambridge, MA (2004a)

  19. 19.

    Federici, D.: Using embryonic stages to increase the evolvability of development. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004) Workshop Program. Springer-Verlag, Berlin (2004b)

  20. 20.

    Fleischer, K., Barr, A.H.: A simulation testbed for the study of multicellular development: The multiple mechanisms of morphogenesis. In: Langton, C.G. (ed.) Artificial Life III, pp. 389–416. Addison-Wesley, Reading (1993)

  21. 21.

    Gruau, F., Whitley, D., Pyeatt, L.: A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza, J.R., Goldberg, D.E., Fogel, D.B., Riolo, R.L., (eds.) Genetic Programming 1996: Proceedings of the First Annual Conference, pp. 81–89. Cambridge, MA: MIT Press (1996)

  22. 22.

    Hornby, G.S., Pollack, J.B.: The advantages of generative grammatical encodings for physical design. In: Proceedings of the 2002 Congress on Evolutionary Computation (2001a)

  23. 23.

    Hornby, G.S., Pollack, J.B.: Body-brain co-evolution using L-systems as a generative encoding. In: Spector, L. et al. (ed.) Proceedings of the Genetic and Evolutionary Computation Conference. Kaufmann, San Francisco (2001b)

  24. 24.

    Jakobi, N.: Harnessing morphogenesis. In: Proceedings of Information Processing in Cells and Tissues, pp. 29–41. University of Liverpool (1995)

  25. 25.

    Kaneko, K., Furusawa, C.: Emergence of multicellular organisms with dynamic differentiation and spatial pattern. Artif. Life 4, 79–93 (1998)

  26. 26.

    Kauffman, S.A.: The Origins of Order. Oxford University Press, New York (1993)

  27. 27.

    Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Syst. 4, 461–476 (1990)

  28. 28.

    Komosinski, M., Rotaru-Varga, A.: Comparison of different genotype encodings for simulated 3D agents. Artif. Life 7, 395–418 (2001)

  29. 29.

    Lindenmayer, A.: Mathematical models for cellular interaction in development parts I and II. J. Theor. Biol. 18, 280–299; 300–315 (1968)

  30. 30.

    Lindenmayer, A.: Adding continuous components to L-systems. In: Rozenberg, G., Salomaa, A. (eds.) L Systems, Lecture Notes in Computer Science 15, pp. 53–68. Springer-Verlag, Heidelberg, Germany (1974)

  31. 31.

    Miller, J.F.: Evolving a self-repairing, self-regulating, French flag organism. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004). Springer Verlag, Berlin (2004)

  32. 32.

    Mjolsness, E., Sharp, D.H., Reinitz, J.: A connectionist model of development. J. Theor. Biol. 152, 429–453 (1991)

  33. 33.

    Nolfi, S., Parisi, D.: Growing neural networks. Technical Report PCIA-91- 15. Institute of Psychology, C.N.R., Rome (1991)

  34. 34.

    Prusinkiewicz, P., Lindenmayer, A.: The Algorithmic Beauty of Plants. Springer-Verlag, Heidelberg, Germany (1990)

  35. 35.

    Schnier, T.: Evolved representations and their use in computational creativity. PhD thesis, University of Sydney Department of Architectural and Design Science, Australia (1998)

  36. 36.

    Sims, K.: Evolving 3D morphology and behavior by competition. In: Brooks, R.A., Maes, P. (eds.) Proceedings of the Fourth International Workshop on the Synthesis and Simulation of Living Systems (Artificial Life IV), pp. 28–39. MIT Press, Cambridge, MA (1994)

  37. 37.

    Turing, A.: The chemical basis of morphogenesis. Phil. T. Roy. Soc. B 237, 37–72 (1952)

  38. 38.

    Gilbert, C.D., Wiesel, T.N.: Receptive field dynamics in adult primary visual cortex. Nature 356, 150–152 (1992)

  39. 39.

    Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. J. Neurophysiol. 28, 229–289 (1965)

  40. 40.

    Behravan, R., Bentley, P.J.: Exploring reaction-diffusion and pattern formation. In: Proceedings of the First Australian Conference on Artificial Life (ACAL 2003) (2003)

  41. 41.

    Cangelosi, A., Parisi, D., Nolfi, S.: Cell division and migration in a genotype for neural networks. Technical Report PCIA-93, Institute of Psychology, C.N.R. Rome (1993)

  42. 42.

    Belew, R.K., Kammeyer, T.E.: In: Forrest, S. (ed.) Proceedings of the Fifth International Conference on Genetic Algorithms. Kaufmann, San Francisco (1993)

  43. 43.

    Stanley, K.O., Reisinger, J., Miikkulainen, R.: Exploiting morphological conventions for genetic reuse. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004) Workshop Program. Springer Verlag, Berlin (2004)

  44. 44.

    Raff, R.A.: The Shape of Life: Genes, Development, and the Evolution of Animal Form. Chicago, The University of Chicago Press (1996)

  45. 45.

    Amores, A., Force, A., Yan, Y.-L., Joly, L., Amemiya, C., Fritz, A., Ho, R.K., Langeland, J., Prince, V., Wang, Y.-L., Westerfield, M., Ekker, M., Postlethwait, J.H.: Zebrafish HOX clusters and vertebrate genome evolution. Science 282, 1711–1784 (1998)

  46. 46.

    Darnell, J.E., Doolittle, W.F.: Speculations on the early course of evolution. Proc. Natl. Acad. Sci. USA 83, 1271–1275 (1986)

  47. 47.

    Force, A., Lynch, M., Pickett, F.B., Amores, A., Lin Yan, Y., Postlethwait, J.: Preservation of duplicate genes by complementary, degenerative mutations. Genetics 151, 1531–1545 (1999)

  48. 48.

    Martin, A.P.: Increasing genomic complexity by gene duplication and the origin of vertebrates. Am. Nat. 154, 111–128 (1999)

  49. 49.

    Watson, J.D., Hopkins, N.H., Roberts, J.W., Steitz, J.A., Weiner, A.M.: Molecular Biology of the Gene, 4th edn. The Benjamin Cummings Publishing Company, Inc, Menlo Park, CA (1987)

  50. 50.

    Altenberg, L.: Evolving better representations through selective genome growth. In: Proceedings of the IEEE World Congress on Computational Intelligence, pp. 182–187. IEEE Press, Piscataway, NJ (1994)

  51. 51.

    Carroll, S.B.: Homeotic genes and the evolution of arthropods and chordates. Nature 376, 479–485 (1995)

  52. 52.

    Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)

  53. 53.

    Meinhardt, H.: Models of Biological Pattern Formation. Academic Press, London (1982)

  54. 54.

    Curtis, D., Apfeld, J., Lehmann, R.: Nanos is an evolutionarily conserved organizer of anterior-posterior polarity. Development 121, 1899–1910 (1995)

  55. 55.

    Lall, S., Patel, N.: Conservation and divergence in molecular mechanisms of axis formation. Annu. Rev. Genet. 35, 407–447 (2001)

  56. 56.

    Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Networks 5, 54–65 (1993)

  57. 57.

    Opitz, D.W., Shavlik, J.W.: Connectionist theory refinement: Genetically searching the space of network topologies. J. Artif. Intell. Res. 6, 177–209 (1997)

  58. 58.

    Pujol, J.C.F., Poli, R.: Evolution of the topology and the weights of neural networks using genetic programming with a dual representation. Technical Report CSRP-97-7, School of Computer Science, The University of Birmingham, Birmingham B15 2TT, UK (1997)

  59. 59.

    Yao, X., Liu, Y.: Towards designing artificial neural networks by evolution. Appl. Math. Comput. 91, 83–90 (1996)

  60. 60.

    Zhang, B.-T., Muhlenbein, H.: Evolving optimal neural networks using genetic algorithms with Occam’s razor. Complex Syst. 7, 199–220 (1993)

  61. 61.

    Miller, J.F.P.T.: Cartesian genetic programming. In: Proceedings of the Third European Conference on Genetic Programming Published as Lecture Notes in Computer Science, vol. 1802, pp. 121–132 (2000)

  62. 62.

    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Slection. MIT Press, Cambridge, MA (1992)

  63. 63.

    Gomez, F., Miikkulainen, R.: Solving non-Markovian control tasks with neuroevolution. In: Proceedings of the 16th International Joint Conference on Artificial Intelligence, pp. 1356–1361. Kaufmann, San Francisco (1999)

  64. 64.

    Saravanan, N., Fogel, D.B.: Evolving neural control systems. IEEE Expert 10, 23–27 (1995)

  65. 65.

    Yao, X.: Evolving artificial neural networks. Proc. IEEE 87, 1423–1447 (1999)

  66. 66.

    Radcliffe, N.J.L: Genetic set recombination and its application to neural network topology optimization. Neural Comput. Appl. 1, 67–90 (1993)

  67. 67.

    Roggen, D., Federici, D.: Multi-cellular development: Is there scalability and robustness to gain. In: Proceedings of the 8th International Conference on Parallel Problem Solving from Nature (PPSN VIII), pp. 391–400 (2004)

  68. 68.

    Reisinger, J., Stanley, K.O., Miikkulainen, R.: Towards an empirical measure of evolvability. In: Genetic and Evolutionary Computation Conference (GECCO2005) Workshop Program, pp. 257–264. ACM Press, Washington, D.C. (2005)

  69. 69.

    Parmee, I.C.: Improving problem definition through interactive evolutionary computation. J. Artif. Intell. Eng. Design Analy. Manuf. Special Issue: Human-computer Interact. Eng. 16, 185–202 (2002)

  70. 70.

    Takagi, H.: Interactive evolutionary computation: Fusion of the capacities of EC optimization and human evaluation. Proc. IEEE 89, 1275–1296 (2001)

  71. 71.

    Baluja, S., Pomerleau, D., Jochem, T.: Towards automated artificial evolution for computer-generated images. Connect. Sci. 6, 325–354 (1994)

  72. 72.

    Dawkins, R.: The evolution of evolvability. In: Langton, C.G. (ed.) Artificial Life, pp. 201–220. Addison-Wesley, Reading, MA (1989)

  73. 73.

    Greenfield, G.R.: Evolving expressions and art by choice. Leonard 33, 93–99 (2000)

  74. 74.

    Lund, H.H., Pagliarini, L., Miglino, P.: Artistic design with GA and NN. In: Proceedings of the 1st Nordic Workshop on Genetic Algorithms and Their Applications (1NWGA), pp. 97–105 (1995)

  75. 75.

    McCormack, J.P.: Interactive evolution of L-system grammars for computer graphics modelling. In: Green, D.G., Bossomaier, T. (eds.) Complex Systems: From Biology to Computation, pp. 118–130. IOS Press, Amsterdam (1993)

  76. 76.

    Sims, K.: Artificial evolution for computer graphics. In: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’91), pp. 319–328. ACM Press, New York, NY (1991)

  77. 77.

    Smith, J.R.: Designing biomorphs with an interactive genetic algorithm. In: Belew, R.K., Booker, L.B. (eds.) Proceedings of the 4th International Conference on Genetic Algorithms (ICGA-91), pp. 535–538. Morgan Kaufmann, San Mateo, CA (1991)

  78. 78.

    Todd, S., Latham, W.: Evolutionary Art and Computers. Academic Press, London (1992)

  79. 79.

    Dawkins, R.: The Blind Watchmaker. Longman, Essex, UK (1986)

  80. 80.

    Nishino, H., Takagi, H., Cho, S.-B., Utsumiya, K.: A 3D modeling system for creative design. In: 15th International Conference on Information Networking, pp. 479–486 (2001)

  81. 81.

    Todd, S., Latham, W.: The Mutation and Growth of Art by Computers, chapter 9, pp. 221–250. Morgan Kaufmann (1999)

  82. 82.

    Husbands, P., Germy, G., McIlhagga, M., Ives, R.: Two applications of genetic algorithms to component design. Evolutionary Computing. LNCS 1143, 50–61 (1996)

  83. 83.

    Andersen, T., Otter, C., Petschulat, C., Eoff, U., Menten, T., Davis, R., Crowley, B.: A biologically-derived approach to tissue modeling. In: Westwood, J. et al., (eds.) Technology and Informatics, pp. 15–21. IOS Press, Amsterdam (2005)

  84. 84.

    Miller, J.F.: Evolving developmental programs for adaptation, morphogenesis, and self-repair. In: Banzhaf, W., Christaller, T., Dittrich, P., Kim, J.T., Ziegler, J. (eds.) Advances in Artificial Life. 7th European Conference on Artificial Life, vol. 2801 of Lecture Notes in Artificial Intelligence, pp. 256–265. Springer, Dortmund, Germany (2003)

Download references

Acknowledgements

Special thanks to Mattias Fagerlund for creating the first NEAT-based genetic art program based on his DelphiNEAT implementation, to Holger Ferstl for creating the second NEAT-based genetic art program based on SharpNEAT, and to Colin Green for creating SharpNEAT. All the software and source code used in this paper, including DNGA, SNGA, DelphiNEAT, and SharpNEAT, is available through http://www.cs.ucf.edu/∼kstanley.

Author information

Correspondence to Kenneth O. Stanley.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Stanley, K.O. Compositional pattern producing networks: A novel abstraction of development. Genet Program Evolvable Mach 8, 131–162 (2007). https://doi.org/10.1007/s10710-007-9028-8

Download citation

Keywords

  • Evolutionary computation
  • Representation
  • Developmental encoding
  • Indirect encoding
  • Artificial embryogeny
  • Generative systems
  • Complexity