Effects of Occam's razor in evolving Sigma-Pi neural nets

  • Byoung-Tak Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 866)

Abstract

Several evolutionary algorithms make use of hierarchical representations of variable size rather than linear strings of fixed length. Variable complexity of the structures provides an additional representational power which may widen the application domain of evolutionary algorithms. The price for this is, however, that the search space is open-ended and solutions may grow to arbitrarily large size. In this paper we study the effects of structural complexity of the solutions on their generalization performance by analyzing the fitness landscape of sigma-pi neural networks. The analysis suggests that smaller networks achieve, on average, better generalization accuracy than larger ones, thus confirming the usefulness of Occam's razor. A simple method for implementing the Occam's razor principle is described and shown to be effective in improving the generalization accuracy without limiting their learning capacity.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Y. S. Abu-Mostafa: The Vapnik-Chervonenkis dimension: Information versus complexity in learning. Neural Computation 1, 312–317 (1989)Google Scholar
  2. 2.
    T. Bäck and H.-P. Schwefel: An overview of evolutionary algorithms for parameter optimization. Evolutionary Computation 1, 1–23 (1993)Google Scholar
  3. 3.
    R. Durbin, D. E. Rumelhart: Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation 1, 133–142 (1989)Google Scholar
  4. 4.
    C. L. Giles, T. Maxwell: Learning, invariance, and generalization in high-order neural networks. Applied Optics 26, 4972–4978 (1987)Google Scholar
  5. 5.
    D. E. Goldberg: Genetic Algorithms in Search, Optimization & Machine Learning. Addison Wesley, 1989Google Scholar
  6. 6.
    F. Gruau: Genetic synthesis of Boolean neural networks with a cell rewriting developmental process. Tech. Rep., Laboratoire de l'Informatique du Parallélisme, 1992Google Scholar
  7. 7.
    S. A. Harp, T. Samad, A. Guha: Towards the genetic synthesis of neural networks. In: D. Schaffer (ed.): Proceedings of the Third International Conference on Genetic Algorithms (ICGA-89). Morgan Kaufmann, 1985, pp. 360–369Google Scholar
  8. 8.
    H. Iba, T. Kurita, H. de Garis, T. Sato: System identification using structured genetic algorithm. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 279–286Google Scholar
  9. 9.
    H. Kargupta, R. E. Smith: System identification with evolving polynomial networks. In: R. Belew, L. Booker (eds.): Proceedings of the Fourth International Conference on Genetic Algorithms (ICGA-91). Morgan Kaufmann, 1991, pp. 370–376Google Scholar
  10. 10.
    K. E. Kinnear Jr.: Generality and difficulty in genetic programming: Evolving a sort. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 287–294Google Scholar
  11. 11.
    H. Kitano: Designing neural networks using genetic algorithms with graph generation system. Complex Systems 4, 461–476 (1990)Google Scholar
  12. 12.
    J. R. Koza: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992Google Scholar
  13. 13.
    H. Mühlenbein: Evolutionary algorithms — Theory and applications. In: E. H. L. Aarts, J. K. Lenstra (eds.): Local Search in Combinatorial Optimization. Wiley, 1993Google Scholar
  14. 14.
    H. Mühlenbein, D. Schierkamp-Voosen: Predictive models for the breeder genetic algorithm I: Continuous parameter optimization. Evolutionary Computation 1, 25–49 (1993)Google Scholar
  15. 15.
    I. Rechenberg: Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Stuttgart: Frommann-Holzboog, 1973Google Scholar
  16. 16.
    J. Rissanen: Stochastic complexity and modeling. The Annuls of Statistics 14, 1080–1100 (1986)Google Scholar
  17. 17.
    J. D. Schaffer: Multiple objective optimization with vector evaluated genetic algorithm. In: J. D. Schaffer (ed.): Proceedings of the International Conference on Genetic Algorithms and Their Applications. Erlbaum Associates, 1985, pp. 93–100Google Scholar
  18. 18.
    W. A. Tackett: Genetic programming for feature discovery and image discrimination. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 303–309Google Scholar
  19. 19.
    D. Whitley, T. Starkweather, C. Bogart: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Computing 14, 347–361 (1990)CrossRefGoogle Scholar
  20. 20.
    B. T. Zhang, H. Mühlenbein: Genetic programming of minimal neural nets using Occam's razor. In: S. Forrest (ed.): Proceedings of the Fifth International Conference on Genetic Algorithms (ICGA-93). Morgan Kaufmann, 1993, pp. 342–349Google Scholar
  21. 21.
    B. T. Zhang, H. Mühlenbein: Evolving optimal neural networks using genetic algorithms with Occam's razor. Complex Systems (1994)Google Scholar
  22. 22.
    B. T. Zhang, H. Mühlenbein: Synthesis of sigma-pi neural networks by the breeder genetic programming. In: Proceedings of the IEEE World Congress on Computational Intelligence (WCCI-94) New York: IEEE Press, 1994Google Scholar

Copyright information

© Springer-Verlag 1994

Authors and Affiliations

  • Byoung-Tak Zhang
    • 1
  1. 1.German National Research Center for Computer Science (GMD)Sankt AugustinGermany

Personalised recommendations