Advertisement

Soft Computing

, Volume 13, Issue 3, pp 291–305 | Cite as

Modifying genetic programming for artificial neural network development for data mining

  • Daniel Rivero
  • Julián Dorado
  • Juan R. Rabuñal
  • Alejandro Pazos
Focus

Abstract

The development of artificial neural networks (ANNs) is usually a slow process in which the human expert has to test several architectures until he finds the one that achieves best results to solve a certain problem. However, there are some tools that provide the ability of automatically developing ANNs, many of them using evolutionary computation (EC) tools. One of the main problems of these techniques is that ANNs have a very complex structure, which makes them very difficult to be represented and developed by these tools. This work presents a new technique that modifies genetic programming (GP) so as to correctly and efficiently work with graph structures in order to develop ANNs. This technique also allows the obtaining of simplified networks that solve the problem with a small group of neurons. In order to measure the performance of the system and to compare the results with other ANN development methods by means of evolutionary computation (EC) techniques, several tests were performed with problems based on some of the most used test databases in the Data Mining domain. These comparisons show that the system achieves good results that are not only comparable to those of the already existing techniques but, in most cases, improve them.

Keywords

Artificial neural networks Evolutionary computation Genetic programming Data mining Soft computing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alba E, Aldana JF, Troya JM (1993) Fully automatic ANN design: A genetic approach. In: Proc. Int. Workshop Artificial Neural Networks (IWANN’93), Lecture Notes in Computer Science, vol. 686. Berlin, Germany: Springer-Verlag, pp 399–404Google Scholar
  2. Alpaydin E (1999) Combined 5 × 2 cv F test for comparing supervised classification learning algorithms. Neural Comput 11: 1885–1892CrossRefGoogle Scholar
  3. Andersen HC, Tsoi AC (1993) A constructive algorithm for the training of a multilayer perceptron based on the genetic algorithm. Complex syst 7(4): 249–268MATHGoogle Scholar
  4. Andrews R, Cable R, Diederich J, Geva S, Golea M, Hayward R, Ho-Stuart C, Tickle AB (1996) An evaluation and comparison of techniques for extracting and refining rules from artificial neural networks (QUT NRC Tech. Rep.). Queensland University of Technology, Neurocomputing Research Centre, QueenslandGoogle Scholar
  5. Angeline PJ, Suders GM, Pollack JB (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5: 54–65CrossRefGoogle Scholar
  6. Belew R, McInerney J, Schraudolph N (1991) Evolving networks: using the genetic algorithm with connectioninst learning. In: Proceedings of the second artificial life conference, Addison-Wesley, New York, pp 511–547Google Scholar
  7. Bengio S, Bengio Y, Cloutier J, Gecsei J, (1992) On the optimization of a synaptic learning rule. In: Preprints of the conference on optimality in artificial and biological neural networks, University of Texas, Dallas)Google Scholar
  8. Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, New YorkGoogle Scholar
  9. Bot M (1999) Application of genetic programming to induction of linear classification trees, final term project report. Vrije Universiteit, AmsterdamGoogle Scholar
  10. Cangelosi A, Nolfi S, Parisi D (1994) Cell division and migration in a ‘genotype’ for neural networks. Network Comput Neural Syst 5: 497–515MATHCrossRefGoogle Scholar
  11. Cantú-Paz E, Kamath C (2005) An empirical comparison of combinatios of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst Man Cybern B Cybern 9: 915–927CrossRefGoogle Scholar
  12. Castillo PA, Arenas MG, Castillo-Valdivieso JJ, Merelo JJ, Prieto A, Romero G (2002) Artificial neural networks desing using evolutionary algorithms. In: Proceedings of the seventh world conference on soft computingGoogle Scholar
  13. Chalmers D (1990) The evolution of learning: an experiment in genetic connectionism. In: Touretzky DS, Elman JL, Hinton GE (eds) Proceedings of the 1990 connectionist models summer school. Morgan Kaufmann, San Mateo, pp 81–90Google Scholar
  14. Cramer NL (1985) A representation for the adaptive generation of simple sequential programs. In: Grefenstette (ed) Proceedings of first international conference on genetic algorithms, Carnegie-Mellon University, Pittsburgh, pp 183–187Google Scholar
  15. Crosher D (1993) The artificial evolution of a generalized class of adaptive processes. In: Yao X (ed) Preprints of AI’93 workshop on evolutionary computation, pp 18–36Google Scholar
  16. Dellaert F, Beer RD (1994) Toward an evolvable model of development for autonomous agent synthesis. In: Brooks R, Maes P (eds) Proceedings of the fourth cenference on artificial life. MIT Press, CambridgeGoogle Scholar
  17. Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7): 1895–1924CrossRefGoogle Scholar
  18. Dodd N, Macfarlane D, Marland C (1991) Optimization of artificial neural network structure using genetic techniques implemented on multiple transputers. In: Welch P, Stiles D, Kunii TL, Bakkers A (eds)Proceedings of Transputing’91, IOS, Amsterdam, pp 687–700Google Scholar
  19. Dorado J (1999) Modelo de un sistema para la selección automática en dominios complejos, con una estrategia cooperativa, de conjuntos de entrenamiento y arquitecturas ideales de redes de neuronas artificiales utilizando algoritmos genéticos. PhD Thesis. University of A CoruñaGoogle Scholar
  20. Dorado J, Rabuñal JR, Puertas J, Santos A, Rivero D (2002) Prediction and modelling of the flor of a typical urban basin through genetic programming. Applications of Evolutionary Computing, Proceedings of EvoWorshops 2002: EvoCOP, AvoIASP, EvoSTIM/EvoPLANGoogle Scholar
  21. Engelbrecht AP, Rouwhorst SE, Schoeman L (2001) A building block approach to genetic programming for rule discovery. Data Mining: a Heuristic Approach. Abbass R, Sarkar C, Newton editors, Idea Group PublishingGoogle Scholar
  22. Fahlman S (1988) Faster-learning variations of back-propagation: an empirical study. In: Touretzky DS, Hinton G, Sejnowski T (eds) Proceedings of the 1988 connectionist models summer school, Morgan Kaufmann, Touretzky, San Mateo, pp 38–51Google Scholar
  23. Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2002) exploring multiple design topologies using genetic programming and bond graphs. In: GECCO 2002: Proceedings of the genetic and evolutionary computation conference. Springer, pp 1073–1080Google Scholar
  24. Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2003) System-level synthesis of mems via genetic programming and bond graphs. Genet Evol Comput—GECCO-2003 2724: 2058–2071CrossRefGoogle Scholar
  25. Fisher RA (1936) The use of multiple measurements in taxonomic problems. Annals of Eugenics (pp 179–188)Google Scholar
  26. Fogel DB, Wasson EC, Porto VW (1995) A step toward computer-assisted mammography using evolutionary programming and neural networks. Cancer Lett 119(1): 93CrossRefGoogle Scholar
  27. Frean M (1990) The upstart algorithm: a method for constructing and training feedforward neural networks. Neural Comput 2(2): 198–209CrossRefGoogle Scholar
  28. Friedberg RM (1958) A learning machine: part I. IBM J Res Dev 2(1): 2–13MathSciNetCrossRefGoogle Scholar
  29. Friedberg RM, Dunham B, North JH (1959) A learning machine: Part II. IBM J ResDevel 3(3): 282–287MathSciNetGoogle Scholar
  30. Fujiki C (1987) Using the genetic algorithm to generate lisp source code to solve the prisoner’s dilemma. International Conference on GAs, pp 236–240Google Scholar
  31. Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, ReadingMATHGoogle Scholar
  32. Greenwood GW (1997) Training partially recurrent neural networks using evolutionary strategies. IEEE Trans Speech Audio Process 5: 192–194CrossRefGoogle Scholar
  33. Gruau F (1994) Automatic definition of modular neural networks. Adapt Behav 3: 151–183CrossRefGoogle Scholar
  34. Hancock PJB (1990) GANNET: Design of a neural net for face recognition by genetic algorithm. Center for Cognitive and Computational Neuroscience, Department of Computer Science and Psychology, Stirling University, Stirling, Tech. Rep. CCCN-6, AugustGoogle Scholar
  35. Harp SA, Samad T, Guha A (1989) Toward the genetic synthesis of neural networks. In: Schafer JD (eds) Procedings of third International Conference on genetic algorithms and their applications. Morgan Kaufmann, San Mateo, pp 360–369Google Scholar
  36. Harp SA, Samad T, Guha A (1990) Designing application-specific neural networks using the genetic algorithm. In: Touretzky DS (eds) Advances in neural information processing systems 2. Morgan Kaufamnn, San Mate, pp 447–454Google Scholar
  37. Haykin S (1999) Neural Netw. 2. Englewood Cliffs, Prentice HallGoogle Scholar
  38. Herrera F, Hervás C, Otero J, Sánchez L (2004) Un estudio empírico preliminar sobre los tests estadísticos más habituales en el aprendizaje automático. Giraldez R, Riquelme JC, Aguilar JS (eds.) Tendencias de la Minería de Datos en España, Red Española de Minería de Datos y Aprendizaje (TIC2002-11124-E). pp 403–412Google Scholar
  39. Holland JJ (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann ArborGoogle Scholar
  40. Husbands P, Harvey I, Cliff D, Miller G (1994) The use of genetic algorithms for the development of sensorimotor control systems. In: Gaussier P, Nicoud JD (eds) From perception to action. IEEE Press, Los alamitosGoogle Scholar
  41. Hwang MW, Choi JY, Park J (1997) Evolutionary projection neural networks. In: Proceedings of IEEE International Conference on Evolutionary Computation, ICEC’97, pp 667–671Google Scholar
  42. Janson DJ, Frenzel JF (1993) Training product unit neural networks with genetic algorithms. IEEE Exp 8: 26–33CrossRefGoogle Scholar
  43. Kantschik W, Dittrich P, Brameier M, Banzhaf W (1999) MetaEvolution in graph GP.In: Proceedings of EuroGP’99, LNCS, vol 1598. Springer, Berlin, pp 15–28Google Scholar
  44. Kantschik W, Banzhaf W (2002) Linear-graph GP—a new GP structure. In: Proceedings of the 4th European conference on genetic programming, EuroGP 2002Google Scholar
  45. Kim H, Jung S, Kim T, Park K (1996) Fast learning method for backpropagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1): 101–106MATHCrossRefGoogle Scholar
  46. Kim J-H, Choi S-S, Moon B-R (2005) Normalization for neural network in genetic search. Genetic and Evolutionary Computation Conference, pp 1–10Google Scholar
  47. Kitano H (1990) Designing neural networks using genetic algorithms with graph generation system. Complex Syst 4: 461–476MATHGoogle Scholar
  48. Kothari B, Paya B, Esat I (1996) Machinery fault diagnostics using direct encoding graph syntax for optimizing artificial neural network structure. In: Procedings of 1996 3rd biennial Joint Conference on Engineering Systems Design and Analysis, ESDA, Part 7 (of 9), ASME, New York, pp 205–210Google Scholar
  49. Koza JR (1992) Genetic programming: on the programming of computers by Jeans of natural selection. MIT Press, CambridgeGoogle Scholar
  50. Liu Y, Yao X (1996) Evolutionary design of artificial neural networks. In: Proceedings of IEEE internationa conference on evolutionary computation (ICEC’96), Nagoya, Japan, pp 670–675Google Scholar
  51. Lovell DR, Tsoi AC (1002) The performance of the neocognitron with various S-cell and C-cell transfer functions, Intell. Machines Lab, Dep. Elect. Eng., Univ. Queensland, Tech. Rep., AprGoogle Scholar
  52. Luke S, Spector L (1996) Evolving graphs and networks with edge encoding: preliminary report. In: Koza J (eds) Late breaking papers at the genetic programming 1996 conference (GP96). Stanford, Stanford Bookstore, pp 117–124Google Scholar
  53. Luke S, Spector L (1998) A revised comparison of crossover and mutation in genetic programming. In: Proceedings of the 3rd annual conference on genetic programming. Morgan Kauffman, San FranciscoGoogle Scholar
  54. Marin FJ, Sandoval F (1993) Genetic synthesis of discrete-time recurrent neural network. In: Procedings of International workshop artificial neural networks (IWANN’93), Lecture notes in computer science, vol 686. Springer, Berlin, pp 179–184Google Scholar
  55. Marshall SJ, Harrison RF (1991) Optimization and training of feedforward neural networks by genetic algorithms. In: Proceedings of the second international conference on artificial neural networks and genetic algorithms, Springer, Berlin, pp 39–43Google Scholar
  56. McCulloch WS, Pitts W (1943) A logical calculus of ideas immanent in nervous activity. Bull Math Biophys (5): 115–133Google Scholar
  57. Merelo J, Patón M, Canas A, Prieto A, Morán F (1993) Genetic optimization of a multilayer neural network for cluster classification tasks. Neural Netw World 3: 175–186Google Scholar
  58. Merrill JWL, Port RF (1991) Fractally configured neural networks. Neural Netw 4(1): 53–60CrossRefGoogle Scholar
  59. Mertz CJ, Murphy PM (2002) UCI repository of machine learning databases. http://www-old.ics.uci.edu/pub/machine-learning-databases
  60. Miller GF, Todd PM, Hedge SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the third international conference on genetic algorithms. Morgan Kaufmann, San Mateo, pp 379–384Google Scholar
  61. Montana DJ (1995) Strongly typed genetic programming. Evol Comput 3(2): 199–200CrossRefGoogle Scholar
  62. Montana D, David L (1989) Training feed-forward neural networks using genetic algorithms. In: Proceedings of 11th international joint Conference on artificial intelligence. Morgan Kaufmann, San Mateo, pp 762–767Google Scholar
  63. Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence and technology of self-organizing machines. MIT Press/Bradford Books, CambridgeGoogle Scholar
  64. Nolfi S, Parisi D (2002a) Evolution and Learning in neural networks. In: Arbib MA (eds) Handbook of brain theory and neural networks, 2nd edn. MIT Press, Cambridge, pp 415–418Google Scholar
  65. Nolfi S, Parisi D (2002b) Evolution of Artificial Neural Networks. In: Arbib MA Handbook of brain theory and neural networks, 2nd edn, MIT Press, Cambridge, 418–421Google Scholar
  66. Orchad G (1993) Neural Computing. Research and applications. Ed. Institute of Physics Publishing, LondresGoogle Scholar
  67. Patel D (1996) Using genetic algorithms to construct a network for finantial prediction. In: Proceedings of SPIE: applications of artificial neural networks in image processing, Society of Photo-Optical Instrumentation Engineers, Bellingham, pp 204–213Google Scholar
  68. Poli R (1997) Evolution of graph-like programs with parallel distributed genetic programming. Genetic algorithms: proceedings of the Seventh international conferenceGoogle Scholar
  69. Prechelt L (1996) Early stopping-but when? neural networks: tricks of the trade, pp 55–69Google Scholar
  70. Prechelt L (1998) Automatic early stopping using cross validation: qualifying the criteria. Neural Netw 11: 761–767CrossRefGoogle Scholar
  71. Rabuñal JR, Dorado J, Puertas J, Pazos A, Santos A, Rivero D (2003) Prediction and modelling of the rainfall-runoff transformation of a typical urban basin using ANN and GP. Appl Artifi Intell, vol 17, (4), ISSN 0883-9514 (HID) (I.I 0-615)Google Scholar
  72. Rabuñal JR, Dorado J, Pazos A, Pereira J, Rivero D (2004) A new approach to the extraction of ann rules and to their generalization capacity through GP. Neural Comput 16: 1483–1524MATHCrossRefGoogle Scholar
  73. Rabuñal JR, Dorado J (eds) (2005) Artificial neural networks in real-life applications. Idea Group IncGoogle Scholar
  74. Reed R (1993) Pruning algorithms—a survey. IEEE Trans Neural Netw 4(5): 740–747CrossRefGoogle Scholar
  75. Ripley BD (1996) Pattern recognition and neural networks. Cambridge University Press, CambridgeMATHGoogle Scholar
  76. Rivero D, Rabuñal JR, Dorado J, Pazos A (2004) Using Genetic Programming for Character Discrimination in Damaged Documents. Applications of Evolutionary Computing, EvoWorkshops2004: EvoBIO, EvoCOMNET, EvoHOT, EvoIASP, EvoMUSART, EvoSTOC (Conference proceedings), p 349–358Google Scholar
  77. Rivero D, Rabuñal JR, Dorado J, Pazos A (2005) Time Series Forecast with Anticipation using Genetic Programming. IWANN 2005: 968–975Google Scholar
  78. Rivero D, Dorado J, Rabuñal J, Pazos A, Pereira J (2006) Artificial neural network development by means of genetic programming with graph codification, ENFORMATIKA. Trans Eng Comput Technol, World Enformatika Society 15: 209–214Google Scholar
  79. Rivero D, Dorado J, Rabuñal J, Pazos A (2006) Using genetic programmning for artificial neural network development and simplification. In: Proceedings of the fifth WSEAS international conference on computational intelligence, Man-Machine Systems and Cybernetics (CIMMACS’06), WSEAS Press, pp 65–71Google Scholar
  80. Ribert A, Stocker E, Lecourtier Y, Ennaji A (1994) Optimizing a neural network architecture with an adaptive parameter genetic algorithm. Lecture notes in computer science, vol 1240, Springer, pp 527–535Google Scholar
  81. Ritchie MD, White BC, Parker JS, Hahn LW, Moore JH (2003) Optimization of neural network architecture using genetic programming improves detection and modelling of gene-gene interactions in studies of human diseases. BMC Bioinformatics, 3(1)Google Scholar
  82. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL (eds) Parallel Distributed processing: explorations in the microstructures of cognition, vol 1. MIT Press, Cambridge, pp 318–362Google Scholar
  83. Sietsma J, Dow RJF (1991) Creating Artificial Neural Networks that generalize. Neural Netw 4(1): 67–79CrossRefGoogle Scholar
  84. Smith RE, Cribbs HB III (1994) Is a learning classifier system a type of neural network. Evol Comput 2(1): 19–36CrossRefGoogle Scholar
  85. Smith RE, Cribbs IHB (1997) Combined biological paradigms: A neural, genetics-based autonomous systems strategy. Robot Autonomous Syst 22(1): 65–74CrossRefGoogle Scholar
  86. Stone M (1978) Cross-validation: a review. Matemastische Operationsforschung Statischen. Ser Statist 9: 127–139MATHGoogle Scholar
  87. Teller A (1996) Evolving programmers: the co-evolution of intelligent recombination operators. In: Angeline P, Kinnear K (eds) Advances in genetic programming II.. MIT Press, CambridgeGoogle Scholar
  88. Teller A, Veloso M (2000) Internal reinforcement in a connectionist genetic programming approach. Artif Intell 120(2): 165–198MATHCrossRefGoogle Scholar
  89. Turney P, Whitley D, Anderson R (1996) Special issue on the baldwinian effect. Evol Comput 4(3): 213–329CrossRefGoogle Scholar
  90. Vonk E, Jain LC, Johnson R (1995) Using genetic algorithms with grammar encoding to generate neural networks. In: Procedings of 1995 IEEE International Conference on Neural Networks, Part 4 (of 6), pp 1928–1931Google Scholar
  91. Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3): 347–361CrossRefGoogle Scholar
  92. Yan W, Zhu Z, Hu R (1997) Hybrid genetic/BP algorithm and its application for radar/target classification. In: Proceedings of 1997 IEEE national aerospace and electronics Conference, NAECON. Part 2 (of 2), pp 981–984Google Scholar
  93. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9): 1423–1447CrossRefGoogle Scholar
  94. Yao X, Shi Y (June 1995) A preliminary study on designing artificial neural networks using co-evolution. In: Proceedings of IEEE Singapore international conference on intelligence control and instrumentation, Singapore, pp 149–154Google Scholar
  95. Yao X, Liu Y (1998) Toward designing artificial neural networks by evolution. Appl Math Comput 91(1): 83–90MATHCrossRefMathSciNetGoogle Scholar
  96. Zomorodian A (1995) Context-free language induction by evolution of deterministic push-down automata using genetic programming. In: Working notes of the genetic programming symposium, AAAI-95, AAAI Press, Eric Siegel and John Koza, chairsGoogle Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  • Daniel Rivero
    • 1
  • Julián Dorado
    • 1
  • Juan R. Rabuñal
    • 1
  • Alejandro Pazos
    • 1
  1. 1.Fac. InformaticaUniversity of A CoruñaA CoruñaSpain

Personalised recommendations