Abstract
The development of artificial neural networks (ANNs) is usually a slow process in which the human expert has to test several architectures until he finds the one that achieves best results to solve a certain problem. However, there are some tools that provide the ability of automatically developing ANNs, many of them using evolutionary computation (EC) tools. One of the main problems of these techniques is that ANNs have a very complex structure, which makes them very difficult to be represented and developed by these tools. This work presents a new technique that modifies genetic programming (GP) so as to correctly and efficiently work with graph structures in order to develop ANNs. This technique also allows the obtaining of simplified networks that solve the problem with a small group of neurons. In order to measure the performance of the system and to compare the results with other ANN development methods by means of evolutionary computation (EC) techniques, several tests were performed with problems based on some of the most used test databases in the Data Mining domain. These comparisons show that the system achieves good results that are not only comparable to those of the already existing techniques but, in most cases, improve them.
Similar content being viewed by others
References
Alba E, Aldana JF, Troya JM (1993) Fully automatic ANN design: A genetic approach. In: Proc. Int. Workshop Artificial Neural Networks (IWANN’93), Lecture Notes in Computer Science, vol. 686. Berlin, Germany: Springer-Verlag, pp 399–404
Alpaydin E (1999) Combined 5 × 2 cv F test for comparing supervised classification learning algorithms. Neural Comput 11: 1885–1892
Andersen HC, Tsoi AC (1993) A constructive algorithm for the training of a multilayer perceptron based on the genetic algorithm. Complex syst 7(4): 249–268
Andrews R, Cable R, Diederich J, Geva S, Golea M, Hayward R, Ho-Stuart C, Tickle AB (1996) An evaluation and comparison of techniques for extracting and refining rules from artificial neural networks (QUT NRC Tech. Rep.). Queensland University of Technology, Neurocomputing Research Centre, Queensland
Angeline PJ, Suders GM, Pollack JB (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5: 54–65
Belew R, McInerney J, Schraudolph N (1991) Evolving networks: using the genetic algorithm with connectioninst learning. In: Proceedings of the second artificial life conference, Addison-Wesley, New York, pp 511–547
Bengio S, Bengio Y, Cloutier J, Gecsei J, (1992) On the optimization of a synaptic learning rule. In: Preprints of the conference on optimality in artificial and biological neural networks, University of Texas, Dallas)
Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, New York
Bot M (1999) Application of genetic programming to induction of linear classification trees, final term project report. Vrije Universiteit, Amsterdam
Cangelosi A, Nolfi S, Parisi D (1994) Cell division and migration in a ‘genotype’ for neural networks. Network Comput Neural Syst 5: 497–515
Cantú-Paz E, Kamath C (2005) An empirical comparison of combinatios of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst Man Cybern B Cybern 9: 915–927
Castillo PA, Arenas MG, Castillo-Valdivieso JJ, Merelo JJ, Prieto A, Romero G (2002) Artificial neural networks desing using evolutionary algorithms. In: Proceedings of the seventh world conference on soft computing
Chalmers D (1990) The evolution of learning: an experiment in genetic connectionism. In: Touretzky DS, Elman JL, Hinton GE (eds) Proceedings of the 1990 connectionist models summer school. Morgan Kaufmann, San Mateo, pp 81–90
Cramer NL (1985) A representation for the adaptive generation of simple sequential programs. In: Grefenstette (ed) Proceedings of first international conference on genetic algorithms, Carnegie-Mellon University, Pittsburgh, pp 183–187
Crosher D (1993) The artificial evolution of a generalized class of adaptive processes. In: Yao X (ed) Preprints of AI’93 workshop on evolutionary computation, pp 18–36
Dellaert F, Beer RD (1994) Toward an evolvable model of development for autonomous agent synthesis. In: Brooks R, Maes P (eds) Proceedings of the fourth cenference on artificial life. MIT Press, Cambridge
Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7): 1895–1924
Dodd N, Macfarlane D, Marland C (1991) Optimization of artificial neural network structure using genetic techniques implemented on multiple transputers. In: Welch P, Stiles D, Kunii TL, Bakkers A (eds)Proceedings of Transputing’91, IOS, Amsterdam, pp 687–700
Dorado J (1999) Modelo de un sistema para la selección automática en dominios complejos, con una estrategia cooperativa, de conjuntos de entrenamiento y arquitecturas ideales de redes de neuronas artificiales utilizando algoritmos genéticos. PhD Thesis. University of A Coruña
Dorado J, Rabuñal JR, Puertas J, Santos A, Rivero D (2002) Prediction and modelling of the flor of a typical urban basin through genetic programming. Applications of Evolutionary Computing, Proceedings of EvoWorshops 2002: EvoCOP, AvoIASP, EvoSTIM/EvoPLAN
Engelbrecht AP, Rouwhorst SE, Schoeman L (2001) A building block approach to genetic programming for rule discovery. Data Mining: a Heuristic Approach. Abbass R, Sarkar C, Newton editors, Idea Group Publishing
Fahlman S (1988) Faster-learning variations of back-propagation: an empirical study. In: Touretzky DS, Hinton G, Sejnowski T (eds) Proceedings of the 1988 connectionist models summer school, Morgan Kaufmann, Touretzky, San Mateo, pp 38–51
Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2002) exploring multiple design topologies using genetic programming and bond graphs. In: GECCO 2002: Proceedings of the genetic and evolutionary computation conference. Springer, pp 1073–1080
Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2003) System-level synthesis of mems via genetic programming and bond graphs. Genet Evol Comput—GECCO-2003 2724: 2058–2071
Fisher RA (1936) The use of multiple measurements in taxonomic problems. Annals of Eugenics (pp 179–188)
Fogel DB, Wasson EC, Porto VW (1995) A step toward computer-assisted mammography using evolutionary programming and neural networks. Cancer Lett 119(1): 93
Frean M (1990) The upstart algorithm: a method for constructing and training feedforward neural networks. Neural Comput 2(2): 198–209
Friedberg RM (1958) A learning machine: part I. IBM J Res Dev 2(1): 2–13
Friedberg RM, Dunham B, North JH (1959) A learning machine: Part II. IBM J ResDevel 3(3): 282–287
Fujiki C (1987) Using the genetic algorithm to generate lisp source code to solve the prisoner’s dilemma. International Conference on GAs, pp 236–240
Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, Reading
Greenwood GW (1997) Training partially recurrent neural networks using evolutionary strategies. IEEE Trans Speech Audio Process 5: 192–194
Gruau F (1994) Automatic definition of modular neural networks. Adapt Behav 3: 151–183
Hancock PJB (1990) GANNET: Design of a neural net for face recognition by genetic algorithm. Center for Cognitive and Computational Neuroscience, Department of Computer Science and Psychology, Stirling University, Stirling, Tech. Rep. CCCN-6, August
Harp SA, Samad T, Guha A (1989) Toward the genetic synthesis of neural networks. In: Schafer JD (eds) Procedings of third International Conference on genetic algorithms and their applications. Morgan Kaufmann, San Mateo, pp 360–369
Harp SA, Samad T, Guha A (1990) Designing application-specific neural networks using the genetic algorithm. In: Touretzky DS (eds) Advances in neural information processing systems 2. Morgan Kaufamnn, San Mate, pp 447–454
Haykin S (1999) Neural Netw. 2. Englewood Cliffs, Prentice Hall
Herrera F, Hervás C, Otero J, Sánchez L (2004) Un estudio empírico preliminar sobre los tests estadísticos más habituales en el aprendizaje automático. Giraldez R, Riquelme JC, Aguilar JS (eds.) Tendencias de la Minería de Datos en España, Red Española de Minería de Datos y Aprendizaje (TIC2002-11124-E). pp 403–412
Holland JJ (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor
Husbands P, Harvey I, Cliff D, Miller G (1994) The use of genetic algorithms for the development of sensorimotor control systems. In: Gaussier P, Nicoud JD (eds) From perception to action. IEEE Press, Los alamitos
Hwang MW, Choi JY, Park J (1997) Evolutionary projection neural networks. In: Proceedings of IEEE International Conference on Evolutionary Computation, ICEC’97, pp 667–671
Janson DJ, Frenzel JF (1993) Training product unit neural networks with genetic algorithms. IEEE Exp 8: 26–33
Kantschik W, Dittrich P, Brameier M, Banzhaf W (1999) MetaEvolution in graph GP.In: Proceedings of EuroGP’99, LNCS, vol 1598. Springer, Berlin, pp 15–28
Kantschik W, Banzhaf W (2002) Linear-graph GP—a new GP structure. In: Proceedings of the 4th European conference on genetic programming, EuroGP 2002
Kim H, Jung S, Kim T, Park K (1996) Fast learning method for backpropagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1): 101–106
Kim J-H, Choi S-S, Moon B-R (2005) Normalization for neural network in genetic search. Genetic and Evolutionary Computation Conference, pp 1–10
Kitano H (1990) Designing neural networks using genetic algorithms with graph generation system. Complex Syst 4: 461–476
Kothari B, Paya B, Esat I (1996) Machinery fault diagnostics using direct encoding graph syntax for optimizing artificial neural network structure. In: Procedings of 1996 3rd biennial Joint Conference on Engineering Systems Design and Analysis, ESDA, Part 7 (of 9), ASME, New York, pp 205–210
Koza JR (1992) Genetic programming: on the programming of computers by Jeans of natural selection. MIT Press, Cambridge
Liu Y, Yao X (1996) Evolutionary design of artificial neural networks. In: Proceedings of IEEE internationa conference on evolutionary computation (ICEC’96), Nagoya, Japan, pp 670–675
Lovell DR, Tsoi AC (1002) The performance of the neocognitron with various S-cell and C-cell transfer functions, Intell. Machines Lab, Dep. Elect. Eng., Univ. Queensland, Tech. Rep., Apr
Luke S, Spector L (1996) Evolving graphs and networks with edge encoding: preliminary report. In: Koza J (eds) Late breaking papers at the genetic programming 1996 conference (GP96). Stanford, Stanford Bookstore, pp 117–124
Luke S, Spector L (1998) A revised comparison of crossover and mutation in genetic programming. In: Proceedings of the 3rd annual conference on genetic programming. Morgan Kauffman, San Francisco
Marin FJ, Sandoval F (1993) Genetic synthesis of discrete-time recurrent neural network. In: Procedings of International workshop artificial neural networks (IWANN’93), Lecture notes in computer science, vol 686. Springer, Berlin, pp 179–184
Marshall SJ, Harrison RF (1991) Optimization and training of feedforward neural networks by genetic algorithms. In: Proceedings of the second international conference on artificial neural networks and genetic algorithms, Springer, Berlin, pp 39–43
McCulloch WS, Pitts W (1943) A logical calculus of ideas immanent in nervous activity. Bull Math Biophys (5): 115–133
Merelo J, Patón M, Canas A, Prieto A, Morán F (1993) Genetic optimization of a multilayer neural network for cluster classification tasks. Neural Netw World 3: 175–186
Merrill JWL, Port RF (1991) Fractally configured neural networks. Neural Netw 4(1): 53–60
Mertz CJ, Murphy PM (2002) UCI repository of machine learning databases. http://www-old.ics.uci.edu/pub/machine-learning-databases
Miller GF, Todd PM, Hedge SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the third international conference on genetic algorithms. Morgan Kaufmann, San Mateo, pp 379–384
Montana DJ (1995) Strongly typed genetic programming. Evol Comput 3(2): 199–200
Montana D, David L (1989) Training feed-forward neural networks using genetic algorithms. In: Proceedings of 11th international joint Conference on artificial intelligence. Morgan Kaufmann, San Mateo, pp 762–767
Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence and technology of self-organizing machines. MIT Press/Bradford Books, Cambridge
Nolfi S, Parisi D (2002a) Evolution and Learning in neural networks. In: Arbib MA (eds) Handbook of brain theory and neural networks, 2nd edn. MIT Press, Cambridge, pp 415–418
Nolfi S, Parisi D (2002b) Evolution of Artificial Neural Networks. In: Arbib MA Handbook of brain theory and neural networks, 2nd edn, MIT Press, Cambridge, 418–421
Orchad G (1993) Neural Computing. Research and applications. Ed. Institute of Physics Publishing, Londres
Patel D (1996) Using genetic algorithms to construct a network for finantial prediction. In: Proceedings of SPIE: applications of artificial neural networks in image processing, Society of Photo-Optical Instrumentation Engineers, Bellingham, pp 204–213
Poli R (1997) Evolution of graph-like programs with parallel distributed genetic programming. Genetic algorithms: proceedings of the Seventh international conference
Prechelt L (1996) Early stopping-but when? neural networks: tricks of the trade, pp 55–69
Prechelt L (1998) Automatic early stopping using cross validation: qualifying the criteria. Neural Netw 11: 761–767
Rabuñal JR, Dorado J, Puertas J, Pazos A, Santos A, Rivero D (2003) Prediction and modelling of the rainfall-runoff transformation of a typical urban basin using ANN and GP. Appl Artifi Intell, vol 17, (4), ISSN 0883-9514 (HID) (I.I 0-615)
Rabuñal JR, Dorado J, Pazos A, Pereira J, Rivero D (2004) A new approach to the extraction of ann rules and to their generalization capacity through GP. Neural Comput 16: 1483–1524
Rabuñal JR, Dorado J (eds) (2005) Artificial neural networks in real-life applications. Idea Group Inc
Reed R (1993) Pruning algorithms—a survey. IEEE Trans Neural Netw 4(5): 740–747
Ripley BD (1996) Pattern recognition and neural networks. Cambridge University Press, Cambridge
Rivero D, Rabuñal JR, Dorado J, Pazos A (2004) Using Genetic Programming for Character Discrimination in Damaged Documents. Applications of Evolutionary Computing, EvoWorkshops2004: EvoBIO, EvoCOMNET, EvoHOT, EvoIASP, EvoMUSART, EvoSTOC (Conference proceedings), p 349–358
Rivero D, Rabuñal JR, Dorado J, Pazos A (2005) Time Series Forecast with Anticipation using Genetic Programming. IWANN 2005: 968–975
Rivero D, Dorado J, Rabuñal J, Pazos A, Pereira J (2006) Artificial neural network development by means of genetic programming with graph codification, ENFORMATIKA. Trans Eng Comput Technol, World Enformatika Society 15: 209–214
Rivero D, Dorado J, Rabuñal J, Pazos A (2006) Using genetic programmning for artificial neural network development and simplification. In: Proceedings of the fifth WSEAS international conference on computational intelligence, Man-Machine Systems and Cybernetics (CIMMACS’06), WSEAS Press, pp 65–71
Ribert A, Stocker E, Lecourtier Y, Ennaji A (1994) Optimizing a neural network architecture with an adaptive parameter genetic algorithm. Lecture notes in computer science, vol 1240, Springer, pp 527–535
Ritchie MD, White BC, Parker JS, Hahn LW, Moore JH (2003) Optimization of neural network architecture using genetic programming improves detection and modelling of gene-gene interactions in studies of human diseases. BMC Bioinformatics, 3(1)
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL (eds) Parallel Distributed processing: explorations in the microstructures of cognition, vol 1. MIT Press, Cambridge, pp 318–362
Sietsma J, Dow RJF (1991) Creating Artificial Neural Networks that generalize. Neural Netw 4(1): 67–79
Smith RE, Cribbs HB III (1994) Is a learning classifier system a type of neural network. Evol Comput 2(1): 19–36
Smith RE, Cribbs IHB (1997) Combined biological paradigms: A neural, genetics-based autonomous systems strategy. Robot Autonomous Syst 22(1): 65–74
Stone M (1978) Cross-validation: a review. Matemastische Operationsforschung Statischen. Ser Statist 9: 127–139
Teller A (1996) Evolving programmers: the co-evolution of intelligent recombination operators. In: Angeline P, Kinnear K (eds) Advances in genetic programming II.. MIT Press, Cambridge
Teller A, Veloso M (2000) Internal reinforcement in a connectionist genetic programming approach. Artif Intell 120(2): 165–198
Turney P, Whitley D, Anderson R (1996) Special issue on the baldwinian effect. Evol Comput 4(3): 213–329
Vonk E, Jain LC, Johnson R (1995) Using genetic algorithms with grammar encoding to generate neural networks. In: Procedings of 1995 IEEE International Conference on Neural Networks, Part 4 (of 6), pp 1928–1931
Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3): 347–361
Yan W, Zhu Z, Hu R (1997) Hybrid genetic/BP algorithm and its application for radar/target classification. In: Proceedings of 1997 IEEE national aerospace and electronics Conference, NAECON. Part 2 (of 2), pp 981–984
Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9): 1423–1447
Yao X, Shi Y (June 1995) A preliminary study on designing artificial neural networks using co-evolution. In: Proceedings of IEEE Singapore international conference on intelligence control and instrumentation, Singapore, pp 149–154
Yao X, Liu Y (1998) Toward designing artificial neural networks by evolution. Appl Math Comput 91(1): 83–90
Zomorodian A (1995) Context-free language induction by evolution of deterministic push-down automata using genetic programming. In: Working notes of the genetic programming symposium, AAAI-95, AAAI Press, Eric Siegel and John Koza, chairs
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Rivero, D., Dorado, J., Rabuñal, J.R. et al. Modifying genetic programming for artificial neural network development for data mining. Soft Comput 13, 291–305 (2009). https://doi.org/10.1007/s00500-008-0317-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-008-0317-9