Abstract
The recently proposed Semantic Learning Machine (SLM) neuroevolution algorithm is able to construct Neural Networks (NNs) over unimodal error landscapes in any supervised learning problem where the error is measured as a distance to the known targets. This chapter studies how different methods of dynamically using the training data affect the resulting generalization of the SLM algorithm. Across four real-world binary classification datasets, SLM is shown to outperform the Multi-layer Perceptron, with statistical significance, after parameter tuning is performed in both algorithms. Furthermore, this chapter also studies how different ensemble constructions methods influence the resulting generalization. The results show that the stochastic nature of SLM already confers enough diversity to the ensembles such that Bagging and Boosting cannot improve upon a simple averaging ensemble construction method. Finally, some initial results with SLM and Convolutional NNs are presented and future Deep Learning perspectives are discussed.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abiodun, O.I., Jantan, A., Omolara, A.E., Dada, K.V., Mohamed, N.A., Arshad, H.: State-of-the-art in artificial neural network applications: A survey. Heliyon 4(11), e00,938 (2018)
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
Alba, E., Aldana, J., Troya, J.M.: Full automatic ANN design: A genetic approach. In: International Workshop on Artificial Neural Networks, pp. 399–404. Springer (1993)
Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks 5(1), 54–65 (1994)
Bornholdt, S., Graudenz, D.: General asymmetric neural networks and structure design by genetic algorithms. Neural Networks 5(2), 327–334 (1992)
Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996).
Chauvin, Y., Rumelhart, D.E.: Backpropagation: Theory, architectures, and applications. Psychology Press (2013)
Cun, Y.L., Denker, J.S., Solla, S.A.: Advances in neural information processing systems 2. chap. Optimal Brain Damage, pp. 598–605. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)
DasGupta, B., Schnitger, G.: Efficient approximation with neural networks: A comparison of gate functions. Pennsylvania State University, Department of Computer Science (1992)
Dill, F.A., Deer, B.C.: An exploration of genetic algorithms for the selection of connection weights in dynamical neural networks. In: Proceedings of the IEEE 1991 National Aerospace and Electronics Conference NAECON 1991, vol. 3, pp. 1111–1115 (1991)
Drucker, H.: Improving regressors using boosting techniques. In: Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pp. 107–115. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1997).
Epitropakis, M.G., Plagianakos, V.P., Vrahatis, M.N.: Evolutionary Algorithm Training of Higher-Order Neural Networks. IGI Global (2009)
Fahlman, S.E., Lebiere, C.: Advances in neural information processing systems 2. chap. The Cascade-correlation Learning Architecture, pp. 524–532. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)
Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: From architectures to learning. Evolutionary Intelligence 1(1), 47–62 (2008)
Frean, M.: The upstart algorithm: A method for constructing and training feedforward neural networks. Neural Computation 2(2), 198–209 (1990)
Galván-López, E., Vázquez-Mendoza, L., Schoenauer, M., Trujillo, L.: On the use of dynamic GP fitness cases in static and dynamic optimisation problems. In: International Conference on Artificial Evolution (Evolution Artificielle), pp. 72–87. Springer (2017)
Garro, B.A., Vázquez, R.A.: Designing artificial neural networks using particle swarm optimization algorithms. Computational Intelligence and Neuroscience 2015, 61 (2015)
Gonçalves, I., Silva, S.: Experiments on controlling overfitting in genetic programming. In: Local proceedings of the 15th Portuguese Conference on Artificial Intelligence, EPIA 2011 (2011)
Gonçalves, I., Silva, S., Melo, J.B., Carreiras, J.M.B.: Random sampling technique for overfitting control in genetic programming. In: Genetic Programming, pp. 218–229. Springer (2012)
Gonçalves, I., Silva, S.: Balancing learning and overfitting in genetic programming with interleaved sampling of training data. In: Genetic Programming, pp. 73–84. Springer (2013)
Gonçalves, I., Silva, S., Fonseca, C.M.: On the generalization ability of geometric semantic genetic programming. In: Genetic Programming, pp. 41–52. Springer (2015)
Gonçalves, I., Silva, S., Fonseca, C.M.: Semantic learning machine: A feedforward neural network construction algorithm inspired by geometric semantic genetic programming. In: Progress in Artificial Intelligence, Lecture Notes in Computer Science, vol. 9273, pp. 280–285. Springer (2015)
Gonçalves, I., Silva, S., Fonseca, C.M., Castelli, M.: Arbitrarily close alignments in the error space: A geometric semantic genetic programming approach. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 99–100. ACM (2016)
Gonçalves, I.: An exploration of generalization and overfitting in genetic programming: Standard and geometric semantic approaches. Ph.D. thesis, Department of Informatics Engineering, University of Coimbra, Portugal (2017)
Gonçalves, I., Silva, S., Fonseca, C.M., Castelli, M.: Unsure when to stop? ask your semantic neighbors. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO’17, pp. 929–936. ACM, New York, NY, USA (2017).
Greenwood, G.W.: Training partially recurrent neural networks using evolutionary strategies. IEEE Transactions on Speech and Audio Processing 5(2), 192–194 (1997)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks 2(5), 359–366 (1989)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Networks 4(2), 251–257 (1991)
Hush, D.R., Horne, B.G.: Progress in supervised neural networks. IEEE Signal Processing Magazine 10(1), 8–39 (1993)
Irie, B., Miyake, S.: Capabilities of three-layered perceptrons. In: IEEE International Conference on Neural Networks, vol. 1, p. 218 (1988)
Jagusch, J.B., Gonçalves, I., Castelli, M.: Neuroevolution under unimodal error landscapes: An exploration of the semantic learning machine algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 159–160. ACM (2018)
Jarrett, K., Kavukcuoglu, K., LeCun, Y., et al.: What is the best multi-stage architecture for object recognition? In: 2009 IEEE 12th International Conference on Computer Vision, pp. 2146–2153. IEEE (2009)
Jian, F., Yugeng, X.: Neural network design based on evolutionary programming. Artificial Intelligence in Engineering 11(2), 155–161 (1997)
Kiefer, J., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. 23(3), 462–466 (1952).
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014).
Kiranyaz, S., Ince, T., Yildirim, A., Gabbouj, M.: Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Networks 22(10), 1448–1462 (2009)
Konda, K., Memisevic, R., Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features. arXiv preprint arXiv:1402.3337 (2014)
Koza, J.R., Rice, J.P.: Genetic generation of both the weights and architecture for a neural network. In: IJCNN-91-Seattle International Joint Conference on Neural Networks, vol. 2, pp. 397–404. IEEE (1991)
Lapa, P., Gonçalves, I., Rundo, L., Castelli, M.: Enhancing classification performance of convolutional neural networks for prostate cancer detection on magnetic resonance images: A study with the semantic learning machine. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’19. ACM, New York, NY, USA (2019).
Lapa, P., Gonçalves, I., Rundo, L., Castelli, M.: Semantic learning machine improves the cnn-based detection of prostate cancer in non-contrast-enhanced mri. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’19. ACM, New York, NY, USA (2019).
Lei, J., He, G., Jiang, J.P.: The state estimation of the cstr system based on a recurrent neural network trained by HGAs. In: Proceedings of International Conference on Neural Networks (ICNN’97), vol. 2, pp. 779–782 (1997)
Lichman, M.: UCI Machine Learning Repository (2013).
Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N., Huisman, H.: “PROSTATEx Challenge data”, The Cancer Imaging Archive. https://wiki.cancerimagingarchive.net/display/Public/SPIE-AAPM-NCI+PROSTATEx+Challenges (2017). Online; Accessed on January 25, 2019
Liu, S., Zheng, H., Feng, Y., Li, W.: Prostate cancer diagnosis using deep learning with 3D multiparametric MRI. In: Medical Imaging 2017: Computer-Aided Diagnosis, Proceedings SPIE, vol. 10134, p. 1013428. International Society for Optics and Photonics (2017).
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. ICML, vol. 30, p. 3 (2013)
Manessi, F., Rozza, A.: Learning combinations of activation functions. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 61–66. IEEE (2018)
Mani, G.: Learning by gradient descent in function space. In: 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings, pp. 242–247 (1990)
Martinez, Y., Trujillo, L., Naredo, E., Legrand, P.: A comparison of fitness-case sampling methods for symbolic regression with genetic programming. In: EVOLVE-A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation V, pp. 201–212. Springer (2014)
Martínez, Y., Naredo, E., Trujillo, L., Legrand, P., López, U.: A comparison of fitness-case sampling methods for genetic programming. Journal of Experimental & Theoretical Artificial Intelligence 29(6), 1203–1224 (2017)
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. In: R. Kozma, C. Alippi, Y. Choe, F.C. Morabito (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing. Amsterdam: Elsevier (2018).
Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: Proceedings of the Third International Conference on Genetic Algorithms, pp. 379–384. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1989)
Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. In: IJCAI, vol. 89, pp. 762–767 (1989)
Moraglio, A., Krawiec, K., Johnson, C.G.: Geometric semantic genetic programming. In: Parallel Problem Solving from Nature-PPSN XII, pp. 21–31. Springer (2012)
Moraglio, A., Mambrini, A.: Runtime analysis of mutation-based geometric semantic genetic programming for basis functions regression. In: Proceedings of the 15th annual conference on Genetic and Evolutionary Computation, pp. 989–996. ACM (2013)
Mozer, M.C., Smolensky, P.: Advances in neural information processing systems 1. chap. Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment, pp. 107–115. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1989)
Nikolopoulos, C., Fellrath, P.: A hybrid expert system for investment advising. Expert Systems 11(4), 245–250 (1994)
Oliker, S., Furst, M., Maimon, O.: Design architectures and training of neural networks with a distributed genetic algorithm. In: IEEE International Conference on Neural Networks, pp. 199–202. IEEE (1993)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Statist. 22(3), 400–407 (1951).
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533 (1986)
S. Ding H. Li, C.S.J.Y.F.J.: Evolutionary artificial neural networks: A review. Artificial Intelligence Review 39, 251–260 (2013).
Samarasinghe, S.: Neural networks for applied sciences and engineering: From fundamentals to complex pattern recognition. Auerbach Publications (2016)
Schaffer, J.D., Caruana, R.A., Eshelman, L.J.: Using genetic search to exploit the emergent behavior of neural networks. Physica D: Nonlinear Phenomena 42(1–3), 244–248 (1990)
Schiffmann, W., Joost, M., Werner, R.: Synthesis and performance analysis of multilayer neural network architectures (1992)
Schoenauer, M., Ronald, E.: Genetic extensions of neural net learning: Transfer functions and renormalisation coefficients
Sietsma, J., Dow, R.J.: Creating artificial neural networks that generalize. Neural Networks 4(1), 67–79 (1991)
Silva, S., Ingalalli, V., Vinga, S., Carreiras, J.M., Melo, J.B., Castelli, M., Vanneschi, L., Gonçalves, I., Caldas, J.: Prediction of forest aboveground biomass: An exercise on avoiding overfitting. In: European Conference on the Applications of Evolutionary Computation, pp. 407–417. Springer (2013)
Srinivas, M., Patnaik, L.M.: Learning neural network weights using genetic algorithms-improving performance by search-space reduction. In: [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, vol. 3, pp. 2331–2336 (1991)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Computation 10(2), 99–127 (2002)
Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nature Machine Intelligence 1(1), 24–35 (2019)
Sutton, R.S.: Two problems with backpropagation and other steepest-descent learning procedures for networks. In: Proceedings of the Eighth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum (1986)
White, D., Ligomenides, P.: Gannet: A genetic algorithm for optimizing topology and weights in neural network design. In: International Workshop on Artificial Neural Networks, pp. 322–327. Springer (1993)
Whitley, D., Starkweather, T., Bogart, C.: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Computing 14(3), 347–361 (1990)
Widrow, B., Lehr, M.A.: 30 years of adaptive neural networks: Perceptron, madaline, and backpropagation. Proceedings of the IEEE 78(9), 1415–1442 (1990)
Wilson, S.W.: Perception redux: Emergence of structure. Physica D: Nonlinear Phenomena 42(1–3), 249–256 (1990)
Yao, X.: Evolving artificial neural networks. Proceedings of the IEEE 87(9), 1423–1447 (1999)
Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Transactions on Neural Networks 8(3), 694–713 (1997)
Zhang, C., Shao, H., Li, Y.: Particle swarm optimisation for evolving artificial neural network. In: Systems, Man, and Cybernetics, 2000 IEEE International Conference on, vol. 4, pp. 2487–2490. IEEE (2000)
Zhang, G., Patuwo, B.E., Hu, M.Y.: Forecasting with artificial neural networks:: The state of the art. International Journal of Forecasting 14(1), 35–62 (1998)
Acknowledgements
This work was partially supported by projects UID/MULTI/00308/2019 and by the European Regional Development Fund through the COMPETE 2020 Programme, FCT—Portuguese Foundation for Science and Technology and Regional Operational Program of the Center Region (CENTRO2020) within project MAnAGER (POCI-01-0145-FEDER-028040). This work was also partially supported by national funds through FCT (Fundação para a Ciência e a Tecnologia) under project DSAIPA/DS/0022/2018 (GADgET).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gonçalves, I., Seca, M., Castelli, M. (2020). Explorations of the Semantic Learning Machine Neuroevolution Algorithm: Dynamic Training Data Use, Ensemble Construction Methods, and Deep Learning Perspectives. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds) Genetic Programming Theory and Practice XVII. Genetic and Evolutionary Computation. Springer, Cham. https://doi.org/10.1007/978-3-030-39958-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-39958-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-39957-3
Online ISBN: 978-3-030-39958-0
eBook Packages: Computer ScienceComputer Science (R0)