Advertisement

Structurally Layered Representation Learning: Towards Deep Learning Through Genetic Programming

  • Lino Rodriguez-CoayahuitlEmail author
  • Alicia Morales-Reyes
  • Hugo Jair Escalante
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10781)

Abstract

We introduce a novel method for representation learning based on genetic programming (GP). Inspired into the way that deep neural networks learn descriptive/discriminative representations from raw data, we propose a structurally layered representation that allows GP to learn a feature space from large scale and high dimensional data sets. Previous efforts from the GP community for feature learning have focused on small data sets with a few input variables, also, most approaches rely on domain expert knowledge to produce useful representations. In this paper, we introduce the structurally layered GP formulation, together with an efficient scheme to explore the search space and show that this framework can be used to learn representations from large data sets of high dimensional raw data. As case of study we describe the implementation and experimental evaluation of an autoencoder developed under the proposed framework. Results evidence the benefits of the proposed framework and pave the way for the development of deep genetic programming.

Keywords

Representation learning Deep learning Feature extraction Genetic programming Evolutionary machine learning 

References

  1. 1.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. PAMI 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  2. 2.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  3. 3.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788 (1999)CrossRefzbMATHGoogle Scholar
  4. 4.
    Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.: Fisher discriminant analysis with kernels. In: Proceeding of Workshop on Neural Networks for Signal Processing (1999)Google Scholar
  5. 5.
    Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemometr. Intell. Lab. Syst. 2(1–3), 37–52 (1987)CrossRefGoogle Scholar
  6. 6.
    Koza, J.R.: Genetic Programming: On the Programming of Computers by means of Natural Selection, vol. 1. MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  7. 7.
    Gomez, G., Morales, E.: Automatic feature construction and a simple rule induction algorithm for skin detection. In: ICML Workshops (2004)Google Scholar
  8. 8.
    Garcia-Limon, M., Escalante, H.J., Morales, E., Morales-Reyes, A.: Simultaneous generation of prototypes and features through genetic programming. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 517–524. ACM (2014)Google Scholar
  9. 9.
    Limón García, M., Escalante, H.J., Morales, E., Pineda, L.V.: Class-specific feature generation for 1NN through genetic programming. In: Proceeding of ROPEC (2015)Google Scholar
  10. 10.
    Bot, M.C.J.: Feature extraction for the k-nearest neighbour classifier with genetic programming. In: Miller, J., Tomassini, M., Lanzi, P.L., Ryan, C., Tettamanzi, A.G.B., Langdon, W.B. (eds.) EuroGP 2001. LNCS, vol. 2038, pp. 256–267. Springer, Heidelberg (2001).  https://doi.org/10.1007/3-540-45355-5_20 CrossRefGoogle Scholar
  11. 11.
    Trujillo, L., Olague, G.: Synthesis of interest point detectors through genetic programming. In: Proceeding of GECCO, pp. 887–894. ACM (2006)Google Scholar
  12. 12.
    Shao, L., Liu, L., Li, X.: Feature learning for image classification via multiobjective genetic programming. IEEE Trans. Neural Netw. Learn. Syst. 25(7), 1359–1371 (2014)CrossRefGoogle Scholar
  13. 13.
    Rumelhart, D.E., Hinton, G., Williams, R.J., et al.: Learning representations by back-propagating errors. Cogn. Model. 5(3), 1 (1988)zbMATHGoogle Scholar
  14. 14.
    LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, pp. 396–404 (1990)Google Scholar
  15. 15.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  16. 16.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceeding of CVPR, pp. 1–9 (2015)Google Scholar
  17. 17.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  18. 18.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  19. 19.
    Hinton, G., Salakhutdinov, R.: Reducing the dimensionality of data with neural networks. science 313(5786), 504–507 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Zhang, Y., Rockett, P.I.: A generic optimising feature extraction method using multiobjective genetic programming. Appl. Soft Comput. 11(1), 1087–1097 (2011)CrossRefGoogle Scholar
  21. 21.
    Lin, J., Ke, H., Chien, B., Yang, W.: Designing a classifier by a layered multi-population genetic programming approach. Pattern Recogn. 40(8), 2211–2225 (2007)CrossRefzbMATHGoogle Scholar
  22. 22.
    Tran, B., Xue, B., Zhang, M.: Genetic programming for feature construction and selection in classification on high-dimensional data. Memet. Comput. 8(1), 3–15 (2016)CrossRefGoogle Scholar
  23. 23.
    Tran, B., Xue, B., Zhang, M.: Using feature clustering for GP-based feature construction on high-dimensional data. In: McDermott, J., Castelli, M., Sekanina, L., Haasdijk, E., García-Sánchez, P. (eds.) EuroGP 2017. LNCS, vol. 10196, pp. 210–226. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-55696-3_14 CrossRefGoogle Scholar
  24. 24.
    Parkins, A., Nandi, A.: Genetic programming techniques for hand written digit recognition. Signal Process. 84(12), 2345–2365 (2004)CrossRefGoogle Scholar
  25. 25.
    LeCun, Y.: Mnist database of handwritten digits (1998). http://yann.lecun.com/exdb/mnist/
  26. 26.
    Sanderson, C.: LFWcrop face dataset (2014)Google Scholar
  27. 27.
    Samaria, F.S., Harter, A.C.: Parameterisation of a stochastic model for human face identification. In: 1994 Proceedings of the Second IEEE Workshop on Applications of Computer Vision, pp. 138–142. IEEE (1994)Google Scholar
  28. 28.
    Abadi, M., Agarwal, A., et al.: Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Lino Rodriguez-Coayahuitl
    • 1
    Email author
  • Alicia Morales-Reyes
    • 1
  • Hugo Jair Escalante
    • 1
  1. 1.Instituto Nacional de Astrofisica, Optica y ElectronicaTonantzintlaMexico

Personalised recommendations