Semi-parametric training of autoencoders with Gaussian kernel smoothed topology learning neural networks
- 20 Downloads
Autoencoders are essential for training multi-hidden layer neural networks. Parametric autoencoder trainings often require user selections of hidden neuron numbers and kernel types. In this paper, a semi-parametric autoencoder training method based on self-organized learning and incremental learning is proposed. The cost function is constructed incrementally by nonparametric learning, and the model parameter is trained by parametric learning. First, a topology learning neural network such as growing neural gas or self-organizing incremental neural network is trained to obtain a discrete representation of the training data. Second, the correlations between different dimensions are modeled as a joint distribution by the neural network representation and kernel smoothers. Finally, the loss function is defined to be the regression prediction errors with each dimension as a response variable in density regression. The parameter of kernels is selected by gradient descent which minimizes the reconstruction error on a data subset. The proposed architecture has the advantage of high training space efficiency because of incremental training, and the advantage of automated selection of hidden neuron numbers. Experiments are carried out on 4 UCI datasets and an image interpolation task. Results show that the proposed methods outperform the perceptron architecture autoencoders and the restricted Boltzmann machine in the task of nonlinear feature learning.
KeywordsAutoencoder Nonparametric learning Kernel density estimation Incremental learning
This work was supported in part by Fundamental Research Program of Shenzhen (Project No. JCYJ20170413162458312) and National Natural Science Foundations of China (No. 61562047).
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- 1.Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mane D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viegas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. ArXiv e-printsGoogle Scholar
- 2.Bache K, Lichman M (2013) UCI machine learning repository, 901:1. http://archive.ics.uci.edu/ml. Accessed 25 Mar 2018
- 3.Bodin E, Malik I, Ek CH, Campbell NDF (2017) Nonparametric inference for auto-encoding variational Bayes. ArXiv e-printsGoogle Scholar
- 6.Csji BC (2001) Approximation with artificial neural networks. Ph.D. thesis, Faculty of Sciences, Etvs Lornd UniversityGoogle Scholar
- 9.Fritzke B et al (1995) A growing neural gas network learns topologies. Adv Neural Inf Process Syst 7:625–632Google Scholar
- 12.Kingma DP, Welling M (2013) Auto-encoding variational Bayes. ArXiv e-printsGoogle Scholar
- 14.Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th international conference on neural information processing systems, vol 1, NIPS’12, pp 1097–1105. Curran Associates Inc., USA. http://dl.acm.org/citation.cfm?id=2999134.2999257. Accessed 25 Mar 2018
- 15.Nalisnick E, Smyth P (2017) Stick-breaking variational autoencoders. In: International conference on learning representations (ICLR). http://par.nsf.gov/biblio/10039928. Accessed 25 Mar 2018
- 16.Oliphant TE (2015) Guide to NumPy, 2nd edn. CreateSpace Independent Publishing Platform, Scotts ValleyGoogle Scholar
- 26.Tieleman T (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th international conference on machine learning, ICML ’08, pp 1064–1071. ACM, New York, NY, USA. https://doi.org/10.1145/1390156.1390290
- 33.Xin M, Zhang H, Sun M, Yuan D (2016) Recurrent temporal sparse autoencoder for attention-based action recognition. In: 2016 International joint conference on neural networks (IJCNN), pp 456–463. https://doi.org/10.1109/IJCNN.2016.7727234
- 34.Yang H, Wang B, Lin S, Wipf D, Guo M, Guo B (2015) Unsupervised extraction of video highlights via robust recurrent auto-encoders. In: 2015 IEEE international conference on computer vision (ICCV), pp 4633–4641. https://doi.org/10.1109/ICCV.2015.526
- 36.Zhao W, Xu L, Bai J, Ji M, Runge T (2017) Sensor-based risk perception ability network design for drivers in snow and ice environmental freeway: a deep learning and rough sets approach. Soft Comput 2:1–10Google Scholar