Abstract
Convolutional neural networks (CNNs) are typical structures for deep learning and are widely used in image recognition and classification. However, the random initialization strategy tends to become stuck at local plateaus or even diverge, which results in rather unstable and ineffective solutions in real applications. To address this limitation, we propose a hybrid deep learning CNN-AdapDAE model, which applies the features learned by the AdapDAE algorithm to initialize CNN filters and then train the improved CNN for classification tasks. In this model, AdapDAE is proposed as a CNN pre-training procedure, which adaptively obtains the noise level based on the principle of annealing, by starting with a high level of noise and lowering it as the training progresses. Thus, the features learned by AdapDAE include a combination of features at different levels of granularity. Extensive experimental results on STL-10, CIFAR-10, andMNIST datasets demonstrate that the proposed algorithm performs favorably compared to CNN (random filters), CNNAE (pre-training filters by autoencoder), and a few other unsupervised feature learning methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7): 1527–1554
Salakhutdinov R, Larochelle H. Efficient learning of deep Boltzmann machines. ResearchGate, 2010, 9(8): 693–700
LeCun Y, Boser B, Denker J S, Henderson D, Howard R E, HubbardW, Jackel L D. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 1(4): 541–551
Tan S Q, Li B. Stacked convolutional auto-encoders for steganalysis of digital images. In: Proceedings of Asia-Pacific Signal and Information Processing Association. 2014, 1–4
Erhan D, Bengio Y, Courville A, Manzagol P A, Vincent P, Bengio S. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 2010, 11(3): 625–660
Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009, 2(1): 1–127
Masci J, Meier U, Ciresan D, Schmidhuber J. Stacked convolutional auto-encoders for hierarchical feature extraction. In: Proceedings of the 21st International Conference on Artificial Neural Networks. 2011, 52–59
Lee H, Grosse R, Ranganath R, Ng A Y. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of International Conference on Machine Learning. 2009, 609–616
Ji M Q, Fang L, Zheng H T, Strese M, Steinbach E. Preprocessing-free surface material classification using convolutional neural networks pretrained by sparse Autoencoder. In: Proceedings of the 25th IEEE International Workshop on Machine Learning for Signal Processing. 2015
Coates A, Ng A Y, Lee H. An analysis of single-layer networks in unsupervised feature learning. Journal of Machine Learning Research, 2011, 15: 215–223
Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. Technical Report. 2009
Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504–507
Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P A. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 2010, 11(6): 3371–3408
Olshausen B A, Field D J. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research, 1997, 37(23): 3311–3325
Ranzato M, Boureau Y L, Lecun Y. Sparse feature learning for deep belief networks. Advances in Neural Information Processing Systems, 2007, 1185–1192
Lee H, Ekanadham C, Ng A Y. Sparse deep belief net model for visual area V2. Advances in Neural Information Processing Systems, 2008, 20: 873–880
Dahl J V, Koch K C, Kleinhans E, Ostwald E, Schulz G, Buell U, Hanrath P. Convolutional networks and applications in vision. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2010, 253–256
Agarwal A, Triggs B. Hyperfeatures -multilevel local coding for visual recognition. In: Proceedings of European Conference on Computer Vision. 2006, 30–43
Geras K J, Sutton C. Scheduled denoising autoencoders. 2014, arXiv preprint arXiv:1406.3269
Chandra B, Sharma R K. Adaptive noise schedule for denoising autoencoder. In: Proceedings of International Conference on Neural Information Processing. 2014, 535–542
Coates A, Ng A Y. Selecting receptive fields in deep networks. Advances in Neural Information Processing Systems, 2011, 2528–2536
Hui K Y. Direct modeling of complex invariances for visual object features. In: Proceedings of International Conference on Machine Learning. 2013
Dosovitskiy A, Springenberg J T, Riedmiller M, Brox T. Discriminative unsupervised feature learning with convolutional neural networks. Advances in Neural Information Processing Systems, 2014, 766–774
Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning ap plied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324
Krizhevsky A. Convolutional deep belief networks on CIFAR-10. Technical Report. 2010
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant Nos. 61322203 and 61332002).
Author information
Authors and Affiliations
Corresponding author
Additional information
Qianjun Zhang received her BS and MS degrees in computer science from Xidian University, China in 2008 and 2011, respectively. Currently, she is working toward a PhD degree at the Machine Intelligence Laboratory, College of Computer Science, Sichuan University, China. Her current research interests include big data, neural networks, and deep learning.
Lei Zhang received the BS and MS degrees in mathematics and the PhD degree in computer science from the University of Electronic Science and Technology of China, China in 2002, 2005, and 2008, respectively. She was a post-doctoral research fellow with the Department of Computer Science and Engineering, Chinese University of Hong Kong, China from 2008 to 2009. She is currently a professor with Sichuan University, China. Her current research interests include theory and applications of neural networks based on neocortex computing and big data analysis methods by infinity deep neural networks.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Zhang, Q., Zhang, L. Convolutional adaptive denoising autoencoders for hierarchical feature extraction. Front. Comput. Sci. 12, 1140–1148 (2018). https://doi.org/10.1007/s11704-016-6107-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-016-6107-0