A Preliminary Approach to Semi-supervised Learning in Convolutional Neural Networks Applying “Sleep-Wake” Cycles
The scarcity of labeled data has limited the capacity of convolutional neural networks (CNNs) until not long ago and still represents a serious problem in a number of image processing applications. Unsupervised methods have been shown to perform well in feature extraction and clustering tasks, but further investigation on unsupervised solutions for CNNs is needed. In this work, we propose a bio-inspired methodology that applies a deep generative model to help the CNN take advantage of unlabeled data and improve its classification performance. Inspired by the human “sleep-wake cycles”, the proposed method divides the learning process into sleep and waking periods. During the waking period, both the generative model and the CNN learn from real training data simultaneously. When sleep begins, none of the networks receive real data and the generative model creates a synthetic dataset from which the CNN learns. The experimental results showed that the generative model was able to teach the CNN and improve its classification performance.
KeywordsSemi-supervised learning Sleep-wake cycles Variational autoencoders Convolutional neural networks Generative models Deep learning
This work has been partially supported by the Spanish Ministry of Science and Technology under the project TIN2016-77356-P (AEI/FEDER, UE).
- 1.Diekelmann, S., Born, J.: The memory function of sleep. Nat. Rev. Neurosci. 11(2), 114–126 (2010)Google Scholar
- 4.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778, January 2016Google Scholar
- 5.Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)Google Scholar
- 6.Kingma, D., Rezende, D., Mohamed, S., Welling, M.: Semi-supervised learning with deep generative models. NIPS 4, 3581–3589 (2014)Google Scholar
- 7.Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks, vol. 2, pp. 1097–1105 (2012)Google Scholar
- 9.LeCun, Y., Cortes, C.: The MNIST database of handwritten digits (1998)Google Scholar
- 11.Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T.: Semi-supervised learning with ladder networks. In: NIPS, pp. 3546–3554, January 2015Google Scholar
- 12.Shinozaki, T.: Semi-supervised learning for convolutional neural networks using mild supervisory signals. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9950, pp. 381–388. Springer, Cham (2016). doi: 10.1007/978-3-319-46681-1_46 CrossRefGoogle Scholar
- 14.Zhang, Y., Lee, K., Lee, H.: Augmenting supervised neural networks with unsupervised objectives for large-scale image classification, vol. 2, pp. 939–957 (2016)Google Scholar