Advertisement

A Preliminary Approach to Semi-supervised Learning in Convolutional Neural Networks Applying “Sleep-Wake” Cycles

  • Mikel Elkano
  • Humberto Bustince
  • Andrew Paplinski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10637)

Abstract

The scarcity of labeled data has limited the capacity of convolutional neural networks (CNNs) until not long ago and still represents a serious problem in a number of image processing applications. Unsupervised methods have been shown to perform well in feature extraction and clustering tasks, but further investigation on unsupervised solutions for CNNs is needed. In this work, we propose a bio-inspired methodology that applies a deep generative model to help the CNN take advantage of unlabeled data and improve its classification performance. Inspired by the human “sleep-wake cycles”, the proposed method divides the learning process into sleep and waking periods. During the waking period, both the generative model and the CNN learn from real training data simultaneously. When sleep begins, none of the networks receive real data and the generative model creates a synthetic dataset from which the CNN learns. The experimental results showed that the generative model was able to teach the CNN and improve its classification performance.

Keywords

Semi-supervised learning Sleep-wake cycles Variational autoencoders Convolutional neural networks Generative models Deep learning 

Notes

Acknowledgments

This work has been partially supported by the Spanish Ministry of Science and Technology under the project TIN2016-77356-P (AEI/FEDER, UE).

References

  1. 1.
    Diekelmann, S., Born, J.: The memory function of sleep. Nat. Rev. Neurosci. 11(2), 114–126 (2010)Google Scholar
  2. 2.
    Franklin, M.S., Zyphur, M.J.: The role of dreams in the evolution of the human mind. Evol. Psychol. 3(1), 59–78 (2005)CrossRefGoogle Scholar
  3. 3.
    Geib, B.R., Stanley, M.L., Dennis, N.A., Woldorff, M.G., Cabeza, R.: From hippocampus to whole-brain: the role of integrative processing in episodic memory retrieval. Hum. Brain Mapp. 38(4), 2242–2259 (2017)CrossRefGoogle Scholar
  4. 4.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778, January 2016Google Scholar
  5. 5.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)Google Scholar
  6. 6.
    Kingma, D., Rezende, D., Mohamed, S., Welling, M.: Semi-supervised learning with deep generative models. NIPS 4, 3581–3589 (2014)Google Scholar
  7. 7.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks, vol. 2, pp. 1097–1105 (2012)Google Scholar
  8. 8.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  9. 9.
    LeCun, Y., Cortes, C.: The MNIST database of handwritten digits (1998)Google Scholar
  10. 10.
    McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419–457 (1995)CrossRefGoogle Scholar
  11. 11.
    Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T.: Semi-supervised learning with ladder networks. In: NIPS, pp. 3546–3554, January 2015Google Scholar
  12. 12.
    Shinozaki, T.: Semi-supervised learning for convolutional neural networks using mild supervisory signals. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9950, pp. 381–388. Springer, Cham (2016). doi: 10.1007/978-3-319-46681-1_46 CrossRefGoogle Scholar
  13. 13.
    Wamsley, E.J.: Dreaming and offline memory consolidation. Curr. Neurol. Neurosci. Rep. 14(3), 433 (2014)CrossRefGoogle Scholar
  14. 14.
    Zhang, Y., Lee, K., Lee, H.: Augmenting supervised neural networks with unsupervised objectives for large-scale image classification, vol. 2, pp. 939–957 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Mikel Elkano
    • 1
    • 2
  • Humberto Bustince
    • 1
    • 2
  • Andrew Paplinski
    • 3
  1. 1.Department of Automatics and ComputationPublic University of NavarrePamplonaSpain
  2. 2.Institute of Smart CitiesPublic University of NavarrePamplonaSpain
  3. 3.Monash UniversityClayton, MelbourneAustralia

Personalised recommendations