Learning Deep Belief Networks from Non-stationary Streams

  • Roberto Calandra
  • Tapani Raiko
  • Marc Peter Deisenroth
  • Federico Montesino Pouzols
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7553)


Deep learning has proven to be beneficial for complex tasks such as classifying images. However, this approach has been mostly applied to static datasets. The analysis of non-stationary (e.g., concept drift) streams of data involves specific issues connected with the temporal and changing nature of the data. In this paper, we propose a proof-of-concept method, called Adaptive Deep Belief Networks, of how deep learning can be generalized to learn online from changing streams of data. We do so by exploiting the generative properties of the model to incrementally re-train the Deep Belief Network whenever new data are collected. This approach eliminates the need to store past observations and, therefore, requires only constant memory consumption. Hence, our approach can be valuable for life-long learning from non-stationary data streams.


Incremental Learning Adaptive Learning Non-stationary Learning Concept drift Deep Learning Deep Belief Networks Generative model Generating samples Adaptive Deep Belief Networks 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, C.C., Han, J., Wang, J., Yu, P.S.: On demand classification of data streams. In: Proceedings of KDD 2004, pp. 503–508 (2004)Google Scholar
  2. 2.
    Angelov, P., Filev, D.P., Kasabov, N.: Evolving Intelligent Systems: Methodology and Applications. Wiley-IEEE Press (2010)Google Scholar
  3. 3.
    Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Proceedings of NIPS 2006, vol. 19, pp. 153–160 (2006)Google Scholar
  4. 4.
    Bifet, A. (ed.): Proceeding of the 2010 Conference on Adaptive Stream Mining: Pattern Learning and Mining from Evolving Data Streams (2010)Google Scholar
  5. 5.
    Cho, K., Raiko, T., Ilin, A.: Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines. In: Proceedings of ICML 2011, pp. 105–112 (2011)Google Scholar
  6. 6.
    Erhan, D., Courville, A., Bengio, Y., Vincent, P.: Why does unsupervised pre-training help deep learning? In: Proceedings of AISTATS 2010, pp. 201–208.Google Scholar
  7. 7.
    Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., Vincent, P.: The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Proceedings of AISTATS 2009, pp. 153–160 (2009)Google Scholar
  8. 8.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18(7), 1527–1554 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Igel, C., Hüsken, M.: Improving the RPROP learning algorithm. In: Proceedings of NC 2000, pp. 115–121 (2000)Google Scholar
  10. 10.
    Last, M.: Online classification of nonstationary data streams. Intelligent Data Analysis 6(2), 129–147 (2002)zbMATHMathSciNetGoogle Scholar
  11. 11.
    LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010),
  12. 12.
    Quiñonero Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press (2009)Google Scholar
  13. 13.
    Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: IEEE International Conference on Neural Networks, vol. 1, pp. 586–591 (1993)Google Scholar
  14. 14.
    Salakhutdinov, R.: Learning deep generative models. PhD thesis, University of Toronto (2009)Google Scholar
  15. 15.
    Zliobaite, I.: Learning under concept drift: an overview. CoRR (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Roberto Calandra
    • 1
  • Tapani Raiko
    • 2
  • Marc Peter Deisenroth
    • 1
  • Federico Montesino Pouzols
    • 3
  1. 1.Fachbereich InformatikTechnische Universität DarmstadtGermany
  2. 2.Department of Information and Computer ScienceAalto UniversityFinland
  3. 3.Department of BiosciencesUniversity of HelsinkiFinland

Personalised recommendations