Skip to main content
SpringerLink
Log in

Information Theoretical Analysis of Deep Learning Representations

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2015)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 9489))

Included in the following conference series:

Abstract

Although deep learning shows high performance in pattern recognition and machine learning, the reasons are little clarified. To tackle this problem, we calculated the information theoretical variables of representations in hidden layers and analyzed their relationship to the performance. We found that the entropy and the mutual information decrease in a different way as the layer gets deeper. This suggests that the information theoretical variables may become a criterion to determine the number of layers in deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comp. 12, 531–545 (2006)

    MathSciNet  MATH  Google Scholar 

  2. Seide, F., Li, G., Yu, D.: Conversational speech transcription using context-dependent deep neural networks. In: Proceedings Interspeech, pp. 437–440 (2011)

    Google Scholar 

  3. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part I. LNCS, vol. 8689, pp. 818–833. Springer, Heidelberg (2014)

    Google Scholar 

  4. Watanabe, S.: Algebraic analysis for nonidentifiable learning machines. Neural Comp. 13, 899–933 (2001)

    Article  MATH  Google Scholar 

  5. Fukumizu, K., Akaho, S., Amari, S.: Critical lines in symmetry of mixture models and its application to component splitting. NIPS 15, 857–864 (2003)

    Google Scholar 

  6. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? JMLR 11, 625–660 (2010)

    MathSciNet  MATH  Google Scholar 

  7. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580 (2012)

  8. Vincent, P., Larochella, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked demonising autoencoders: learning useful representations in a deep network with a local denoising criterion. JMLR 11, 3371–3408 (2010)

    MATH  Google Scholar 

  9. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Article  Google Scholar 

  10. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. NIPS 19, 153–160 (2007)

    Google Scholar 

  11. Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contracting auto-encoders: explicit invariance during feature extraction. In: Proceedings ICML, pp. 833–840 (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nara Institute of Science and Technology, Ikoma, Nara, 630-0192, Japan

    Yasutaka Furusho, Takatomi Kubo & Kazushi Ikeda

Authors
  1. Yasutaka Furusho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Takatomi Kubo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kazushi Ikeda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kazushi Ikeda .

Editor information

Editors and Affiliations

  1. University of Istanbul, Istanbul, Turkey

    Sabri Arik

  2. University at Qatar, Doha, Qatar

    Tingwen Huang

  3. Tunku Abdul Rahman University College, Kuala Lumpur, Malaysia

    Weng Kin Lai

  4. University of Science Technology, Wuhan, China

    Qingshan Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Furusho, Y., Kubo, T., Ikeda, K. (2015). Information Theoretical Analysis of Deep Learning Representations. In: Arik, S., Huang, T., Lai, W., Liu, Q. (eds) Neural Information Processing. ICONIP 2015. Lecture Notes in Computer Science(), vol 9489. Springer, Cham. https://doi.org/10.1007/978-3-319-26532-2_66

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-26532-2_66

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-26531-5

  • Online ISBN: 978-3-319-26532-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation