Abstract
Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some inputs and hidden units in the learning process with a probability, p, and then, the neglected inputs and hidden units are combined with the learned network to express the final output. We find that the process of combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyze dropout learning from this point of view.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)
Deng, L., Li, J., et al.: Recent advances in deep learning for speech research at Microsoft. In: ICASSP (2013)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. The Computing Research Repository CoRR, abs/ 1207.0580 (2012)
Hara, K., Okada, M.: Ensemble learning of linear perceptrons: on-line learning theory. J. Phys. Soc. Jpn. 74(11), 2966–2972 (2005)
Biehl, M., Schwarze, H.: Learning by on-line gradient descent. J. Phys. A Math. General Phys. 28, 643–656 (1995)
Saad, D., Solla, S.A.: On-line learning in soft-committee machines. Phys. Rev. E 52, 4225–4243 (1995)
Wager, S., Wang, S., Liang, P.: Dropout training as adaptive regularization. Adv. Neural Inf. Process. Syst. 26, 351–359 (2013)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)
Acknowledgments
The authors thank Dr. Masato Okada and Dr. Hideitsu Hino for insightful discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Hara, K., Saitoh, D., Shouno, H. (2016). Analysis of Dropout Learning Regarded as Ensemble Learning. In: Villa, A., Masulli, P., Pons Rivero, A. (eds) Artificial Neural Networks and Machine Learning – ICANN 2016. ICANN 2016. Lecture Notes in Computer Science(), vol 9887. Springer, Cham. https://doi.org/10.1007/978-3-319-44781-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-44781-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-44780-3
Online ISBN: 978-3-319-44781-0
eBook Packages: Computer ScienceComputer Science (R0)