Advertisement

Skipping Two Layers in ResNet Makes the Generalization Gap Smaller than Skipping One or No Layer

  • Yasutaka FurushoEmail author
  • Tongliang Liu
  • Kazushi Ikeda
Conference paper
Part of the Proceedings of the International Neural Networks Society book series (INNS, volume 1)

Abstract

The ResNet skipping two layers (ResNet2) is known to have a smaller expected risk than that skipping one layer (ResNet1) or no layer (MLP), however, the mechanism of the small expected risk is still unclear. The expected risk is divided into the three components, the generalization gap, the optimization error, and the sample expressivity, and the last two components are known to contribute the fast convergence of ResNet. We calculated the first component, the generalization gap, in the linear case, and show that ResNet2 has a smaller generalization gap than ResNet1 or MLP. Our numerical experiments confirmed the validity of our analysis and the applicability to the case with the ReLU activation function.

Keywords

Deep neural network ResNet Skip-connection Generalization gap Loss landscape 

Notes

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number JP18J15055, JP18K19821, and NAIST Big Data Project.

References

  1. 1.
    Bartlett, P.L., Helmbold, D.P., Long, P.M.: Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In: International Conference on Machine Learning (2018)Google Scholar
  2. 2.
    Bengio, Y., et al.: Learning deep architectures for AI. Found. Trends® Mach. Learn. 2(1), 1–127 (2009)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bousquet, O., Elisseeff, A.: Stability and generalization. J. Mach. Learn. Res. 2(Mar), 499–526 (2002)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Charles, Z., Papailiopoulos, D.: Stability and generalization of learning algorithms that converge to global optima. In: International Conference on Machine Learning (2018)Google Scholar
  5. 5.
    Hardt, M., Ma, T.: Identity matters in deep learning. In: International Conference on Learning Representations (2017)Google Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645. Springer (2016)Google Scholar
  8. 8.
    Kawaguchi, K.: Deep learning without poor local minima. In: Advances in Neural Information Processing Systems, pp. 586–594 (2016)Google Scholar
  9. 9.
    Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: generalization gap and sharp minima. In: International Conference on Learning Representations (2017)Google Scholar
  10. 10.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)CrossRefGoogle Scholar
  11. 11.
    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.R.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp. 9–48. Springer (2012)Google Scholar
  12. 12.
    Li, H., Xu, Z., Taylor, G., Goldstein, T.: Visualizing the loss landscape of neural nets. In: International Conference on Learning Representations (2018)Google Scholar
  13. 13.
    Li, S., Jiao, J., Han, Y., Weissman, T.: Demystifying ResNet. arXiv preprint arXiv:1611.01186 (2016)
  14. 14.
    Montufar, G.F., Pascanu, R., Cho, K., Bengio, Y.: On the number of linear regions of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2924–2932 (2014)Google Scholar
  15. 15.
    Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., Sohl-Dickstein, J.: On the expressive power of deep neural networks. In: International Conference on Machine Learning, pp. 2847–2854 (2017)Google Scholar
  16. 16.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Saxe, A.M., McClelland, J.L., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In: International Conference on Learning Representations (2014)Google Scholar
  18. 18.
    Telgarsky, M.: Benefits of depth in neural networks. In: Conference on Learning Theory, pp. 1517–1539 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Nara Institute of Science and TechnologyNaraJapan
  2. 2.The University of SydneyDarlingtonAustralia

Personalised recommendations