Skip to main content
SpringerLink
Log in

Residual Networks as Flows of Diffeomorphisms

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

This paper addresses the understanding and characterization of residual networks (ResNet), which are among the state-of-the-art deep learning architectures for a variety of supervised learning problems. We focus on the mapping component of ResNets, which map the embedding space toward a new unknown space where the prediction or classification can be stated according to linear criteria. We show that this mapping component can be regarded as the numerical implementation of continuous flows of diffeomorphisms governed by ordinary differential equations. In particular, ResNets with shared weights are fully characterized as numerical approximation of exponential diffeomorphic operators. We stress both theoretically and numerically the relevance of the enforcement of diffeomorphic properties and the importance of numerical issues to make consistent the continuous formulation and the discretized ResNet implementation. We further discuss the resulting theoretical and computational insights into ResNet architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. F. Chollet et al. Keras, 2015. https://keras.io.

References

  1. Arsigny, V.: Processing data in Lie groups: an algebraic approach. Application to non-linear registration and diffusion tensor MRI. Ph.D. thesis, Ecole Polytechnique (2006)

  2. Arsigny, V., Commowick, O., Pennec, X., Ayache, N.: A log-Euclidean framework for statistics on diffeomorphisms. In: International Conference on Medical Image Computing and Computer-Assisted Intervention: MICCAI, vol. 9(Pt 1), pp. 924–931 (2006)

  3. Ashburner, J.: A fast diffeomorphic image registration algorithm. NeuroImage 38(1), 95–113 (2007)

    Article  Google Scholar 

  4. Avants, B., Gee, J.C.: Geodesic estimation for large deformation anatomical shape averaging and interpolation. NeuroImage 23, S139–S150 (2004)

    Article  Google Scholar 

  5. Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)

    Article  Google Scholar 

  6. Bietti, A., Mairal, J.: Group invariance, stability to deformations, and complexity of deep convolutional representations. J. Mach. Learn. Res. 20(25), 1–49 (2019)

    MathSciNet  MATH  Google Scholar 

  7. Chang, B., Meng, L., Haber, E., Ruthotto, L., Begert, D., Holtham, E.: Reversible architectures for arbitrarily deep residual neural networks. In: Neural Network cs.CV (2017)

  8. Davis, P.J., Rabinowitz, P.: Methods of Numerical Integration. Academic Press, London (1984)

    MATH  Google Scholar 

  9. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS (2010)

  10. Gomez, A.N., Ren, M., Urtasun, R., Grosse, R.B.: The reversible residual network—backpropagation without storing activations. In: NIPS (2017)

  11. Gouk, H., Frank, E., Pfahringer, B., Cree, M.J.: Regularisation of neural networks by enforcing Lipschitz continuity. In: Neural Networks stat.ML (2018)

  12. Haber, E., Ruthotto, L.: Stable architectures for deep neural net works. Inverse Probl. (2018). https://doi.org/10.1088/1361-6420/aa9a90

  13. Hauser, M., Ray, A.: Principles of Riemannian geometry in neural networks. In: NIPS (2017)

  14. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE ICCV (2015). https://doi.org/10.1109/ICCV.2015.123

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE (2016)

  16. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: ECCV (2016). https://doi.org/10.1007/978-3-319-46493-0_38

  17. Jacobsen, J.H., Smeulders, A., Oyallon, E.: i-RevNet: deep invertible networks. In: International Conference on Learning Representations (ICLR) (2018)

  18. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)

  19. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: CVPR (2016)

  20. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014)

  21. Klein, A., Andersson, J., Ardekani, B.A., Ashburner, J., Avants, B., Chiang, M.C., Christensen, G.E., Collins, D.L., Gee, J., Hellier, P., Song, J.H., Jenkinson, M., Lepage, C., Rueckert, D., Thompson, P., Vercauteren, T., Woods, R.P., Mann, J.J., Parsey, R.V.: Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage 46(3), 786–802 (2009)

    Article  Google Scholar 

  22. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. University of Toronto (2009)

  23. Liao, Q., Poggio, T.A.: Bridging the gaps between residual learning, recurrent neural networks and visual cortex. In: Neural Networks (2016)

  24. Mallat, S.: Understanding deep convolutional networks. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 374(20), 20150203-16 (2016)

    Google Scholar 

  25. Ruthotto, L., Haber, E.: Deep neural networks motivated by partial differential equations (2018)

  26. Scholkopf, B., Smola, A.J.: Learning with kernels. Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002). https://mitpress.mit.edu/books/learning-kernels

  27. Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: a survey. IEEE Trans. Med. Imaging 32(7), 1153–1190 (2013)

    Article  Google Scholar 

  28. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: ICLR (2014)

  29. Thirion, J.: Image matching as a diffusion process: an analogy with Maxwell’s demons. Med. Image Anal. 2(3), 243–260 (1998)

    Article  Google Scholar 

  30. Trouvé, A.: Diffeomorphisms groups and pattern matching in image analysis. Int. J. Comput. Vis. 28(3), 213–221 (1998)

    Article  MathSciNet  Google Scholar 

  31. Weinan, E.: A proposal on machine learning via dynamical systems. Commun. Math. Stat. 5(1), 1–11 (2017)

    MathSciNet  MATH  Google Scholar 

  32. Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR, pp. 5987–5995 (2017)

  33. Younes, L.: Shapes and diffeomorphisms. Applied Mathematical Sciences, vol. 171. Springer, Berlin (2010)

  34. Younes, L.: Diffeomorphic learning (2018)

Download references

Acknowledgements

We thank B. Chapron and C. Herzet for their comments and suggestions. The research leading to these results has been supported by the ANR MAIA Project, Grant ANR-15-CE23-0009 of the French National Research Agency, INSERM and Institut Mines Télécom Atlantique (Chaire “Imagerie médicale en thérapie interventionnelle”) and Fondation pour la Recherche Médicale (FRM Grant DIC20161236453), Labex Cominlabs (Grant SEACS), CNES (Grant OSTST-MANATEE) and Microsoft trough AI-for-Earth EU Oceans Grant (AI4Ocean). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations

  1. IMT Atlantique, LaTIM, UMR INSERM 1101, 29238, Brest, France

    François Rousseau

  2. IMT Atlantique, Lab-STICC, UMR CNRS 6285, 29238, Brest, France

    Lucas Drumetz & Ronan Fablet

Authors
  1. François Rousseau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lucas Drumetz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ronan Fablet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to François Rousseau.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rousseau, F., Drumetz, L. & Fablet, R. Residual Networks as Flows of Diffeomorphisms. J Math Imaging Vis 62, 365–375 (2020). https://doi.org/10.1007/s10851-019-00890-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-019-00890-3

Keywords

Search

Navigation