Skip to main content
SpringerLink
Log in

Can Generalised Divergences Help for Invariant Neural Networks?

  • Conference paper
  • First Online:
Geometric Science of Information (GSI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14071))

Included in the following conference series:

Abstract

We consider a framework including multiple augmentation regularisation by generalised divergences to induce invariance for non-group transformations during training of convolutional neural networks. Experiments on supervised classification of images at different scales not considered during training illustrate that our proposed method performs better than classical data augmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Altstidl, T., et al.: Just a matter of scale? reevaluating scale equivariance in convolutional neural networks. arXiv preprint arXiv:2211.10288 (2022)

  2. Andén, J., Mallat, S.: Deep scattering spectrum. IEEE Trans. Signal Process. 62(16), 4114–4128 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  3. Barnard, E., Casasent, D.: Invariance and neural nets. IEEE Trans. Neural Netw. 2(5), 498–508 (1991)

    Article  Google Scholar 

  4. Benton, G., Finzi, M., Izmailov, P., Wilson, A.G.: Learning invariances in neural networks from training data. Adv. Neural. Inf. Process. Syst. 33, 17605–17616 (2020)

    Google Scholar 

  5. Botev, A., Bauer, M., De, S.: Regularising for invariance to data augmentation improves supervised learning. arXiv preprint arXiv:2203.03304 (2022)

  6. Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P.: Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021)

  7. Cazelles, E., Tobar, F., Fontbona, J.: A novel notion of barycenter for probability distributions based on optimal weak mass transport. Adv. Neural. Inf. Process. Syst. 34, 13575–13586 (2021)

    Google Scholar 

  8. Cesa, G., Lang, L., Weiler, M.: A program to build E(N)-equivariant steerable CNNs. In: International Conference on Learning Representations (2022)

    Google Scholar 

  9. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    Article  MATH  Google Scholar 

  10. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)

    Google Scholar 

  11. Chen, W., Tian, L., Fan, L., Wang, Y.: Augmentation invariant training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0 (2019)

    Google Scholar 

  12. Cohen, T., Welling, M.: Group equivariant convolutional networks. In: International Conference on Machine Learning, pp. 2990–2999. PMLR (2016)

    Google Scholar 

  13. Fort, S., Brock, A., Pascanu, R., De, S., Smith, S.L.: Drawing multiple augmentation samples per image during training efficiently decreases test error. arXiv preprint arXiv:2105.13343 (2021)

  14. Fralick, S.: Learning to recognize patterns without a teacher. IEEE Trans. Inf. Theory 13(1), 57–64 (1967)

    Article  Google Scholar 

  15. Geiping, J., Goldblum, M., Somepalli, G., Shwartz-Ziv, R., Goldstein, T., Wilson, A.G.: How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.06441 (2022)

  16. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

    Google Scholar 

  17. Haasdonk, B., Vossen, A., Burkhardt, H.: Invariance in kernel methods by Haar-Integration kernels. In: Kalviainen, H., Parkkinen, J., Kaarna, A. (eds.) SCIA 2005. LNCS, vol. 3540, pp. 841–851. Springer, Heidelberg (2005). https://doi.org/10.1007/11499145_85

    Chapter  Google Scholar 

  18. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)

    Google Scholar 

  19. Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386 (2015)

  20. Kondor, R., Trivedi, S.: On the generalization of equivariance and convolution in neural networks to the action of compact groups. In: International Conference on Machine Learning, pp. 2747–2755. PMLR (2018)

    Google Scholar 

  21. Kullback, S.: Information theory and statistics. Courier Corporation (1997)

    Google Scholar 

  22. Laptev, D., Savinov, N., Buhmann, J.M., Pollefeys, M.: Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 289–297 (2016)

    Google Scholar 

  23. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  24. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  25. Mroueh, Y., Voinea, S., Poggio, T.A.: Learning with group invariant features: A kernel perspective. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  26. Penaud-Polge, V., Velasco-Forero, S., Angulo, J.: Fully trainable Gaussian derivative convolutional layer. In: 2022 IEEE International Conference on Image Processing (ICIP), pp. 2421–2425. IEEE (2022)

    Google Scholar 

  27. Penaud-Polge, V., Velasco-Forero, S., Angulo, J.: Genharris-resnet: A rotation invariant neural network based on elementary symmetric polynomials. In: Scale Space and Variational Methods in Computer Vision, pp. 149–161. Springer International Publishing, Cham (2023). https://doi.org/10.1007/978-3-031-31975-4_12

  28. Procesi, C.: Lie groups: an approach through invariants and representations, vol. 115. Springer (2007). https://doi.org/10.1007/978-0-387-28929-8

  29. Rota, G.C.: Reynolds operators. In: Proceedings of Symposia in Applied Mathematics. vol. 16, pp. 70–83. American Mathematical Society Providence, RI (1964)

    Google Scholar 

  30. Sangalli, M., Blusseau, S., Velasco-Forero, S., Angulo, J.: Scale equivariant neural networks with morphological scale-spaces. In: Lindblad, J., Malmberg, F., Sladoje, N. (eds.) DGMM 2021. LNCS, vol. 12708, pp. 483–495. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-76657-3_35

    Chapter  Google Scholar 

  31. Sangalli, M., Blusseau, S., Velasco-Forero, S., Angulo, J.: Moving frame net: SE (3)-equivariant network for volumes. In: NeurIPS Workshop on Symmetry and Geometry in Neural Representations, pp. 81–97. PMLR (2023)

    Google Scholar 

  32. Scudder, H.: Adaptive communication receivers. IEEE Trans. Inf. Theory 11(2), 167–174 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  33. Sgarro, A.: Informational divergence and the dissimilarity of probability distributions. Calcolo 18(3), 293–302 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  34. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. big data 6(1), 1–48 (2019)

    Article  Google Scholar 

  35. Sibson, R.: Information radius. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 14(2), 149–160 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  36. Smets, B.M., Portegies, J., Bekkers, E.J., Duits, R.: Pde-based group equivariant convolutional neural networks. J. Math. Imag. Vision 65(1), 209–239 (2023)

    Article  MathSciNet  MATH  Google Scholar 

  37. Wilk, M.v.d., Bauer, M., John, S., Hensman, J.: Learning invariances using the marginal likelihood. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 9960–9970 (2018)

    Google Scholar 

  38. Worrall, D., Welling, M.: Deep scale-spaces: Equivariance over scale. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  39. Yang, S., Dong, Y., Ward, R., Dhillon, I.S., Sanghavi, S., Lei, Q.: Sample efficiency of data augmentation consistency regularization. In: AISTATS (2023)

    Google Scholar 

  40. Zhang, H., Cissé, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: ICLR (2018)

    Google Scholar 

  41. Zhu, X., Ghahramani, Z., Lafferty, J.D.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912–919 (2003)

    Google Scholar 

Download references

Acknowledgements

This work was granted access to the Jean Zay supercomputer under the allocation 2022-AD011012212R2.

Author information

Authors and Affiliations

  1. MINES Paris-PSL-Research University-Centre de Morphologie Mathématique, Paris, France

    Santiago Velasco-Forero

Authors
  1. Santiago Velasco-Forero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santiago Velasco-Forero .

Editor information

Editors and Affiliations

  1. Sony Computer Science Laboratories Inc., Tokyo, Japan

    Frank Nielsen

  2. THALES Land and Air Systems, Meudon, France

    Frédéric Barbaresco

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Velasco-Forero, S. (2023). Can Generalised Divergences Help for Invariant Neural Networks?. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2023. Lecture Notes in Computer Science, vol 14071. Springer, Cham. https://doi.org/10.1007/978-3-031-38271-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-38271-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-38270-3

  • Online ISBN: 978-3-031-38271-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation