Skip to main content
Book cover

Southern African Conference for Artificial Intelligence Research

SACAIR 2022: Artificial Intelligence Research pp 31–45Cite as

Jacobian Norm Regularisation and Conditioning in Neural ODEs

  • 182 Accesses

Part of the Communications in Computer and Information Science book series (CCIS,volume 1734)

Abstract

A recent line of work regularises the dynamics of neural ordinary differential equations (neural ODEs), in order to reduce the number of function evaluations needed by a numerical ODE solver during training. For instance, in the context of continuous normalising flows, the Frobenius norm of Jacobian matrices are regularised under the hypothesis that complex dynamics relate to an ill-conditioned ODE and require more function evaluations from the solver. Regularising the Jacobian norm also relates to sensitivity analysis in the broader neural network literature, where it is believed that regularised models should be more robust to random and adversarial perturbations in their input. We investigate the conditioning of neural ODEs under different Jacobian regularisation strategies, in a binary classification setting. Regularising the Jacobian norm indeed reduces the number of function evaluations required, but at a cost to generalisation. Moreover, naively regularising the Jacobian norm can make the ODE system more ill-conditioned, contrary to what is believed in the literature. As an alternative, we regularise the condition number of the Jacobian and observe a lower number of function evaluations without a significant decrease in generalisation performance. We also find that Jacobian regularisation does not guarantee adversarial robustness, but it can lead to larger margin classifiers.

Keywords

  • Neural ODEs
  • Regularisation
  • Sensitivity

This work is based on research supported by the National Research Foundation of South Africa (grant number 138341).

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-22321-1_3
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   69.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-22321-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   89.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.

References

  1. Bai, S., Kolter, J.Z., Koltun, V.: Deep equilibrium models. In: Advances in Neural Information Processing Systems, vol. 32, pp. 690–701 (2019)

    Google Scholar 

  2. Bai, S., Koltun, V., Kolter, J.Z.: Stabilizing equilibrium models by Jacobian regularization. In: International Conference on Machine Learning, vol. 139, pp. 554–565 (2021)

    Google Scholar 

  3. Chen, R.T.: Torchdiffeq: PyTorch implementation of differentiable ODE solvers (2018). https://github.com/rtqichen/torchdiffeq

  4. Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems, vol. 31, pp. 6572–6583 (2018)

    Google Scholar 

  5. Coddington, E.A., Levinson, N.: Theory of Ordinary Differential Equations. Tata McGraw-Hill Education (1955)

    Google Scholar 

  6. De Brouwer, E., Simm, J., Arany, A., Moreau, Y.: GRU-ODE-Bayes: continuous modeling of sporadically-observed time series. In: Advances in Neural Information Processing Systems, vol. 32, pp. 7377–7388 (2019)

    Google Scholar 

  7. Dupont, E., Doucet, A., Teh, Y.W.: Augmented neural ODEs. In: Advances in Neural Information Processing Systems, vol. 32, pp. 3134–3144 (2019)

    Google Scholar 

  8. Falorsi, L., Forré, P.: Neural ordinary differential equations on manifolds. arXiv:2006.06663 (2020)

  9. Finlay, C., Jacobsen, J.H., Nurbekyan, L., Oberman, A.: How to train your neural ODE: the world of Jacobian and kinetic regularization. In: International Conference on Machine Learning, vol. 119, pp. 3154–3164 (2020)

    Google Scholar 

  10. Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (2015)

    Google Scholar 

  11. Gu, S., Rigazio, L.: Towards deep neural network architectures robust to adversarial examples. In: International Conference on Learning Representations, ICLR Workshop Track Proceedings (2015)

    Google Scholar 

  12. Hanshu, Y., Jiawei, D., Vincent, T., Jiashi, F.: On robustness of neural ordinary differential equations. In: International Conference on Learning Representations (2020)

    Google Scholar 

  13. Hoffman, J., Roberts, D.A., Yaida, S.: Robust learning with Jacobian regularization. arXiv:1908.02729 (2019)

  14. Johansson, A., Strannegård, C., Engsner, N., Mostad, P.: Exact spectral norm regularization for neural networks. arXiv:2206.13581 (2022)

  15. Kelly, J., Bettencourt, J., Johnson, M.J., Duvenaud, D.K.: Learning differential equations that are easy to solve. In: Advances in Neural Information Processing Systems, vol. 33, pp. 4370–4380 (2020)

    Google Scholar 

  16. Kidger, P., Morrill, J., Foster, J., Lyons, T.: Neural controlled differential equations for irregular time series. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6696–6707 (2020)

    Google Scholar 

  17. Lou, A., et al.: Neural manifold ordinary differential equations. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17548–17558 (2020)

    Google Scholar 

  18. Massaroli, S., Poli, M., Park, J., Yamashita, A., Asama, H.: Dissecting neural ODEs. In: Advances in Neural Information Processing Systems, vol. 33, pp. 3952–3963 (2020)

    Google Scholar 

  19. Mathieu, E., Nickel, M.: Riemannian continuous normalizing flows. In: Advances in Neural Information Processing Systems, vol. 33, pp. 2503–2515 (2020)

    Google Scholar 

  20. Miles, C., Sam, G., Stephan, H., Peter, B., David, S., Shirley, H.: Lagrangian neural networks. In: International Conference on Learning Representations, Workshop on Integration of Deep Neural Models and Differential Equations (2020)

    Google Scholar 

  21. Rubanova, Y., Chen, R.T., Duvenaud, D.K.: Latent ordinary differential equations for irregularly-sampled time series. In: Advances in Neural Information Processing Systems, vol. 32, pp. 5321–5331 (2019)

    Google Scholar 

  22. Sanchez-Gonzalez, A., Bapst, V., Cranmer, K., Battaglia, P.: Hamiltonian graph networks with ODE integrators. arXiv:1909.12790 (2019)

  23. Sokolić, J., Giryes, R., Sapiro, G., Rodrigues, M.R.: Robust large margin deep neural networks. IEEE Trans. Signal Process. 65(16), 4265–4280 (2017)

    CrossRef  MathSciNet  MATH  Google Scholar 

  24. Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: International Conference on Learning Representations (2020)

    Google Scholar 

  25. Wang, W., Axelrod, S., Gómez-Bombarelli, R.: Differentiable molecular simulations for control and learning. In: International Conference on Learning Representations, Workshop on Integration of Deep Neural Models and Differential Equations (2020)

    Google Scholar 

  26. Yoshida, Y., Miyato, T.: Spectral norm regularization for improving the generalizability of deep learning. arXiv:1705.10941 (2017)

  27. Zhong, Y.D., Dey, B., Chakraborty, A.: Symplectic ODE-net: learning Hamiltonian dynamics with control. In: International Conference on Learning Representations (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Applied Mathematics, Stellenbosch University, Stellenbosch, South Africa

    Shane Josias & Willie Brink

  2. School for Data Science and Computational Thinking, Stellenbosch University, Stellenbosch, South Africa

    Shane Josias

Authors
  1. Shane Josias
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Willie Brink
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shane Josias .

Editor information

Editors and Affiliations

  1. University of KwaZulu-Natal, Durban, South Africa

    Anban Pillay

  2. University of KwaZulu-Natal, Durban, South Africa

    Edgar Jembere

  3. University of Pretoria, Pretoria, South Africa

    Aurona Gerber

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Josias, S., Brink, W. (2022). Jacobian Norm Regularisation and Conditioning in Neural ODEs. In: Pillay, A., Jembere, E., Gerber, A. (eds) Artificial Intelligence Research. SACAIR 2022. Communications in Computer and Information Science, vol 1734. Springer, Cham. https://doi.org/10.1007/978-3-031-22321-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22321-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22320-4

  • Online ISBN: 978-3-031-22321-1

  • eBook Packages: Computer ScienceComputer Science (R0)