Skip to main content
SpringerLink
Log in

Randomized Newton’s Method for Solving Differential Equations Based on the Neural Network Discretization

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

We develop a randomized Newton’s method for solving differential equations, based on a fully connected neural network discretization. In particular, the randomized Newton’s method randomly chooses equations from the overdetermined nonlinear system resulting from the neural network discretization and solves the nonlinear system adaptively. We theoretically prove that the randomized Newton’s method has a quadratic convergence locally. We also apply this new method to various numerical examples, from one to high-dimensional differential equations, to verify its feasibility and efficiency. Moreover, the randomized Newton’s method can allow the neural network to “learn” multiple solutions for nonlinear systems of differential equations, such as pattern formation problems, and provides an alternative way to study the solution structure of nonlinear differential equations overall.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

References

  1. Ames, W.F.: Nonlinear partial differential equations in engineering, vol. 18. Academic press (1965)

  2. Baranyi, J., Roberts, T.A., McClure, P.: A non-autonomous differential equation to model bacterial growth. Food Microbiol. 10(1), 43–59 (1993)

    Article  Google Scholar 

  3. Bramble, J.H., Pasciak, J.E., Wang, J., Xu, J.: Convergence estimates for multigrid algorithms without regularity assumptions. Math. Comput. 57(195), 23–45 (1991)

    Article  MathSciNet  Google Scholar 

  4. Bungartz, Hans-Joachim., Griebel, Michael: Sparse grids. Acta Numer 13, 147–269 (2004)

    Article  MathSciNet  Google Scholar 

  5. Chen, J., Lopes, M.: Estimating the error of randomized newton methods: A bootstrap approach. In: International Conference on Machine Learning, pp. 1649–1659. PMLR (2020)

  6. Chou, C., Shu, C.: High order residual distribution conservative finite difference weno schemes for steady state problems on non-smooth meshes. J. Comput. Phys. 214(2), 698–724 (2006)

    Article  MathSciNet  Google Scholar 

  7. Chou, C., Shu, C.: High order residual distribution conservative finite difference weno schemes for convection-diffusion steady state problems on non-smooth meshes. J. Comput. Phys. 224(2), 992–1020 (2007)

    Article  MathSciNet  Google Scholar 

  8. Courant, R., Hilbert, D.: Methods of Mathematical Physics: Partial Differential Equations. John Wiley & Sons (2008)

  9. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems 2(4), 303–314 (1989)

    Article  MathSciNet  Google Scholar 

  10. De Lellis, C., Otto, F., Westdickenberg, M.: Minimal entropy conditions for burgers equation. Q. Appl. Math. 62(4), 687–700 (2004)

    Article  MathSciNet  Google Scholar 

  11. Della-Dora, J.: Numerical linear algorithms and group theory. Linear Algebra Appl. 10(3), 267–283 (1975)

    Article  MathSciNet  Google Scholar 

  12. Han, W.E.J., Jentzen, A.: Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics 5(4), 349–380 (2017)

    Article  MathSciNet  Google Scholar 

  13. Evans, L.C.: Partial Differential Equations. Graduate studies in mathematics. American Mathematical Society (2010)

  14. Farrell, P.E., Birkisson, A., Funke, S.W.: Deflation techniques for finding distinct solutions of nonlinear partial differential equations. SIAM J. Sci. Comput. 37(4), A2026–A2045 (2015)

    Article  MathSciNet  Google Scholar 

  15. Fedosov, D.A., Lei, H., Caswell, B., Suresh, S., Karniadakis, G.E.: Multiscale modeling of red blood cell mechanics and blood flow in malaria. PLoS Comput. Biol. 7(12), e1002270 (2011)

    Article  MathSciNet  Google Scholar 

  16. Forsyth, D.A., Ponce, J.: Computer vision: a modern approach. Prentice Hall Professional Technical Reference (2002)

  17. Gierer, A., Meinhardt, H.: A theory of biological pattern formation. Kybernetik 12(1), 30–39 (1972)

    Article  Google Scholar 

  18. Hao, W., Hauenstein, J.D., Hu, B., Sommese, A.J.: A bootstrapping approach for computing multiple solutions of differential equations. J. Comput. Appl. Math. 258, 181–190 (2014)

    Article  MathSciNet  Google Scholar 

  19. Hao, W., Hauenstein, J.D., Shu, C., Sommese, A.J., Xu, Z., Zhang, Y.: A homotopy method based on weno schemes for solving steady state problems of hyperbolic conservation laws. J. Comput. Phys. 250, 332–346 (2013)

    Article  MathSciNet  Google Scholar 

  20. He, J., Li, L., Xu, J., Zheng, C.: Relu deep neural networks and linear finite elements. arXiv preprint arXiv:1807.03973 (2018)

  21. Hu, B.: Blow-up theories for semilinear parabolic equations. Springer (2011)

  22. Karniadakis, G., Sherwin, S.: Spectral/hp element methods for computational fluid dynamics. Oxford University Press (2013)

  23. Kloeden, P.E., Platen, E.: Numerical solution of stochastic differential equations, vol. 23. Springer Science & Business Media (2013)

  24. Kondo, S., Miura, T.: Reaction-diffusion model as a framework for understanding biological pattern formation. Science 329(5999), 1616–1620 (2010)

    Article  MathSciNet  Google Scholar 

  25. Lee, K.J., McCormick, W.D., Ouyang, Q., Swinney, H.L.: Pattern formation by interacting chemical fronts. Science 261(5118), 192–194 (1993)

    Article  Google Scholar 

  26. Lei, H., Karniadakis, G.E.: Quantifying the rheological and hemodynamic characteristics of sickle cell anemia. Biophys. J . 102(2), 185–194 (2012)

    Article  Google Scholar 

  27. Leykin, A.: Numerical algebraic geometry. Journal of Software for Algebra and Geometry 3(1), 5–10 (2011)

    Article  MathSciNet  Google Scholar 

  28. Li, Q., Tai, C., W.E.: Stochastic modified equations and adaptive stochastic gradient algorithms. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2101–2110. JMLR. org (2017)

  29. Liu, Y., Cheng, Y., Chen, S., Zhang, Y.: Krylov implicit integration factor discontinuous galerkin methods on sparse grids for high dimensional reaction-diffusion equations. J. Comput. Phys. 388, 90–102 (2019)

    Article  MathSciNet  Google Scholar 

  30. Maini, P.K., Woolley, T.E., Baker, R.E., Gaffney, E.A., Lee, S.S.: Turing’s model for biological pattern formation and the robustness problem. Interface focus 2(4), 487–496 (2012)

    Article  Google Scholar 

  31. Manning, C.D., Schütze, H.: Foundations of statistical natural language processing. MIT press (1999)

  32. Marion, M., Xu, J.: Error estimates on a new nonlinear galerkin method based on two-grid finite elements. SIAM J. Numer. Anal. 32(4), 1170–1184 (1995)

    Article  MathSciNet  Google Scholar 

  33. Medler, D.A., Dawson, M.R.W.: Training redundant artificial neural networks: Imposing biology on technology. Psychol. Res. 57(1), 54–62 (1994)

    Article  Google Scholar 

  34. Medler, D.A., Dawson, M.R.W.: Using redundancy to improve the performance of artificial neural networks. In: Proceedings of the Biennial Conference-Canadian Society for Computational Studies of Intelligence, pages 131–138. CANADIAN INFORMATION PROCESSING SOCIETY (1994)

  35. Nobile, F., Tempone, R., Webster, C.G.: A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46(5), 2309–2345 (2008)

    Article  MathSciNet  Google Scholar 

  36. Nocedal, J., Yuan, Y.: Combining trust region and line search techniques. In: Advances in nonlinear programming, pages 153–175. Springer (1998)

  37. Paidoussis, M.P.: Fluid-structure interactions: slender structures and axial flow, vol. 1. Academic press (1998)

  38. Pearson, J.E.: Complex patterns in a simple system. Science 261(5118), 189–192 (1993)

    Article  Google Scholar 

  39. Pilanci, M., Wainwright, M.: Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence. SIAM J. Optim. 27(1), 205–245 (2017)

    Article  MathSciNet  Google Scholar 

  40. Piscopo, M.L., Spannowsky, M., Waite, P.: Solving differential equations with neural networks: Applications to the calculation of cosmological phase transitions. Phys. Rev. D 100, 016002 (2019)

    Article  MathSciNet  Google Scholar 

  41. Qin, T., Wu, K., Xiu, D.: Data driven governing equations approximation using deep neural networks. J. Comput. Phys. 395, 620–635 (2019)

    Article  MathSciNet  Google Scholar 

  42. Russell, R.D., Shampine, L.F.: A collocation method for boundary value problems. Numer. Math. 19(1), 1–28 (1972)

    Article  MathSciNet  Google Scholar 

  43. Schumack, M.R., Schultz, W.W., Boyd, J.P.: Spectral method solution of the stokes equations on nonstaggered grids. J. Comput. Phys. 94(1), 30–58 (1991)

    Article  MathSciNet  Google Scholar 

  44. Shen, J., Tang, T., Wang, L.: Spectral methods: algorithms, analysis and applications, vol. 41. Springer Science & Business Media (2011)

  45. Shen, J., Yu, H.: Efficient spectral sparse grid methods and applications to high-dimensional elliptic problems. SIAM J. Sci. Comput. 32(6), 3228–3250 (2010)

    Article  MathSciNet  Google Scholar 

  46. Shu, C.: High-order finite difference and finite volume weno schemes and discontinuous galerkin methods for cfd. International Journal of Computational Fluid Dynamics 17(2), 107–118 (2003)

    Article  MathSciNet  Google Scholar 

  47. Siegel, Jonathan W., Xu, Jinchao: High-order approximation rates for neural networks with \(\operatorname{ReLU}^k\) activation functions. arXiv preprint arXiv:2012.07205 (2020)

  48. Sirignano, J., Spiliopoulos, K.: Dgm: A deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375, 1339–1364 (2018)

    Article  MathSciNet  Google Scholar 

  49. Smith, Barry, Bjorstad, Petter, Gropp, William: Domain decomposition: parallel multilevel methods for elliptic partial differential equations. Cambridge University Press (2004)

  50. Sommese, A.J., Verschelde, J., Wampler, C.W.: Introduction to numerical algebraic geometry. In: Solving polynomial equations, pages 301–337. Springer (2005)

  51. Toselli, A., Widlund, O.: Domain decomposition methods-algorithms and theory, vol. 34. Springer Science & Business Media (2006)

  52. Turing, A.M.: The chemical basis of morphogenesis. Bull. Math. Biol. 52(1–2), 153–197 (1990)

    Article  Google Scholar 

  53. Wampler, C.W., Sommese, A.J.: Numerical algebraic geometry and algebraic kinematics. Acta Numer 20, 469–567 (2011)

    Article  MathSciNet  Google Scholar 

  54. Wang, Y., Hao, W., Lin, G.: Two-level spectral methods for nonlinear elliptic equations with multiple solutions. SIAM J. Sci. Comput. 40(4), B1180–B1205 (2018)

    Article  MathSciNet  Google Scholar 

  55. Wang, Y., Lin, G.: Efficient deep learning techniques for multiphase flow simulation in heterogeneous porous media. Journal of Computational Physics, page 108968 (2019)

  56. Weickert, J., Schnörr, C.: A theoretical framework for convex regularizers in pde-based computation of image motion. Int. J. Comput. Vision 45(3), 245–264 (2001)

    Article  Google Scholar 

  57. Wu, K., Xiu, D.: Data-driven deep learning of partial differential equations in modal space. arXiv preprint arXiv:1910.06948 (2019)

  58. Xing, Y., Shu, C.: High order finite difference weno schemes with the exact conservation property for the shallow water equations. J. Comput. Phys. 208(1), 206–227 (2005)

    Article  MathSciNet  Google Scholar 

  59. Xiu, D.: Numerical methods for stochastic computations: a spectral method approach. Princeton University Press (2010)

  60. Xu, J., Zikatanov, L.: A monotone finite element scheme for convection-diffusion equations. Mathematics of Computation of the American Mathematical Society 68(228), 1429–1446 (1999)

    Article  MathSciNet  Google Scholar 

  61. Xu, J., Zikatanov, L.: On an energy minimizing basis for algebraic multigrid methods. Comput. Vis. Sci. 7(3–4), 121–127 (2004)

    Article  MathSciNet  Google Scholar 

  62. Xu, Jinchao: The finite neuron method and convergence analysis. arXiv preprint arXiv:2010.01458 (2020)

Download references

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. Department of Mathematics, Pennsylvania State University, University Park, PA, 16802, USA

    Qipin Chen & Wenrui Hao

Authors
  1. Qipin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenrui Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenrui Hao.

Ethics declarations

Conflict of interest

This research is supported by NSF via DMS-1818769. There is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Q., Hao, W. Randomized Newton’s Method for Solving Differential Equations Based on the Neural Network Discretization. J Sci Comput 92, 49 (2022). https://doi.org/10.1007/s10915-022-01905-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-022-01905-9

Keywords

Search

Navigation