Skip to main content
SpringerLink
Log in

Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting

  • Original Paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

Physics-informed neural networks (PINNs) are an emerging method for solving partial differential equations (PDEs) and have been widely applied in the field of scientific computing. In this paper, we introduce a novel adaptive PINN model for solving PDEs. The model draws on the idea of traditional adaptive methods and incorporates the adaptive collocation point movement method into the PINNs model. It can use residual information from the PDE or characteristics of the solution function itself to guide the movement of collocation points, giving an appropriate distribution of collocation points for specific problems, improving the predictive accuracy of the model, and avoiding overfitting. Additionally, the model introduces an adaptive loss weighting strategy, which updates adaptive weights continuously by minimizing negative log-likelihood estimation to achieve adaptive weighting of the loss function, thereby improving the convergence rate and accuracy of the model. Finally, we conduct extensive experiments, including the one-dimensional Poisson equation, two-dimensional Poisson equation, Burgers equation, Klein–Gordon equation, Helmholtz equation, and Lid-Driven problem, to demonstrate the effectiveness and accuracy of the proposed model. The experimental results show that the model can significantly improve predictive accuracy and generalization ability. The data and code can be found at https://github.com/hsbhc/AMAW-PINN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Data availability

The data and code can be found at https://github.com/hsbhc/AMAW-PINN.

References

  1. Renardy, M., Rogers, R.C.: An Introduction to Partial Differential Equations, vol. 13. Springer, New York (2006)

    MATH  Google Scholar 

  2. Strauss, W.A.: Partial Differential Equations: An Introduction. John Wiley & Sons, Hoboken (2007)

    Google Scholar 

  3. Ricardo, H.J.: A Modern Introduction to Differential Equations. Academic Press, London (2020)

    Google Scholar 

  4. Murray, J.D.: Mathematical Biology: I. An Introduction. Springer, New York (2002)

    MATH  Google Scholar 

  5. Mészáros, P., Fox, D.B., Hanna, C., Murase, K.: Multi-messenger astrophysics. Nat. Rev. Phys. 1(10), 585–599 (2019)

    Google Scholar 

  6. Smolarkiewicz, P.K., Kühnlein, C., Wedi, N.P.: Semi-implicit integrations of perturbation equations for all-scale atmospheric dynamics. J. Comput. Phys. 376, 145–159 (2019)

    MathSciNet  MATH  Google Scholar 

  7. Balla, C.S., Alluguvelli, R., Naikoti, K., Makinde, O.D.: Effect of chemical reaction on bioconvective flow in oxytactic microorganisms suspended porous cavity. J. Appl. Comput. Mech. 6(3), 653–664 (2020)

    Google Scholar 

  8. Lye, K.O., Mishra, S., Ray, D.: Deep learning observables in computational fluid dynamics. J. Comput. Phys. 410, 109339 (2020)

    MathSciNet  MATH  Google Scholar 

  9. Markowich, P.: Applied Partial Differential Equations: A Visual Approach. Springer, New York (2007)

    MATH  Google Scholar 

  10. Zhang, Y.: A finite difference method for fractional partial differential equation. Appl. Math. Comput. 215(2), 524–529 (2009)

    MathSciNet  MATH  Google Scholar 

  11. Taylor, C.A., Hughes, T.J., Zarins, C.K.: Finite element modeling of blood flow in arteries. Comput. Methods Appl. Mech. Eng. 158(1–2), 155–196 (1998)

    MathSciNet  MATH  Google Scholar 

  12. Eymard, R., Gallouët, T., Herbin, R.: Finite volume methods. Handb. Num. Anal. 7, 713–1018 (2000)

    MathSciNet  MATH  Google Scholar 

  13. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Google Scholar 

  14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Google Scholar 

  15. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781 (2013)

  16. Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al.: Deep speech 2: end-to-end speech recognition in english and mandarin. In: International Conference on Machine Learning, pp. 173–182 (2016). PMLR

  17. Cai, X., Li, X., Razmjooy, N., Ghadimi, N., et al.: Breast cancer diagnosis by convolutional neural network and advanced thermal exchange optimization algorithm. In: Computational and Mathematical Methods in Medicine (2021)

  18. Guo, Z., Xu, L., Si, Y., Razmjooy, N.: Novel computer-aided lung cancer detection based on convolutional neural network-based and feature-based classifiers using metaheuristics. Int. J. Imaging Syst. Technol. 31(4), 1954–1969 (2021)

    Google Scholar 

  19. Huang, Q., Ding, H., Razmjooy, N.: Optimal deep learning neural network using ISSA for diagnosing the oral cancer. Biomed. Signal Process. Control 84, 104749 (2023)

    Google Scholar 

  20. Raissi, M., Yazdani, A., Karniadakis, G.E.: Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations. Science 367(6481), 1026–1030 (2020)

    MathSciNet  MATH  Google Scholar 

  21. Lu, L., Meng, X., Mao, Z., Karniadakis, G.E.: DeepXDE: a deep learning library for solving differential equations. SIAM Rev. 63(1), 208–228 (2021)

    MathSciNet  MATH  Google Scholar 

  22. Sirignano, J., Spiliopoulos, K.: DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375, 1339–1364 (2018)

    MathSciNet  MATH  Google Scholar 

  23. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019)

    MathSciNet  MATH  Google Scholar 

  24. Yuan, L., Ni, Y.-Q., Deng, X.-Y., Hao, S.: A-PINN: auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations. J. Comput. Phys. 462, 111260 (2022)

    MathSciNet  MATH  Google Scholar 

  25. Yu, J., Lu, L., Meng, X., Karniadakis, G.E.: Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Comput. Methods Appl. Mech. Eng. 393, 114823 (2022)

    MathSciNet  MATH  Google Scholar 

  26. Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E.: Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell. 3(3), 218–229 (2021)

    Google Scholar 

  27. Wang, S., Wang, H., Perdikaris, P.: Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Sci. Adv. 7(40), 8605 (2021)

    Google Scholar 

  28. Long, Z., Lu, Y., Dong, B.: PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019)

    MathSciNet  MATH  Google Scholar 

  29. Bai, Y., Chaolu, T., Bilige, S.: Solving huxley equation using an improved PINN method. Nonlinear Dyn. 105(4), 3439–3450 (2021)

    Google Scholar 

  30. Wang, S., Teng, Y., Perdikaris, P.: Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM J. Sci. Comput. 43(5), 3055–3081 (2021)

    MathSciNet  MATH  Google Scholar 

  31. Meng, X., Li, Z., Zhang, D., Karniadakis, G.E.: PPINN: parareal physics-informed neural network for time-dependent PDEs. Comput. Methods Appl. Mech. Eng. 370, 113250 (2020)

    MathSciNet  MATH  Google Scholar 

  32. Pang, G., D’Elia, M., Parks, M., Karniadakis, G.E.: nPINNs: nonlocal Physics-Informed Neural Networks for a parametrized nonlocal universal Laplacian operator. Algorithms and Applications. J. Comput. Phys. 422, 109760 (2020)

    MathSciNet  MATH  Google Scholar 

  33. Jagtap, A.D., Kawaguchi, K., Karniadakis, G.E.: Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. J. Comput. Phys. 404, 109136 (2020)

    MathSciNet  MATH  Google Scholar 

  34. Li, Y., Xu, L., Ying, S.: Dwnn: deep wavelet neural network for solving partial differential equations. Mathematics 10(12), 1976 (2022)

    Google Scholar 

  35. Chen, M., Niu, R., Zheng, W.: Adaptive multi-scale neural network with resnet blocks for solving partial differential equations. Nonlinear Dyn. 111(7), 6499–6518 (2023)

    Google Scholar 

  36. Yue, J., Li, J.: The physics informed neural networks for the unsteady Stokes problems. Int. J. Num. Methods Fluids 94(9), 1416–1433 (2022)

    MathSciNet  Google Scholar 

  37. Pang, G., Lu, L., Karniadakis, G.E.: fPINNs: fractional physics-informed neural networks. SIAM J. Sci. Comput. 41(4), 2603–2626 (2019)

    MathSciNet  MATH  Google Scholar 

  38. Han, J., Jentzen, A., et al.: Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat. 5(4), 349–380 (2017)

    MathSciNet  MATH  Google Scholar 

  39. Lyu, L., Zhang, Z., Chen, M., Chen, J.: MIM: a deep mixed residual method for solving high-order partial differential equations. J. Comput. Phys. 452, 110930 (2022)

    MathSciNet  MATH  Google Scholar 

  40. Jin, X., Cai, S., Li, H., Karniadakis, G.E.: NSFnets (Navier-Stokes flow nets): physics-informed neural networks for the incompressible Navier–Stokes equations. J. Comput. Phys. 426, 109951 (2021)

    MathSciNet  MATH  Google Scholar 

  41. Jiang, X., Wang, D., Fan, Q., Zhang, M., Lu, C., Lau, A.P.T.: Physics-informed neural network for nonlinear dynamics in fiber optics. Laser Photon. Rev. 16(9), 2100483 (2022)

    Google Scholar 

  42. Cai, S., Wang, Z., Wang, S., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks for heat transfer problems. J. Heat Transf. 143, 6 (2021)

    Google Scholar 

  43. Raymond, S.J., Cecchi, N.J., Alizadeh, H.V., Callan, A.A., Rice, E., Liu, Y., Zhou, Z., Zeineh, M., Camarillo, D.B.: Physics-informed machine learning improves detection of head impacts. Ann. Biomed. Eng. 1, 12 (2022)

    Google Scholar 

  44. Zheng, Q., Zeng, L., Karniadakis, G.E.: Physics-informed semantic inpainting: application to geostatistical modeling. J. Comput. Phys. 419, 109676 (2020)

    MathSciNet  MATH  Google Scholar 

  45. Bai, Y., Chaolu, T., Bilige, S.: The application of improved physics-informed neural network (IPINN) method in finance. Nonlinear Dyn. 107(4), 3655–3667 (2022)

    Google Scholar 

  46. Wen, X.-K., Wu, G.-Z., Liu, W., Dai, C.-Q.: Dynamics of diverse data-driven solitons for the three-component coupled nonlinear schrödinger model by the MPS-PINN method. Nonlinear Dyn. 109(4), 3041–3050 (2022)

    Google Scholar 

  47. Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L.: Physics-informed machine learning. Nat. Rev. Phys. 3(6), 422–440 (2021)

    Google Scholar 

  48. Chiu, P.-H., Wong, J.C., Ooi, C., Dao, M.H., Ong, Y.-S.: CAN-PINN: a fast physics-informed neural network based on coupled-automatic-numerical differentiation method. Comput. Methods Appl. Mech. Eng. 395, 114909 (2022)

    MathSciNet  MATH  Google Scholar 

  49. Wu, C., Zhu, M., Tan, Q., Kartha, Y., Lu, L.: A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 403, 115671 (2023)

    MathSciNet  MATH  Google Scholar 

  50. Nabian, M.A., Gladstone, R.J., Meidani, H.: Efficient training of physics-informed neural networks via importance sampling. Computer-Aided Civil Infrastr. Eng. 36(8), 962–977 (2021)

    Google Scholar 

  51. Daw, A., Bu, J., Wang, S., Perdikaris, P., Karpatne, A.: Rethinking the Importance of Sampling in Physics-Informed Neural Networks. arXiv preprint arXiv:2207.02338 (2022)

  52. Gao, W., Wang, C.: Active learning based sampling for high-dimensional nonlinear partial differential equations. J. Comput. Phys. 475, 111848 (2023)

    MathSciNet  MATH  Google Scholar 

  53. Tang, K., Wan, X., Yang, C.: DAS-PINNs: a deep adaptive sampling method for solving high-dimensional partial differential equations. J. Comput. Phys. 476, 111868 (2023)

    MathSciNet  MATH  Google Scholar 

  54. Zeng, S., Zhang, Z., Zou, Q.: Adaptive deep neural networks methods for high-dimensional partial differential equations. J. Comput. Phys. 463, 111232 (2022)

    MathSciNet  MATH  Google Scholar 

  55. Hanna, J.M., Aguado, J.V., Comas-Cardona, S., Askri, R., Borzacchiello, D.: Residual-based adaptivity for two-phase flow simulation in porous media using Physics-informed Neural Networks. Comput. Methods Appl. Mech. Eng. 396, 115100 (2022)

    MathSciNet  MATH  Google Scholar 

  56. Zhang, W., Almgren, A., Beckner, V., Bell, J., Blaschke, J., Chan, C., Day, M., Friesen, B., Gott, K., Graves, D., et al.: Amrex: a framework for block-structured adaptive mesh refinement. J. Open Sourc. Softw. 4(37), 1370–1370 (2019)

    Google Scholar 

  57. Berger, M.J., Oliger, J.: Adaptive mesh refinement for hyperbolic partial differential equations. J. Comput. Phys. 53(3), 484–512 (1984)

    MathSciNet  MATH  Google Scholar 

  58. Díez, P., Huerta, A.: A unified approach to remeshing strategies for finite element h-adaptivity. Comput. Methods Appl. Mech. Eng. 176(1–4), 215–229 (1999)

    MATH  Google Scholar 

  59. McClenny, L., Braga-Neto, U.: Self-adaptive Physics-Informed Neural Networks Using a Soft Attention Mechanism. arXiv preprint arXiv:2009.04544 (2020)

  60. Xiang, Z., Peng, W., Liu, X., Yao, W.: Self-adaptive loss balanced physics-informed neural networks. Neurocomputing 496, 11–34 (2022)

    Google Scholar 

  61. Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491 (2018)

  62. Subramanian, S., Kirby, R.M., Mahoney, M.W., Gholami, A.: Adaptive Self-Supervision Algorithms for Physics-Informed Neural Networks. arXiv preprint arXiv:2207.04084 (2022)

  63. Mao, Z., Jagtap, A.D., Karniadakis, G.E.: Physics-informed neural networks for high-speed flows. Comput. Methods Appl. Mech. Eng. 360, 112789 (2020)

    MathSciNet  MATH  Google Scholar 

  64. Bateman, H.: Some recent researches on the motion of fluids. Monthly Weather Rev. 43(4), 163–170 (1915)

    Google Scholar 

  65. Basdevant, C., Deville, M., Haldenwang, P., Lacroix, J., Ouazzani, J., Peyret, R., Orlandi, P., Patera, A.: Spectral and finite difference solutions of the Burgers equation. Comput. Fluids 14(1), 23–41 (1986)

    MATH  Google Scholar 

  66. Wazwaz, A.-M.: New travelling wave solutions to the Boussinesq and the Klein–Gordon equations. Commun. Nonlinear Sci. Numer. Simulat. 13(5), 889–901 (2008)

    MathSciNet  MATH  Google Scholar 

  67. Caudrey, P., Eilbeck, J., Gibbon, J.: The sine-Gordon equation as a model classical field theory. Il Nuovo Cimento B 25(2), 497–512 (1975)

    Google Scholar 

  68. Nédélec, J.-C.: Acoustic and Electromagnetic Equations: Integral Representations for Harmonic Problems, vol. 144. Springer, New York (2001)

    MATH  Google Scholar 

  69. Bruneau, C.-H., Saad, M.: The 2D lid-driven cavity problem revisited. Comput. Fluids 35(3), 326–348 (2006)

    MATH  Google Scholar 

Download references

Funding

This work is supported by the National Key Research and Development Program of China (No. 2021YFA1003004).

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Baoshan, Shanghai, 200444, People’s Republic of China

    Jie Hou & Ying Li

  2. Department of Mathematics, School of Science, Shanghai University, Baoshan, Shanghai, 200444, People’s Republic of China

    Shihui Ying

Authors
  1. Jie Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shihui Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. The experiments, data collection, and analysis were performed by JH. The first draft of the manuscript was written by JH, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ying Li.

Ethics declarations

Comnflict of interests

The authors declare that there is no conflict of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hou, J., Li, Y. & Ying, S. Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting. Nonlinear Dyn 111, 15233–15261 (2023). https://doi.org/10.1007/s11071-023-08654-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-023-08654-w

Keywords

Search

Navigation