Abstract
We consider solving the forward and inverse partial differential equations (PDEs) which have sharp solutions with physics-informed neural networks (PINNs) in this work. In particular, to better capture the sharpness of the solution, we propose the adaptive sampling methods (ASMs) based on the residual and the gradient of the solution. We first present a residual only-based ASM denoted by ASM I. In this approach, we first train the neural network using a small number of residual points and divide the computational domain into a certain number of sub-domains, then we add new residual points in the sub-domain which has the largest mean absolute value of the residual, and those points which have the largest absolute values of the residual in this sub-domain as new residual points. We further develop a second type of ASM (denoted by ASM II) based on both the residual and the gradient of the solution due to the fact that only the residual may not be able to efficiently capture the sharpness of the solution. The procedure of ASM II is almost the same as that of ASM I, and we add new residual points which have not only large residuals but also large gradients. To demonstrate the effectiveness of the present methods, we use both ASM I and ASM II to solve a number of PDEs, including the Burger equation, the compressible Euler equation, the Poisson equation over an L-shape domain as well as the high-dimensional Poisson equation. It has been shown from the numerical results that the sharp solutions can be well approximated by using either ASM I or ASM II, and both methods deliver much more accurate solutions than the original PINNs with the same number of residual points. Moreover, the ASM II algorithm has better performance in terms of accuracy, efficiency, and stability compared with the ASM I algorithm. This means that the gradient of the solution improves the stability and efficiency of the adaptive sampling procedure as well as the accuracy of the solution. Furthermore, we also employ the similar adaptive sampling technique for the data points of boundary conditions (BCs) if the sharpness of the solution is near the boundary. The result of the L-shape Poisson problem indicates that the present method can significantly improve the efficiency, stability, and accuracy.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
LECUN, Y., BENGIO, Y., and HINTON, G. Deep learning. nature, 521(7553), 436–444 (2015)
MIKOLOV, T., DEORAS, A., POVEY, D., BURGET, L., and ČERNOCKỲ, J. Strategies for training large scale neural network language models. 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, Hawaii, 196–201 (2011)
HINTON, G., DENG, L., YU, D., DAHL, G., MOHAMED, A. R., JAITLY, N., SENIOR, A., VANHOUCKE, V., NGUYEN, P., SAINATH, T. N., and KINGSBURY, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6), 82–97 (2012)
SAINATH, T. N., MOHAMED, A. R., KINGSBURY, B., and RAMABHADRAN, B. Deep convolutional neural networks for LVCSR. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Canada, 8614–8618 (2013)
KRIZHEVSKY, A., SUTSKEVER, I., and HINTON, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90 (2017)
TOMPSON, J. J., JAIN, A., LECUN, Y., and BREGLER, C. Joint training of a convolutional network and a graphical model for human pose estimation. Advances in Neural Information Processing Systems, 27 1799–1807 (2014)
RAISSI, M., PERDIKARIS, P., and KARNIADAKIS, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707 (2019)
WEINAN, E. and YU, B. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1), 1–12 (2018)
HAN, J., JENTZEN, A., and WEINAN, E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34), 8505–8510 (2018)
LONG, Z., LU, Y., MA, X., and DONG, B. PDE-Net: learning PDEs from data. International Conference on Machine Learning, 80, 3208–3216 (2018)
LONG, Z., LU, Y., and DONG, B. PDE-Net 2.0: learning PDEs from data with a numericsymbolic hybrid deep network. Journal of Computational Physics, 399, 108925 (2019)
SIRIGNANO, J. and SPILIOPOULOS, K. DGM: a deep learning algorithm for solving partial differential equations. Journal of computational physics, 375, 1339–1364 (2018)
PANG, G., LU, L., and KARNIADAKIS, G. E. FPINNs: fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4), 2603–2626 (2019)
MENG, X. and KARNIADAKIS, G. E. A composite neural network that learns from multifidelity data: application to function approximation and inverse PDE problems. Journal of Computational Physics, 401, 109020 (2020)
ZHANG, D., LU, L., GUO, L., and KARNIADAKIS, G. E. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397, 108850 (2019)
LU, L., MENG, X., MAO, Z., and KARNIADAKIS, G. E. DeepXDE: a deep learning library for solving differential equations. SIAM Review, 63(1), 208–228 (2021)
RAISSI, M., YAZDANI, A., and KARNIADAKIS, G. E. Hidden fluid mechanics: a Navier-Stokes informed deep learning framework for assimilating flow visualization data. arXiv Preprint, arXiv:1808.04327 (2018) https://doi.org/10.48550/arXiv.1808.04327
RAISSI, M., WANG, Z., TRIANTAFYLLOU, M. S., and KARNIADAKIS, G. E. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861, 119–137 (2019)
CAI, S., MAO, Z., WANG, Z., YIN, M., and KARNIADAKIS, G. E. Physics-informed neural networks (PINNs) for fluid mechanics: a review. Acta Mechanica Sinica, 37(12), 1727–1738 (2021)
MAO, Z., JAGTAP, A. D., and KARNIADAKIS, G. E. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360, 112789 (2020)
JAGTAP, A. D., MAO, Z., ADAMS, N., and KARNIADAKIS, G. E. Physics-informed neural networks for inverse problems in supersonic flows. Journal of Computational Physics, 466, 111402 (2022)
BAYDIN, A. G., PEARLMUTTER, B. A., RADUL, A. A., and SISKIND, J. M. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18, 1–43 (2018)
BERGER, M. J. and OLIGER, J. Adaptive mesh refinement for hyperbolic partial differential equations. Journal of Computational Physics, 53(3), 484–512 (1984)
BERGER, M. J. and COLELLA, P. Local adaptive mesh refinement for shock hydrodynamics. Journal of Computational Physics, 82(1), 64–84 (1989)
VERFÜRTH, R. A posteriori error estimation and adaptive mesh-refinement techniques. Journal of Computational and Applied Mathematics, 50(1–3), 67–83 (1994)
BAEZA, A. and MULET, P. Adaptive mesh refinement techniques for high-order shock capturing schemes for multi-dimensional hydrodynamic simulations. International Journal for Numerical Methods in Fluids, 52(4), 455–471 (2006)
YU, J., LU, L., MENG, X., and KARNIADAKIS, G. E. Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393, 114823 (2022)
WU, C., ZHU, M., TAN, Q., KARTHA, Y., and LU, L. A comprehensive study of nonadaptive and residual-based adaptive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 403, 115671 (2023)
GAO, Z., YAN, L., and ZHOU, T. Failure-informed adaptive sampling for PINNs. arXiv Preprint, arXiv: 2210.00279 (2022) https://doi.org/10.48550/arXiv.2210.00279
GUO, J., WANG, H., and HOU, C. A novel adaptive causal sampling method for physicsinformed neural networks. arXiv Preprint, arXiv: 2210.12914 (2022) https://doi.org/10.48550/arXiv.2210.12914
HAN, J., CAI, Z., WU, Z., and ZHOU, X. Residual-quantile adjustment for adaptive training of physics-informed neural network. arXiv Preprint, arXiv: 2209.05315 (2022) https://doi.org/10.48550/arXiv.2209.05315
Acknowledgements
The authors also would like to acknowledge the Xiaomi Young Talents Program.
Author information
Authors and Affiliations
Corresponding author
Additional information
Conflict of interest
The authors declare no conflict of interest.
Citation: MAO, Z. P. and MENG, X. H. Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions. Applied Mathematics and Mechanics (English Edition), 44(7), 1069–1084 (2023) https://doi.org/10.1007/s10483-023-2994-7
Project supported by the National Key R&D Program of China (No. 2022YFA1004504), the National Natural Science Foundation of China (Nos. 12171404 and 12201229), and the Fundamental Research Funds for Central Universities of China (No. 20720210037)
Rights and permissions
Open access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Mao, Z., Meng, X. Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions. Appl. Math. Mech.-Engl. Ed. 44, 1069–1084 (2023). https://doi.org/10.1007/s10483-023-2994-7
Received:
Revised:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1007/s10483-023-2994-7
Key words
- physics-informed neural network (PINN)
- adaptive sampling
- high-dimension
- L-shape Poisson equation
- accuracy