Abstract
Recently, physics-informed neural networks (PINNs) have offered a powerful new paradigm for solving problems relating to differential equations. Compared to classical numerical methods, PINNs have several advantages, for example their ability to provide mesh-free solutions of differential equations and their ability to carry out forward and inverse modelling within the same optimisation problem. Whilst promising, a key limitation to date is that PINNs have struggled to accurately and efficiently solve problems with large domains and/or multi-scale solutions, which is crucial for their real-world application. Multiple significant and related factors contribute to this issue, including the increasing complexity of the underlying PINN optimisation problem as the problem size grows and the spectral bias of neural networks. In this work, we propose a new, scalable approach for solving large problems relating to differential equations called finite basis physics-informed neural networks (FBPINNs). FBPINNs are inspired by classical finite element methods, where the solution of the differential equation is expressed as the sum of a finite set of basis functions with compact support. In FBPINNs, neural networks are used to learn these basis functions, which are defined over small, overlapping subdomains. FBINNs are designed to address the spectral bias of neural networks by using separate input normalisation over each subdomain and reduce the complexity of the underlying optimisation problem by using many smaller neural networks in a parallel divide-and-conquer approach. Our numerical experiments show that FBPINNs are effective in solving both small and larger, multi-scale problems, outperforming standard PINNs in both accuracy and computational resources required, potentially paving the way to the application of PINNs on large, real-world problems.
Data Availibility
All our training/test data were generated synthetically, and all the code required to reproduce this data and our results is available here: https://github.com/benmoseley/FBPINNs.
References
Giorgi, F.: Thirty years of regional climate modeling: where are we and where are we going next? Journal of Geophysical Research: Atmospheres 124(11), 5696–5723 (2019). https://doi.org/10.1029/2018JD030094
Prein, A.F., Langhans, W., Fosser, G., Ferrone, A., Ban, N., Goergen, K., Keller, M., Tölle, M., Gutjahr, O., Feser, F., Brisson, E., Kollet, S., Schmidli, J., Van Lipzig, N.P.M., Leung, R.: A review on regional convection-permitting climate modeling: demonstrations, prospects, and challenges. Rev. Geophys. 53(2), 323–361 (2015). https://doi.org/10.1002/2014RG000475
Agostinelli, S., Allison, J., Amako, K., Apostolakis, J., Araujo, H., Arce, P., Asai, M., Axen, D., Banerjee, S., Barrand, G., Behner, F., Bellagamba, L., Boudreau, J., Broglia, L., Brunengo, A., Burkhardt, H., Chauvie, S., Chuma, J., Chytracek, R., Cooperman, G., Cosmo, G., Degtyarenko, P., Dell’Acqua, A., Depaola, G., Dietrich, D., Enami, R., Feliciello, A., Ferguson, C., Fesefeldt, H., Folger, G., Foppiano, F., Forti, A., Garelli, S., Giani, S., Giannitrapani, R., Gibin, D., Gomez Cadenas, J.J., Gonzalez, I., Gracia Abril, G., Greeniaus, G., Greiner, W., Grichine, V., Grossheim, A., Guatelli, S., Gumplinger, P., Hamatsu, R., Hashimoto, K., Hasui, H., Heikkinen, A., Howard, A., Ivanchenko, V., Johnson, A., Jones, F.W., Kallenbach, J., Kanaya, N., Kawabata, M., Kawabata, Y., Kawaguti, M., Kelner, S., Kent, P., Kimura, A., Kodama, T., Kokoulin, R., Kossov, M., Kurashige, H., Lamanna, E., Lampen, T., Lara, V., Lefebure, V., Lei, F., Liendl, M., Lockman, W., Longo, F., Magni, S., Maire, M., Medernach, E., Minamimoto, K., Mora de Freitas, P., Morita, Y., Murakami, K., Nagamatu, M., Nartallo, R., Nieminen, P., Nishimura, T., Ohtsubo, K., Okamura, M., O’Neale, S., Oohata, Y., Paech, K., Perl, J., Pfeiffer, A., Pia, M.G., Ranjard, F., Rybin, A., Sadilov, S., di Salvo, E., Santin, G., Sasaki, T., Savvas, N., Sawada, Y., Scherer, S., Sei, S., Sirotenko, V., Smith, D., Starkov, N., Stoecker, H., Sulkimo, J., Takahata, M., Tanaka, S., Tcherniaev, E., Safai Tehrani, E., Tropeano, M., Truscott, P., Uno, H., Urban, L., Urban, P., Verderi, M., Walkden, A., Wander, W., Weber, H., Wellisch, J.P., Wenaus, T., Williams, D.C., Wright, D., Yamada, T., Yoshida, H., Zschiesche, D.: GEANT4 - a simulation toolkit. Nuclear Instruments and Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 506(3), 250–303 (2003). https://doi.org/10.1016/S0168-9002(03)01368-8
Jasak, H.: OpenFOAM: open source CFD in research and industry. International Journal of Naval Architecture and Ocean Engineering 1(2), 89–94 (2009). https://doi.org/10.2478/ijnaoe-2013-0011
Leng, K., Nissen-Meyer, T., van Driel, M., Hosseini, K., Al-Attar, D.: AxiSEM3D: broad-band seismic wavefields in 3-D global earth models with undulating discontinuities. Geophys. J. Int. 217(3), 2125–2146 (2019). https://doi.org/10.1093/gji/ggz092
Nochetto, R.H., Siebert, K.G., Veeser, A.: Theory of adaptive finite element methods: an introduction. In: DeVore, R., Kunoth, A. (eds.) Multiscale, Nonlinear and Adaptive Approximation, pp. 409–542. Springer, (2009)
Rasp, S., Pritchard, M.S., Gentine, P.: Deep learning to represent subgrid processes in climate models. Proc. Natl. Acad. Sci. U. S. A. 115(39), 9684–9689 (2018). arXiv:1806.04731. https://doi.org/10.1073/pnas.1810286115
Moseley, B., Nissen-Meyer, T., Markham, A.: Deep learning for fast simulation of seismic waves in complex media. Solid Earth 11(4), 1527–1549 (2020). https://doi.org/10.5194/se-11-1527-2020
Kasim, M.F., Watson-Parris, D., Deaconu, L., Oliver, S., Hatfield, P., Froula, D.H., Gregori, G., Jarvis, M., Khatiwala, S., Korenaga, J., Topp-Mugglestone, J., Viezzer, E., Vinko, S.M.: Building high accuracy emulators for scientific simulations with deep neural architecture search. Machine Learning: Science and Technology 3(1), 015013 (2022). arXiv:2001.08055. https://doi.org/10.1088/2632-2153/ac3ffa
Baker, N., Alexander, F., Bremer, T., Hagberg, A., Kevrekidis, Y., Najm, H., Parashar, M., Patra, A., Sethian, J., Wild, S., Willcox, K., Lee, S.: Workshop report on basic research needs for scientific machine learning: core technologies for artificial intelligence. Technical report, USDOE Office of Science (SC) (United States) (2019). https://doi.org/10.2172/1478744. http://www.osti.gov/servlets/purl/1478744/
Willard, J., Jia, X., Xu, S., Steinbach, M., Kumar, V.: Integrating scientific knowledge with machine learning for engineering and environmental systems. arXiv 1, 1 (2020). arXiv:2003.04919
Lagaris, I.E., Likas, A., Fotiadis, D.I.: Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks 9(5), 987–1000 (1998).arXiv:9705023[physics]. https://doi.org/10.1109/72.712178
Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019). https://doi.org/10.1016/j.jcp.2018.10.045
Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L.: Physics-informed machine learning. Nature Reviews Physics, 1–19 (2021). https://doi.org/10.1038/s42254-021-00314-5
Cuomo, S., di Cola, V.S., Giampaolo, F., Rozza, G., Raissi, M., Piccialli, F.: Scientific machine learning through physics-informed neural networks: where we are and what’s next. arXiv (2022). arXiv:2201.05624
Raissi, M., Yazdani, A., Karniadakis, G.E.: Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations. Science 367(6481), 1026–1030 (2020). https://doi.org/10.1126/science.aaw4741
Sun, L., Gao, H., Pan, S., Wang, J.X.: Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Comput. Methods Appl. Mech. Eng. 361 (2020). arXiv:1906.02382. https://doi.org/10.1016/j.cma.2019.112732
Sahli Costabal, F., Yang, Y., Perdikaris, P., Hurtado, D.E., Kuhl, E.: Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics 8, 42 (2020). https://doi.org/10.3389/fphy.2020.00042
Moseley, B., Markham, A., Nissen-Meyer, T.: Solving the wave equation with physics-informed deep learning. arXiv (2020). arXiv:2006.11894
Cai, S., Wang, Z., Fuest, F., Jeon, Y.J., Gray, C., Karniadakis, G.E.: Flow over an espresso cup: inferring 3-D velocity and pressure fields from tomographic background oriented Schlieren via physics-informed neural networks. J. Fluid Mech. 915, 102 (2021). arXiv:2103.02807. https://doi.org/10.1017/jfm.2021.135
Yang, L., Meng, X., Karniadakis, G.E.: B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. J. Comput. Phys. 425, 109913 (2021). https://doi.org/10.1016/j.jcp.2020.109913
Kharazmi, E., Zhang, Z., Karniadakis, G.E.M.: VPINNs: variational physics-informed neural networks for solving partial differential equations. arXiv (2019). arXiv:1912.00873
Zhu, Y., Zabaras, N., Koutsourelakis, P.S., Perdikaris, P.: Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. J. Comput. Phys. 394, 56–81 (2019). arXiv:1901.06314. https://doi.org/10.1016/j.jcp.2019.05.024
Geneva, N., Zabaras, N.: Modeling the dynamics of PDE systems with physics-constrained deep auto-regressive networks. J. Comput. Phys. 403, 109056 (2020). arXiv:1906.05747. https://doi.org/10.1016/j.jcp.2019.109056
Gao, H., Sun, L., Wang, J.X.: PhyGeoNet: physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain. J. Comput. Phys. 428, 110079 (2021). arXiv:2004.13145. https://doi.org/10.1016/j.jcp.2020.110079
Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E.: Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence 3(3), 218–229 (2021). https://doi.org/10.1038/s42256-021-00302-5
Wang, S., Wang, H., Perdikaris, P.: Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science Advances 7(40), 8605–8634 (2021). arXiv:2103.10974. https://doi.org/10.1126/sciadv.abi8605
Pang, G., Lu, L.U., Karniadakis, G.E.M.: FPinns: fractional physics-informed neural networks. SIAM J. Sci. Comput. 41(4), 2603–2626 (2019). arXiv:1811.08967. https://doi.org/10.1137/18M1229845
Yang, L., Zhang, D., Karniadakis, G.E.: Physics-informed generative adversarial networks for stochastic differential equations. SIAM J . Sci. Comput. 42(1), 292–317 (2020). arXiv:1811.02033. https://doi.org/10.1137/18M1225409
Zhang, D., Lu, L., Guo, L., Karniadakis, G.E.: Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. J. Comput. Phys. 397, 108850 (2019). arXiv:1809.08327. https://doi.org/10.1016/j.jcp.2019.07.048
Chen, Z., Liu, Y., Sun, H.: Physics-informed learning of governing equations from scarce data. Nat. Commun. 12(1), 1–13 (2021). arXiv:2005.03448. https://doi.org/10.1038/s41467-021-26434-1
Lu, L., Meng, X., Mao, Z., Karniadakis, G.E.: DeepXDE: a deep learning library for solving differential equations. SIAM Review 63( 1), 208–228 (2021). arXiv:1907.04502. https://doi.org/10.1137/19M1274067
Hennigh, O., Narasimhan, S., Nabian, M.A., Subramaniam, A., Tangsali, K., Rietmann, M., Ferrandis, J.d.A., Byeon, W., Fang, Z., Choudhry, S.: NVIDIA SimNet\(^{TM}\): an AI-accelerated multi-physics simulation framework. arXiv (2020) arXiv:2012.07938
Koryagin, A., Khudorozkov, R., Tsimfer, S.: PyDEns: a Python-framework for solving differential equations with neural networks. arXiv (2019). arXiv:1909.11544
Chen, F., Sondak, D., Protopapas, P., Mattheakis, M., Liu, S., Agarwal, D., Di Giovanni, M.: NeuroDiffEq: a Python package for solving differential equations with neural networks. Journal of Open Source Software 5(46), 1931 (2020). https://doi.org/10.21105/joss.01931
Wang, S., Teng, Y., Perdikaris, P.: Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM J. Sci. Comput. 43(5), 3055–3081 (2021). arXiv:2001.04536. https://doi.org/10.1137/20M1318043
Wang, S., Yu, X., Perdikaris, P.: When and why PINNs fail to train: a neural tangent kernel perspective. J. Comput. Phys. 449, 110768 (2022). arXiv:2007.14527. https://doi.org/10.1016/j.jcp.2021.110768
Shin, Y., Darbon, J., Karniadakis, G.E.: On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. Communications in Computational Physics 28(5), 2042–2074 (2020). arXiv:2004.01806. https://doi.org/10.4208/cicp.oa-2020-0193
Mishra, S., Molinaro, R.: Estimates on the generalization error of physics-informed neural networks for approximating PDEs. IMA J. Numer. Anal. 00, 1–43 (2022). arXiv:2006.16144. https://doi.org/10.1093/imanum/drab093
Xu, Z.Q.J., Zhang, Y., Luo, T., Xiao, Y., Ma, Z.: Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics 28( 5), 1746–1767 (2020). arXiv:1901.06523. https://doi.org/10.4208/CICP.OA-2020-0085
Rahaman, N., Baratin, A., Arpit, D., Draxlcr, F., Lin, M., Hamprecht, F.A., Bengio, Y., Courville, A.: On the spectral bias of neural networks. In: 36th International Conference on Machine Learning, ICML 2019, vol. 2019-June, pp. 9230–9239. International Machine Learning Society (IMLS), (2019). arXiv:1806.08734
Basri, R., Jacobs, D., Kasten, Y., Kritchman, S.: The convergence rate of neural networks for learned functions of different frequencies. In: Advances in Neural Information Processing Systems, vol. 32. Neural information processing systems foundation, (2019). arXiv:1906.00425
Cao, Y., Fang, Z., Wu, Y., Zhou, D.-X., Gu, Q.: Towards understanding the spectral bias of deep learning. IJCAI (2021). arXiv:1912.01198
Wang, S., Wang, H., Perdikaris, P.: On the eigenvector bias of Fourier feature networks: from regression to solving multi-scale PDEs with physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 384, 113938 (2021). arXiv:2012.10047. https://doi.org/10.1016/j.cma.2021.113938
Tancik, M., Srinivasan, P.P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J.T., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. In: Advances in Neural Information Processing Systems, vol. 2020-Decem. Neural information processing systems foundation, (2020). https://doi.org/10.48550/arxiv.2006.10739. arXiv:2006.10739v1
Liu, Z., Cai, W., Xu, Z.Q.J.: Multi-scale deep neural network (MscaleDNN) for solving Poisson-Boltzmann equation in complex domains. Communications in Computational Physics 28(5), 1970–2001 (2020). arXiv:2007.11207. https://doi.org/10.4208/CICP.OA-2020-0179
Heinlein, A., Klawonn, A., Lanser, M., Weber, J.: Combining machine learning and domain decomposition methods for the solution of partial differential equations–a review. GAMM-Mitteilungen 44(1), 202100001 (2021). https://doi.org/10.1002/gamm.202100001
Jagtap, A.D., Karniadakis, G.E.: Extended physics-informed neural networks (XPINNs): a generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics 28(5), 2002–2041 (2020). https://doi.org/10.4208/CICP.OA-2020-0164
Shukla, K., Jagtap, A.D., Karniadakis, G.E.: Parallel physics-informed neural networks via domain decomposition. J. Comput. Phys. 447, 110683 (2021). arXiv:2104.10013. https://doi.org/10.1016/j.jcp.2021.110683
Dwivedi, V., Parashar, N., Srinivasan, B.: Distributed learning machines for solving forward and inverse problems in partial differential equations. Neurocomputing 420, 299–316 (2021). https://doi.org/10.1016/j.neucom.2020.09.006
Dong, S., Li, Z.: Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Comput. Methods Appl. Mech. Eng. 387, 114129 (2021) arXiv:2012.02895. https://doi.org/10.1016/j.cma.2021.114129
Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70(1–3), 489–501 (2006). https://doi.org/10.1016/j.neucom.2005.12.126
Li, K., Tang, K., Wu, T., Liao, Q.: D3M: a deep domain decomposition method for partial differential equations. IEEE Access 8, 5283–5294 (2020). arXiv:1909.12236. https://doi.org/10.1109/ACCESS.2019.2957200
Stiller, P., Bethke, F., Böhme, M., Pausch, R., Torge, S., Debus, A., Vorberger, J., Bussmann, M., Hoffmann, N.: Large-scale neural solvers for partial differential equations. In: Nichols, J., Verastegui, B., Maccabe, A.B., Hernandez, O., Parete-Koon, S., Ahearn, T. (eds.) Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI, pp. 20–34. Springer, (2020)
Kharazmi, E., Zhang, Z., Karniadakis, G.E.: hp-VPINNs: variational physics-informed neural networks with domain decomposition. Comput. Methods Appl. Mech. Eng. 374 (2021). arXiv:2003.05385. https://doi.org/10.1016/j.cma.2020.113547
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d\(\backslash \)textquotesingle Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
Lagaris, I.E., Likas, A.C., Papageorgiou, D.G.: Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks 11(5), 1041–1049 (2000). https://doi.org/10.1109/72.870037
Berg, J., Nyström, K.: A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing 317, 28–41 (2018). arXiv:1711.06464. https://doi.org/10.1016/j.neucom.2018.06.056
Leake, C., Mortari, D.: Deep theory of functional connections: a new method for estimating the solutions of partial differential equations. Machine Learning and Knowledge Extraction 2(1), 37–55 (2020). https://doi.org/10.3390/make2010004
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR 2015 (2015). arXiv:1412.6980
Basdevant, C., Deville, M., Haldenwang, P., Lacroix, J.M., Ouazzani, J., Peyret, R., Orlandi, P., Patera, A.T.: Spectral and finite difference solutions of the Burgers equation. Computers and Fluids 14(1), 23–41 (1986). https://doi.org/10.1016/0045-7930(86)90036-8
Igel, H.: Computational seismology: a practical introduction. Oxford University Press, (2017)
Komatitsch, D., Martin, R.: An unsplit convolutional perfectly matched layer improved at grazing incidence for the seismic wave equation. Geophysics 72(5), 155–167 (2007)
Jacotécole, A., Jacotécole, J., Fédérale De Lausanne, P., Gabriel, F.: Neural tangent kernel: convergence and generalization in neural networks. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Funding
This work was funded by the UKRI EPSRC Center for Doctoral Training in Autonomous Intelligent Machines and Systems (AIMS CDT).
Author information
Authors and Affiliations
Contributions
Contributions (CRediT taxonomy) BM: conceptualisation, formal analysis, investigation, methodology, software, validation, visualisation, writing - original draft. AM, TNM: supervision, writing - review and editing.
Corresponding author
Ethics declarations
Competing of interests
The authors declare no competing interests.
Additional information
Communicated by: Siddhartha Mishra
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Moseley, B., Markham, A. & Nissen-Meyer, T. Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations. Adv Comput Math 49, 62 (2023). https://doi.org/10.1007/s10444-023-10065-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10444-023-10065-9
Keywords
- Physics-informed neural networks
- Domain decomposition
- Multi-scale modelling
- Forward modelling
- Differential equations
- Parallel computing
Mathematics Subject Classification (2010)
- 65M99
- 68T01