Abstract
Many physical processes such as weather phenomena or fluid mechanics are governed by partial differential equations (PDEs). Modelling such dynamical systems using Neural Networks is an active research field. However, current methods are still very limited, as they do not exploit the knowledge about the dynamical nature of the system, require extensive prior knowledge about the governing equations or are limited to linear or first-order equations. In this work we make the observation that the Method of Lines used to solve PDEs can be represented using convolutions which makes convolutional neural networks (CNNs) the natural choice to parametrize arbitrary PDE dynamics. We combine this parametrization with differentiable ODE solvers to form the NeuralPDE Model, which explicitly takes into account the fact that the data is governed by differential equations. We show in several experiments on toy and real-world data that our model consistently outperforms state-of-the-art models used to learn dynamical systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Our code will be made publicly available upon publication.
- 2.
- 3.
References
Ayed, I., de Bézenac, E., Pajot, A., Brajard, J., Gallinari, P.: Learning dynamical systems from partial observations. CoRR abs/1902.11136 (2019). http://arxiv.org/abs/1902.11136
Belbute-Peres, F.D.A., Economon, T., Kolter, Z.: Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. In: International Conference on Machine Learning, pp. 2402–2411. PMLR, November 2020. https://proceedings.mlr.press/v119/de-avila-belbute-peres20a.html
Berg, J., Nyström, K.: Data-driven discovery of PDEs in complex datasets. J. Comput. Phys. 384, 239–252 (2019). https://doi.org/10.1016/j.jcp.2019.01.036, http://arxiv.org/abs/1808.10788, arXiv: 1808.10788
Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond Euclidean data. IEEE Sig. Process. Mag. 34(4), 18–42 (2017). https://doi.org/10.1109/MSP.2017.2693418, http://arxiv.org/abs/1611.08097, arXiv: 1611.08097 version: 1
Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018). https://papers.nips.cc/paper/2018/hash/69386f6bb1dfed68692a24c8686939b9-Abstract.html
Dormand, J., Prince, P.: A family of embedded runge-kutta formulae. J. Comput. Appl. Math. 6(1), 19–26 (1980). https://doi.org/10.1016/0771-050X(80)90013-3, https://www.sciencedirect.com/science/article/pii/0771050X80900133
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. arXiv:1603.05027 [cs], July 2016. http://arxiv.org/abs/1603.05027, arXiv: 1603.05027
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Iakovlev, V., Heinonen, M., Lähdesmäki, H.: Learning continuous-time pdes from sparse data with graph neural networks. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=aUX5Plaq7Oy
Iooss, G., Helleman, R.H.G.: Chaotic behaviour of deterministic systems. North-Holland, Netherlands (1983). http://inis.iaea.org/search/search.aspx?orig_q=RN:16062000
Karlbauer, M., Otte, S., Lensch, H.P.A., Scholten, T., Wulfmeyer, V., Butz, M.V.: A distributed neural network architecture for robust non-linear spatio-temporal prediction. arXiv:1912.11141 [cs], December 2019. http://arxiv.org/abs/1912.11141, arXiv: 1912.11141
Kuznetsov, Y.A.: Introduction to Dynamical Systems, pp. 1–35. Springer, New York (1995). https://doi.org/10.1007/978-1-4757-2421-9_1
LeCun, Y., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, vol. 2. Morgan-Kaufmann (1990). https://proceedings.neurips.cc/paper/1989/hash/53c3bce66e43be4f209556518c2fcb54-Abstract.html
Li, J., Sun, G., Zhao, G., Lehman, L.W.H.: Robust low-rank discovery of data-driven partial differential equations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, pp. 767–774, April 2020. https://doi.org/10.1609/aaai.v34i01.5420, https://ojs.aaai.org/index.php/AAAI/article/view/5420
Lipton, Z.C., Berkowitz, J., Elkan, C.: A critical review of recurrent neural networks for sequence learning. arXiv:1506.00019 [cs], October 2015. http://arxiv.org/abs/1506.00019, arXiv: 1506.00019
Long, Z., Lu, Y., Dong, B.: PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019). https://doi.org/10.1016/j.jcp.2019.108925, http://arxiv.org/abs/1812.04426, arXiv: 1812.04426
Praditia, T., Karlbauer, M., Otte, S., Oladyshkin, S., Butz, M.V., Nowak, W.: Finite volume neural network: modeling subsurface contaminant transport. arXiv:2104.06010 [cs], April 2021. http://arxiv.org/abs/2104.06010, arXiv: 2104.06010
Rackauckas, C., et al.: Universal differential equations for scientific machine learning. arXiv:2001.04385 [cs, math, q-bio, stat], August 2020. http://arxiv.org/abs/2001.04385, arXiv: 2001.04385
Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part I): data-driven solutions of nonlinear partial differential equations. arXiv:1711.10561 [cs, math, stat], November 2017. http://arxiv.org/abs/1711.10561, arXiv: 1711.10561
Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part II): data-driven discovery of nonlinear partial differential equations. arXiv:1711.10566 [cs, math, stat], November 2017. http://arxiv.org/abs/1711.10566, arXiv: 1711.10566 version: 1
Rasp, S., Dueben, P.D., Scher, S., Weyn, J.A., Mouatadid, S., Thuerey, N.: WeatherBench: a benchmark data set for data-driven weather forecasting. J. Adv. Model. Earth Syst. 12(11), e2020MS002203 (2020). https://doi.org/10.1029/2020MS002203, https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002203
Rasp, S., Thuerey, N.: Data-driven medium-range weather prediction with a resnet pretrained on climate simulations: a new model for weatherbench. J. Adv. Model. Earth Syst. 13(2), e2020MS002405 (2021). https://doi.org/10.1029/2020MS002405, https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002405
Rudolph, M.: Attaining exponential convergence for the flux error with second- and fourth-order accurate finite-difference equations. II. Application to systems comprising first-order chemical reactions. J. Comput. Chem. 26(6), 633–641 (2005). https://doi.org/10.1002/jcc.20201, https://onlinelibrary.wiley.com/doi/abs/10.1002/jcc.20201
Scher, S., Messori, G.: Weather and climate forecasting with neural networks: using general circulation models (GCMS) with different complexity as a study ground. Geosci. Model Dev. 12(7), 2797–2809 (2019). https://doi.org/10.5194/gmd-12-2797-2019, https://gmd.copernicus.org/articles/12/2797/2019/
Schiesser, W.E.: The numerical method of lines: integration of partial differential equations. Elsevier, July 2012. google-Books-ID: 2YDNCgAAQBAJ
Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., WOO, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc. (2015). https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
So, C.C., Li, T.O., Wu, C., Yung, S.P.: Differential spectral normalization (DSN) for PDE discovery. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, pp. 9675–9684, May 2021. https://ojs.aaai.org/index.php/AAAI/article/view/17164
Wang, D., Yang, Y., Ning, S.: DeepSTCL: a deep spatio-temporal ConvLSTM for travel demand prediction. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, Julu 2018. https://doi.org/10.1109/IJCNN.2018.8489530, iSSN: 2161–4407
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dulny, A., Hotho, A., Krause, A. (2022). NeuralPDE: Modelling Dynamical Systems from Data. In: Bergmann, R., Malburg, L., Rodermund, S.C., Timm, I.J. (eds) KI 2022: Advances in Artificial Intelligence. KI 2022. Lecture Notes in Computer Science(), vol 13404. Springer, Cham. https://doi.org/10.1007/978-3-031-15791-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-15791-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15790-5
Online ISBN: 978-3-031-15791-2
eBook Packages: Computer ScienceComputer Science (R0)