Abstract
This review presents a modern perspective on dynamical systems in the context of current goals and open challenges. In particular, our review focuses on the key challenges of discovering dynamics from data and finding data-driven representations that make nonlinear systems amenable to linear analysis. We explore various challenges in modern dynamical systems, along with emerging techniques in data science and machine learning to tackle them. The two chief challenges are (1) nonlinear dynamics and (2) unknown or partially known dynamics. Machine learning is providing new and powerful techniques for both challenges. Dimensionality reduction methods are used for projecting dynamical methods in reduced form, and these methods perform computational efficiency on real-world data. Data-driven models drive to discover the governing equations and give laws of physics. The identification of dynamical systems through deep learning techniques succeeds in inferring physical systems. Machine learning provides advanced new and powerful algorithms for nonlinear dynamics. Advanced deep learning methods like autoencoders, recurrent neural networks, convolutional neural networks, and reinforcement learning are used in modeling of dynamical systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Atencia M, Joya G, Sandoval F (2005) Hopfield neural networks for parametric identification of dynamical systems. Neural Process Lett 21:143–152. https://doi.org/10.1007/s11063-004-3424-3
Baek SH, Garcia-Diaz A, Dai Y (2020) Multi-choice wavelet thresholding based binary classification method. Methodology 16(2):127–146. https://doi.org/10.5964/meth.2787
Bai Z, Kaiser E et al (2020) Dynamic mode decomposition for compressive. Syst Identif 58(2):561–574. https://doi.org/10.2514/1.J057870
Benjamin Erichson N, Manohar K et al (2020) Randomized CP tensor decomposition. Mach Learn Sci Technol 1(2). https://doi.org/10.1088/2632-2153/ab8240
Berg J, Nyström K (2019) Data-driven discovery of PDEs in complex datasets. J Comput Phys 384:239–252. https://doi.org/10.1016/j.jcp.2019.01.036
Bongard J, Lipson H (2007) Automated reverse engineering of nonlinear dynamical systems. Proc Natl Acad Sci 104(24):9943–9948. https://doi.org/10.1073/pnas.0609476104
Boots B, Gordon GJ (2011) An online spectral learning algorithm for partially observable nonlinear dynamical systems. AAAI'11: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, pp 293–300. https://doi.org/10.5555/2900423.2900469
Brunton SL, Kutz JN (2019) Methods for data-driven multi-scale model discovery for materials. J Phys Mater 2:044002. https://doi.org/10.1088/2515-7639/ab291e
Brunton SL, Brunton BW, Proctor JL, Kutz JN (2016a) Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PLoS One 11(2):e0150171. https://doi.org/10.1371/journal.pone.0150171
Brunton SL, Proctorb JL, Kutz JN (2016b) Discovering governing equations from data by sparse identification of nonlinear dynamical systems. PNAS 113(15):3932–3937. https://doi.org/10.1073/pnas.1517384113
Chang H, Zhang D (2019) Machine learning subsurface flow equations from data. Comput Geosci 23:895–910. https://doi.org/10.1007/s10596-019-09847-2
Chen RTQ, Rubanova Y et al (2018) Neural ordinary differential equations, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada
Cireşan D, Meier U et al (2012) Multi-column deep neural network for traffic sign classification. Neural Netw 32:333–338. https://doi.org/10.1016/j.neunet.2012.02.023
Davoudia R, Millera GR, Nathan Kutz J (2018) Data-driven vision-based inspection for reinforced concrete beams and slabs: quantitative damage and load estimation. Autom Constr 96:292–309. https://doi.org/10.1016/j.autcon.2018.09.024
Dsilva CJ et al (2016) Data-driven reduction for a class of multiscale fast-slow stochastic dynamical systems. SIAM J Appl Dyn Syst 15(3):1327–1351
Erichson NB, Brunton SL, Kutz JN (2019a) Compressed dynamic mode decomposition for background modeling. J Real-Time Image Proc 16:1479–1492. https://doi.org/10.1007/s11554-016-0655-2
Erichson NB et al (2019b) Randomized dynamic mode decomposition. SIAM J Appl Dyn Syst 18:1867–1891. https://doi.org/10.1137/18M1215013
Erichson NB et al (2020) Sparse principal component analysis via variable projection. SIAM J Appl Math 80:977–1002. https://doi.org/10.1137/18m1211350
Frank Pai P (2013) Time–frequency analysis for parametric and non-parametric identification of nonlinear dynamical systems. Mech Syst Signal Process 36(2):332–353. https://doi.org/10.1016/j.ymssp.2012.12.002
Fujii K, Kawahara Y (2019) Supervised dynamic mode decomposition via multitask learning. Pattern Recogn Lett 122:7–13. https://doi.org/10.1016/j.patrec.2019.02.010
Giannakis D (2019) Data-driven spectral decomposition and forecasting of ergodic dynamical systems. Appl Comput Harmon Anal 47(2):338–396. https://doi.org/10.1016/j.acha.2017.09.001
Hartman D, Mestha LK (2017) A deep learning framework for model reduction of dynamical systems. IEEE Conference on Control Technology and Applications (CCTA), Mauna Lani, pp 1917–1922. https://doi.org/10.1109/CCTA.2017.8062736
He J, Xu J (2019) MgNet: a unified framework of multigrid and convolutional neural network. Sci China Math 62:1331–1354. https://doi.org/10.1007/s11425-019-9547-2
Ibañez R et al (2020) On the data-driven modeling of reactive extrusion. Fluids 5:94. https://doi.org/10.3390/fluids5020094
Kaptanoglu AA, Morgan KD, Hansen CJ, Brunton SL (2020) Characterizing magnetized plasmas with dynamic mode decomposition. Phys Plasmas 27:032108. https://doi.org/10.1063/1.5138932
Kumar R, Srivastava S, Gupta JRP, Mohindru A (2018) Diagonal recurrent neural network based identification of nonlinear dynamical systems with Lyapunov stability based adaptive learning rates. Neurocomputing 287:102–117. https://doi.org/10.1016/j.neucom.2018.01.073
Lechner M, Hasani R, Rus D, Grosu R (2020) Gershgorin loss stabilizes the recurrent neural network compartment of an end-to-end robot learning scheme, 2020 International Conference on Robotics and Automation (ICRA), IEEE. https://doi.org/10.1109/ICRA40945.2020.9196608
Lee K, Carlberg KT (2020) Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. J Comput Phys 404:2020. https://doi.org/10.1016/j.jcp.2019.108973
Lee JH, Shin J, Realff MJ (2018) Machine learning: overview of the recent progresses and implications for the process systems engineering field. Comput Chem Eng 114:111–121. https://doi.org/10.1016/j.compchemeng.2017.10.008
Li S-J, Liu Y-X (2006) An improved approach to nonlinear dynamical system identification using PID neural networks. Int J Nonlinear Sci Numer Simul 7(2):177–182. https://doi.org/10.1515/IJNSNS.2006.7.2.177
Li S et al (2019) Discovering time-varying aerodynamics of a prototype bridge by sparse identification of nonlinear dynamical systems. Phys Rev E 100:022220. https://doi.org/10.1103/PhysRevE.100.022220
Lu Y, Zhong A, Li Q, Dong B (2018) Beyond finite layer neural networks: bridging deep architectures and numerical differential equations, Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80
Lusch B, Nathan Kutz J, Brunton SL (2018) Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun 9:4950. https://doi.org/10.1038/s41467-018-07210-0
Mangan NM, Brunton SL et al (2016) Inferring biological networks by sparse identification of nonlinear dynamics. IEEE Trans Mol Biol Multi-Scale Commun 2(1). https://doi.org/10.1109/TMBMC.2016.2633265
Mangan NM, Askham T et al (2019) Model selection for hybrid dynamical systems via sparse regression. Proc R Soc A 475:20180534. https://doi.org/10.1098/rspa.2018.0534
Murthy N, Saravana R, Rajendra P (2018) Critical comparison of north east monsoon rainfall for different regions through analysis of means technique. Mausam. 69:411–418
Narasimha Murthy KV, Saravana R, Rajendra P (2019) Unobserved component modeling for seasonal rainfall patterns in Rayalaseema region, India 1951–2015. Meteorog Atmos Phys 131:1387–1399. https://doi.org/10.1007/s00703-018-0645-y
Qiao J-F, Han H-G (2012) Identification and modeling of nonlinear dynamical systems using a novel self-organizing RBF-based approach. Automatica 48(8):1729–1734. https://doi.org/10.1016/j.automatica.2012.05.034
Qin T, Wu K, Xiu D (2019) Data driven governing equations approximation using deep neural networks. J Comput Phys 395:620–635. https://doi.org/10.1016/j.jcp.2019.06.042
Rahul-Vigneswaran K, Sachin-Kumar S, Mohan N, Soman KP (2019) Dynamic mode decomposition based feature for image classification. TENCON 2019–2019 IEEE Region 10 Conference (TENCON), Kochi, pp 745–750. https://doi.org/10.1109/TENCON.2019.8929663
Raissi M (2018) Deep hidden physics models: deep learning of nonlinear partial differential equations. J Mach Learn Res 19:1–24 https://www.jmlr.org/papers/volume19/18-046/18-046.pdf
Raissi M, Karniadakis GE (2018) Hidden physics models: Machine learning of nonlinear partial differential equations. J Comput Phys 357:125–141. https://doi.org/10.1016/j.jcp.2017.11.039
Raissi M, Perdikaris P, Karniadakis GE (2017) Machine learning of linear differential equations using Gaussian processes. J Comput Phys 348:683–693. https://doi.org/10.1016/j.jcp.2017.07.050
Rajendra P, Subbarao A, Ramu G et al (2018) Prediction of drug solubility on parallel computing architecture by support vector machines. Netw Model Anal Health Inform Bioinform 7:13. https://doi.org/10.1007/s13721-018-0174-0
Rajendra P, Subbarao A, Ramu G, Boadh R (2019a) Identification of nonlinear systems through convolutional neural network. IJRTE 8(3):2019 https://www.ijrte.org/wp-content/uploads/papers/v8i3/C5058098319.pdf
Rajendra P, Murthy KVN, Subbarao A et al (2019b) Use of ANN models in the prediction of meteorological data. Model Earth Syst Environ 5:1051–1058. https://doi.org/10.1007/s40808-019-00590-2
Rao AS, Sainath S, Rajendra P, Ramu G (2018) Mathematical modeling of hydromagnetic Casson non-newtonian nanofluid convection slip flow from an isothermal sphere. Nonlinear Eng 8(1):645–660. https://doi.org/10.1515/nleng-2018-0016
Regazzoni F, Dedè L, Quarteroni A (2019) Machine learning for fast and reliable solution of time-dependent differential equations. J Comput Phys 397:108852. https://doi.org/10.1016/j.jcp.2019.07.050
Rudy SH, Brunton SL et al (2017) Data-driven discovery of partial differential equations. Sci Adv 3(4). https://doi.org/10.1126/sciadv.1602614
Rudy SH, Nathan Kutz J, Brunton SL (2019a) Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. J Comput Phys 396:483–506. https://doi.org/10.1016/j.jcp.2019.06.056
Rudy S, Alla A, Brunton SL, Nathan Kutz J (2019b) Data-driven identification of parametric partial differential equations. SIAM J Appl Dyn Syst 18(2):643–660. https://doi.org/10.1137/18M1191944
San O, Maulik R (2018) Neural network closures for nonlinear model order reduction. Adv Comput Math, Vol 44:1717–1750. https://doi.org/10.1007/s10444-018-9590-z
Schaeffer H (2017) Learning partial differential equations via data discovery and sparse optimization. Proc R Soc A 473:20160446. https://doi.org/10.1098/rspa.2016.0446
Schulze P, Unger B (2016) Data-driven interpolation of dynamical systems with delay. Syst Control Lett 97:125–131. https://doi.org/10.1016/j.sysconle.2016.09.007
Sirignano J, Spiliopoulos K (2018) DGM: a deep learning algorithm for solving partial differential equations. J Comput Phys 375:1339–1364. https://doi.org/10.1016/j.jcp.2018.08.029
Song L, Huang J, Smola A, Fukumizu K (2009) Hilbert space embeddings of conditional distributions with applications to dynamical systems. ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning, Pages 961–968. https://doi.org/10.1145/1553374.1553497
Suarez JL, Garca S, Herrera F (2020) pyDML: a Python library for distance metric learning. J Mach Learn Res 21:1–7
Subba Rao A et al (2017) Numerical study of non-newtonian polymeric boundary layer flow and heat transfer from a permeable horizontal isothermal cylinder. Front Heat Mass Transfer, 9–2. https://doi.org/10.5098/hmt.9.2
Suna L, Gaoa H et al (2020) Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Comput Methods Appl Mech Eng 361:112732. https://doi.org/10.1016/j.cma.2019.112732
Suzuki K, Mori H, Ogata T (2018) Motion switching with sensory and instruction signals by designing dynamical systems using deep neural network. IEEE Robot Autom Lett 3(4). https://doi.org/10.1109/LRA.2018.2853651
Takeishix N, Kawaharay Y, Yairi T (2017) Learning Koopman invariant subspaces for dynamic mode decomposition, NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp 1130–1140
Trischler AP, D’Eleuterio GMT (2016) Synthesis of recurrent neural networks for dynamical system simulation. Neural Netw 80:67–78. https://doi.org/10.1016/j.neunet.2016.04.001
Wang Y-J, Lin C-T (1998) Runge–Kutta neural network for identification of dynamical systems in high accuracy. IEEE Trans Neural Netw 9(2). https://doi.org/10.1109/72.661124
Watson JR, Gelbaum Z, Titus M, Zoch G, Wrathall D (2020) Identifying multiscale spatio-temporal patterns in human mobility using manifold learning. Peer J Comput Sci 6:e276. https://doi.org/10.7717/peerj-cs.276
Wei Z, Zhang Z, Gu WW, Fang N (2020) Visualization classification and prediction based on data mining. Journal of Physics: Conference Series, Vol 1550, Machine Learning, Intelligent data analysis and Data Mining https://doi.org/10.1088/1742-6596/1550/3/032122
Weimer D, Scholz-Reiter B, Shpitalni M (2016) Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Ann Manuf Technol 65:417–420. https://doi.org/10.1016/j.cirp.2016.04.072
Weinan E (2017) A proposal on machine learning via dynamical systems. Commun Math Stat 5:1–11. https://doi.org/10.1007/s40304-017-0103-z
Wolfe B, James MR, Singh S (2005) Learning predictive state representations in dynamical systems without reset. ICML '05: Proceedings of the 22nd international conference on Machine learning, August 2005, pages 980–987. https://doi.org/10.1145/1102351.1102475
Wu K, Xiu D (2020) Data-driven deep learning of partial differential equations in modal space. J Comput Phys 408:109307. https://doi.org/10.1016/j.jcp.2020.109307
Wu Z, Yang G et al (2018) A weighted deep representation learning model for imbalanced fault diagnosis in cyber-physical systems. Sensors 18:1096. https://doi.org/10.3390/s18041096
Yu Y, Zhang Y, Qian S, Wang S, Hu Y, Yin B (2020) A low rank dynamic mode decomposition model for short-term traffic flow prediction. IEEE Trans Intell Transp Syst. https://doi.org/10.1109/TITS.2020.2994910
Zhang W, Wu P, Peng Y, Liu D (2019) Roll motion prediction of unmanned surface vehicle based on coupled CNN and LSTM. Future Internet 11:243. https://doi.org/10.3390/fi11110243
Zhang S, Duan X, Li C, Liang M (2021) Pre-classified reservoir computing for the fault diagnosis of 3D printers. Mech Syst Signal Process 146:106961. https://doi.org/10.1016/j.ymssp.2020.106961
Zhu F, Ye F, Fu Y et al (2019) Electrocardiogram generation with a bidirectional LSTM-CNN generative adversarial network. Sci Rep 9:6734. https://doi.org/10.1038/s41598-019-42516-z
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rajendra, P., Brahmajirao, V. Modeling of dynamical systems through deep learning. Biophys Rev 12, 1311–1320 (2020). https://doi.org/10.1007/s12551-020-00776-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12551-020-00776-4