Deep Learning of Biological Models from Data: Applications to ODE Models

Abstract

Mathematical equations are often used to model biological processes. However, for many systems, determining analytically the underlying equations is highly challenging due to the complexity and unknown factors involved in the biological processes. In this work, we present a numerical procedure to discover dynamical physical laws behind biological data. The method utilizes deep learning methods based on neural networks, particularly residual networks. It is also based on recently developed mathematical tools of flow-map learning for dynamical systems. We demonstrate that with the proposed method, one can accurately construct numerical biological models for unknown governing equations behind measurement data. Moreover, the deep learning model can also incorporate unknown parameters in the biological process. A successfully trained deep neural network model can then be used as a predictive tool to produce system predictions of different settings and allows one to conduct detailed analysis of the underlying biological process. In this paper, we use three biological models—SEIR model, Morris–Lecar model and the Hodgkin–Huxley model—to show the capability of our proposed method.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/. Software available from tensorflow.org

  2. Bianchini M, Scarselli F (2014) On the complexity of neural network classifiers: a comparison between shallow and deep architectures. IEEE Trans Neural Netw Learn Syst 25:1553–1565

    Article  Google Scholar 

  3. Bongard J, Lipson H (2007) Automated reverse engineering of nonlinear dynamical systems. Proc Natl Acad Sci 104:9943–9948

    Article  Google Scholar 

  4. Chan S, Elsheikh A (2018) A machine learning approach for efficient uncertainty quantification using multiscale methods. J Comput Phys 354:494–511

    MathSciNet  Article  Google Scholar 

  5. Chen RTQ, Rubanova Y, Bettencourt J, Duvenaud D (2018) Neural ordinary differential equations. Preprint arXiv:1806.07366

  6. Chollet F et al (2015) Keras. https://github.com/fchollet/keras

  7. Daniels BC, Nemenman I (2015a) Automated adaptive inference of phenomenological dynamical models. Nat Commun 6:8133

    Article  Google Scholar 

  8. Daniels BC, Nemenman I (2015b) Efficient inference of parsimonious phenomenological models of cellular dynamics using S-systems and alternating regression. PLoS ONE 10:e0119821

    Article  Google Scholar 

  9. DeAngelis DL, Yurek S (2015) Equation-free modeling unravels the behavior of complex ecological systems. https://doi.org/10.1073/pnas.1503154112

  10. Du KL, Swamy M (2014) Neural networks and statistical learning. Springer, London

    Google Scholar 

  11. Eldan R, Shamir O (2016) The power of depth for feedforward neural networks. Conf Learn Theory 2016:907–940

    Google Scholar 

  12. Fu X, Chang LB, Xiu D (2020) Learning reduced systems via deep neural networks with memory. J Mach Learn Model Comput 2020:1

    Google Scholar 

  13. Giannakis D, Majda AJ (2012) Nonlinear Laplacian spectral analysis for time series with intermittency and low-frequency variability. Proc Natl Acad Sci 109:2222–2227

    MathSciNet  Article  Google Scholar 

  14. Gonzalez-Garcia R, Rico-Martinez R, Kevrekidis IG (1998) Identification of distributed parameter systems: a neural net based approach. Comput Chem Eng 22:S965–S968

    Article  Google Scholar 

  15. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, London

    Google Scholar 

  16. Grimm V, Railsback SF (2005) Individual-based modeling and ecology. Princeton University Press, Princeton

    Google Scholar 

  17. Hau DT, Coiera EW (1995) Learning qualitative models from physiological signals internal accession date only. Technical report

  18. Hesthaven J, Ubbiali S (2018) Non-intrusive reduced order modeling of nonlinear problems using neural networks. J Comput Phys 363:55–78

    MathSciNet  Article  Google Scholar 

  19. Hethcote HW (2000) The mathematics of infectious diseases. SIAM Rev 42:599–653

    MathSciNet  Article  Google Scholar 

  20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  21. Hodgkin AL, Huxley AF (Aug 1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1392413/

  22. Kevrekidis IG, Gear CW, Hyman JM, Kevrekidid PG, Runborg O, Theodoropoulos C et al (2003) Equation-free, coarse-grained multiscale computation: enabling mocroscopic simulators to perform system-level analysis. Commun Math Sci 1:715–762

    MathSciNet  Article  Google Scholar 

  23. Khoo Y, Lu J, Ying L (2018) Solving parametric PDE problems with artificial neural networks. Preprint arXiv:1707.03351

  24. Kingma DP, Ba J (2017) Adam: a method for stochastic optimization. arXiv:1412.6980

  25. Long Z, Lu Y, Ma X, Dong B (2018) PDE-net: learning PDEs from data. In: Proceedings of the 35th international conference on machine learning, 10–15 edn. vol 80 of proceedings of machine learning research, Stockholmsmässan, Stockholm, Sweden, pp 3208–3216

  26. Mangan NM, Brunton SL, Proctor JL, Kutz JN (2016) Inferring biological networks by sparse identification of nonlinear dynamics. IEEE Trans Mol Biol Multiscale Commun 2:52–63

    Article  Google Scholar 

  27. Mardt A, Pasquali L, Wu H, Noe F (2018) VAMPnets for deep learning of molecular kinetics. Nat Commun 9:5

    Article  Google Scholar 

  28. Montufar GF, Pascanu R, Pascanu K, Bengio Y (2014) On the number of linear regions of deep neural networks. Adv Neural Inf Process Syst 2014:2924–2932

    Google Scholar 

  29. Morris C, Lecar H (1981) Voltage oscillations in the barnacle giant muscle fiber. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1327511/

  30. Perretti CT, Munch SB, Sugihara G (2013) Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data. Proc Natl Acad Sci USA 110:5253–5257. https://doi.org/10.1073/pnas.1216076110

    Article  Google Scholar 

  31. Poggio T, Mhaskar H, Rosasco L, Miranda B, Liao Q (2017) Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review. Int J Autom Comput 14:503–519

    Article  Google Scholar 

  32. Qin T, Wu K, Xiu D (2019) Data driven governing equations approximation using deep neural networks. J Comput Phys 395:620–635

    MathSciNet  Article  Google Scholar 

  33. Qin T, Chen Z, Jakeman J, Xiu D (2020) Data-driven learning of non-autonomous systems. SIAM J Sci Comput 2020:1

    Google Scholar 

  34. Radhakrishnan K, Hindmarsh AC (1993) Description and use of LSODE, the Livemore solver for ordinary differential equations. Technical report, Lawrence Livermore National Laboratory (LLNL), Livermore. https://doi.org/10.2172/15013302. http://www.osti.gov/servlets/purl/15013302-7Xcq5X/native/

  35. Ray D, Hesthaven J (2018) An artificial neural network as a troubled-cell indicator. J Comput Phys 367:166–191

    MathSciNet  Article  Google Scholar 

  36. Rinzel J, Ermentrout G (1998) Analysis of neural excitability and oscillations. In: Koch C, Segev I (eds) Methods in neuronal modeling, 2nd edn. MIT Press, London, pp 251–291

    Google Scholar 

  37. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  38. Schmidt M, Lipson H (2009) Distilling free-form natural laws from experimental data. Science 324:81–85

    Article  Google Scholar 

  39. Schmidt MD, Vallabhajosyula RR, Jenkins JW, Hood JE, Soni AS, Wikswo JP, Lipson H (2011) Automated refinement and inference of analytical models for metabolic networks. Phys Biol 8:055011

    Article  Google Scholar 

  40. Sugihara G, May R, Ye H, Hsieh C, Deyle E, Fogarty M, Munch S (2012) Detecting causality in complex ecosystems. Science 338:496–500

    Article  Google Scholar 

  41. Tripathy R, Bilionis I (2018) Deep UQ: learning deep neural network surrogate model for high dimensional uncertainty quantification. J Comput Phys 375:565–588

    MathSciNet  Article  Google Scholar 

  42. Tsumoto K, Kitajima H, Yoshinaga T, Aihara K, Kawakami H (2006) Bifurcations in Morris–Lecar neuron model. Neurocomputing 69:293–316. https://doi.org/10.1016/j.neucom.2005.03.006

    Article  Google Scholar 

  43. Voss HU, Kolodner P, Abel M, Kurths J (1999) Amplitude equations from spatiotemporal binary-fluid convection data. Phys Rev Lett 83:3422

    Article  Google Scholar 

  44. Wang Y, Cheung SW, Chung ET, Efendiev Y, Wang M (2018) Deep multiscale model learning. Preprint arXiv:1806.04830

  45. Weinan E, Engquist B, Huang Z (2003) Heterogeneous multiscale method: a general methodology for multiscale modeling. Phys. Rev. B 67:092101

    Article  Google Scholar 

  46. Wood SN, Thomas MB (1999) Super-sensitivity to structure in biological models. Proc R Soc B Biol Sci 266:565–570. https://doi.org/10.1098/rspb.1999.0673

    Article  Google Scholar 

  47. Wu K, Xiu D (2019) Numerical aspects for approximating governing equations using data. J Comput Phys 384:200–221

    MathSciNet  Article  Google Scholar 

  48. Ye H, Beamish RJ, Glaser SM, Grant SCH, Hsieh C, Richards LJ, Schnute JT, Sugihara G (2015) Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proc Natl Acad Sci 112:1569–E1576

    Article  Google Scholar 

  49. Yodzis P (1988) The indeterminacy of ecological interactions as perceived through perturbation. Technical report, p 2

  50. Zhu Y, Zabaras N (2018) Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. J Comput Phys 366:415–447

    MathSciNet  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Dongbin Xiu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Su, WH., Chou, CS. & Xiu, D. Deep Learning of Biological Models from Data: Applications to ODE Models. Bull Math Biol 83, 19 (2021). https://doi.org/10.1007/s11538-020-00851-7

Download citation

Keywords

  • Deep neural network
  • Residual network
  • Mathematical biology
  • Governing equation discovery