Skip to main content
SpringerLink
Log in

Inference of hidden variables in systems of differential equations with genetic programming

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

The data-driven modeling of dynamical systems is an important scientific activity, and many studies have applied genetic programming (GP) to the task of automatically constructing such models in the form of systems of ordinary differential equations (ODEs). These previous studies assumed that data measurements were available for all variables in the system, whereas in real-world settings, it is typically the case that one or more variables are unmeasured or “hidden.” Here, we investigate the prospect of automatically constructing ODE models of dynamical systems from time series data with GP in the presence of hidden variables. Several examples with both synthetic and physical systems demonstrate the unique challenges of this problem and the circumstances under which it is possible to reverse-engineer both the form and parameters of ODE models with hidden variables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. F. Adachi, T. Washio, H. Motoda, Scientific discovery of dynamic models based on scale-type constraints. IPSJ Digit. Cour. 2, 607–619 (2006)

    Article  Google Scholar 

  2. L.A. Aguirre, U.S. Freitas, C. Letellier, J. Maquet, Structure-selection techniques applied to continuous-time nonlinear models. Physica D 158, 1–18 (2001)

    Article  MATH  Google Scholar 

  3. L.A. Aguirre, C. Letellier, Modeling nonlinear dynamics and chaos: a review. Math. Probl. Eng. (2009). doi:10.1155/2009/238960

    MathSciNet  Google Scholar 

  4. D.P. Ahalpara, A. Sen, in Proceedings of the EuroGP 2011, ed. by S. Silva, J.A. Foster, M. Nicolau, P. Machado, M. Giacobini. A sniffer technique for an efficient deduction of model dynamical equations using genetic programming (Springer, Berlin, 2011), pp. 1–12

  5. S. Ando, E. Sakamoto, H. Iba, Evolutionary modeling and inference of gene network. Inf. Sci. 145(3–4), 237–259 (2002)

    Article  MathSciNet  Google Scholar 

  6. H. Andrew, in Proceedings of the 2nd International Conference on Adaptive Computing in Engineering Design and Control. System identification using genetic programming (University of Plymouth, Plymouth, 1996), pp. 57–62

  7. P.J. Angeline, D.B. Fogel, in Proceedings of the SPIE. An evolutionary program for the identification of dynamical systems (1997), pp. 409–417

  8. B. Argemí, J. Saurina, Study of the degradation of 5-azacytidine as a model of unstable drugs using a stopped-flow method and further data analysis with multivariate curve resolution. Talanta 74, 176–182 (2007)

    Article  Google Scholar 

  9. K.J. Åström, P. Eykhoff, System identification—a survey. Automatica 7(2), 123–162 (1971)

    Article  MATH  Google Scholar 

  10. E. Baake, M. Baake, H.G. Bock, K.M. Briggs, Fitting ordinary differential equations to chaotic data. Phys. Rev. A 45(8), 5524–5529 (1992)

    Article  Google Scholar 

  11. V. Babovic, M. Keijzer, in Proceedings of the 4th International Conference on Hydroinformatics. Evolutionary algorithms approach to induction of differential equations (Iowa City, USA, 2000), pp. 251–258

  12. R. Bakker, J.C. Schouten, C.L. Giles, F. Takens, C.M. van den Bleek, Learning chaotic attractors. Neural Comput. 12(10), 2355–2383 (2000)

    Article  Google Scholar 

  13. W. Banzhaf, P. Nordin, R.E. Keller, F.D. Francone, Genetic Programming: An Introduction on the Automatic Evolution of Computer Programs and Its Applications (Morgan Kaufmann, San Francisco, 1997)

    Google Scholar 

  14. J.A. Beisler, Isolation, characterization, and properties of a labile hydrolysis product of the antitumor nucleoside, 5-azacytidine. J. Med. Chem. 21(2), 204–208 (1978)

    Article  Google Scholar 

  15. H.S. Bernardino, H.J.C. Barbosa, Inferring systems of ordinary differential equations via grammar-based immune programming. LNCS 6825, 198–211 (2011)

    Google Scholar 

  16. C.M. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006)

    MATH  Google Scholar 

  17. S. Boccaletti, The Synchronized Dynamics of Complex Systems (Elsevier, Amsterdam, 2008)

    MATH  Google Scholar 

  18. J. Bongard, H. Lipson, Automated reverse engineering of nonlinear dynamical systems. PNAS 104(24), 9943–9948 (2007)

    Article  MATH  Google Scholar 

  19. E. Bradley, M. Easley, R. Stolle, Reasoning about nonlinear system identification. Artif. Intell. 133, 139–188 (2001)

    Article  MATH  Google Scholar 

  20. M. Brameier, W. Banzhaf, Linear Genetic Programming (Springer, New York, 2007)

    MATH  Google Scholar 

  21. J.L. Breeden, A. Hübler, Reconstructing equations of motion from experimental data with unobserved variables. Phys. Rev. A 42(10), 5817–5826 (1990)

    Article  MathSciNet  Google Scholar 

  22. W. Bridewell, P. Langley, L. Todorovski, S. Dzeroski, Inductive process modeling. Mach. Learn. 71, 1–32 (2008)

    Article  Google Scholar 

  23. H. Cao, L. Kang, Y. Chen, J. Yu, Evolutionary modeling of systems of ordinary differential equations with genetic programming. Genet. Program Evolvable Mach. 1(4), 309–337 (2000)

    Article  MATH  Google Scholar 

  24. O.-T. Chis, J.R. Banga, E. Balsa-Canto, Structural identifiability of systems biology models: a critical comparison of methods. PLoS ONE 6(11), e27755 (2011)

    Article  Google Scholar 

  25. K.-H. Cho, S.-Y. Shin, H.-W. Kim, O. Wolkenhauer, B. McFerran, W. Kolch, Mathematical modeling of the influence of RKIP on the ERK signaling pathway. LNCS 2602, 127–141 (2003)

    Google Scholar 

  26. D. Clery, D. Voss, All for one and one for all. Science 308, 809 (2005)

    Article  Google Scholar 

  27. J. Cremers, A. Hübler, Construction of differential equations from experimental data. Zeitschrift für Naturforschung A 42(8), 797–802 (1987)

    Google Scholar 

  28. J.P. Crutchfield, B.S. McNamara, Equations of motion from a data series. Complex Syst. 1(3), 417–452 (1987)

    MathSciNet  MATH  Google Scholar 

  29. S. Džeroski, L. Todorovski, in Proceedings of the 10th International Conference on Machine Learning. Discovering dynamics (Kaufmann, Amherst, 1993), pp. 97–103

  30. J. Evans, A. Rzhetsky, Machine science. Science 329, 3994400 (2010)

    Article  Google Scholar 

  31. G. Gouesbet, Reconstruction of the vector fields of continuous dynamical systems from numerical scalar time series. Phys. Rev. A 43(10), 5321–5331 (1991)

    Article  MathSciNet  Google Scholar 

  32. P. Grassberger, I. Procaccia, Measuring the strangeness of strange attractors. Physica D 9, 189–208 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  33. G.J. Gray, D.J. Murray-Smith, Y. Li, K.C. Sharman, in Proceedings of the 2nd International Conference on Genetic Algorithms in Engineering Systems: Innovations and Applications. Nonlinear model structure identification using genetic programming (Inst of Electrical Engineers, London, 1997), pp. 308–313

  34. G.J. Gray, D.J. Murray-Smith, Y. Li, K.C. Sharman, T. Weinbrenner, Nonlinear model structure identification using genetic programming. Control Eng. Pract. 6, 1341–1352 (1998)

    Article  Google Scholar 

  35. M.S. Grewal, K. Glover, Identifiability of linear and nonlinear dynamical systems. IEEE Trans. Automat. Control 21(6), 833–837 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  36. S. Hengl, C. Kreutz, J. Timmer, T. Maiwald, Data-based identifiability analysis of non-linear dynamical models. Bioinformatics 23(19), 2612–2618 (2007)

    Article  Google Scholar 

  37. A.L. Hodgkin, A.F. Huxley, A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Phys. 117, 500–544 (1952)

    Google Scholar 

  38. X. Hong, R.J. Mitchell, S. Chen, C.J. Harris, K. Li, G.W. Irwin, Model selection approaches for non-linear system identification: a review. Int. J. Syst. Sci. 39(10), 925–946 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  39. H. Iba, H. de Garis, T. Sato, in Proceedings of the PPSN III, ed. by Y. Davidor, H.-P. Schwefel, R. Männer. Genetic programming with local hill-climbing (Springer, London, 1994), pp. 302–311

  40. H. Iba, Inference of differential equation models by genetic programming. Inf. Sci. 178(23), 4453–4468 (2008)

    Article  Google Scholar 

  41. S.J. Julier, J.K. Uhlmann, in Proceedings of the International Symposium on Aerospace/Defense Sensing, Simulation, and Controls. A new extension of the Kalman filter to nonlinear systems (1997), pp. 182–193

  42. M.A. Kaboudan, Genetic programming prediction of stock prices. Comput. Econ. 16(3), 207–236 (2000)

    Article  MATH  Google Scholar 

  43. M.B. Kennel, R. Brown, H.D. Abarbanel, Determining embedding dimension for phase-space reconstruction using a geometrical construction. Phys. Rev. A 45(6), 3403–3411 (1992)

    Article  Google Scholar 

  44. R.D. King, K.E. Whelan, F.M. Jones, P.G.K. Reiser, C.H. Bryant, S.H. Muggleton, D.B. Kell, S.G. Oliver, Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427, 247–252 (2004)

    Article  Google Scholar 

  45. L.D. Kissinger, N.L. Stemm, Determination of the antileukemia agents cytarabine and azacitidine and their respective degradation products by high-performance liquid chromatography. J. Chromatogr. 353, 309–318 (1986)

    Article  Google Scholar 

  46. J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, Cambridge, 1992)

    MATH  Google Scholar 

  47. P. Langley, Data-driven discovery of physical laws. Cognitive Sci. 5, 31–54 (1981)

    Article  Google Scholar 

  48. P. Langley, H.A. Simon, G.L. Bradshaw, J.M. Zytkow, Scientific Discovery: Computational Explorations of the Creative Processes (MIT Press, Cambridge, 1987)

    Google Scholar 

  49. P. Langley, J. Sanchez, L. Todorovski, S. Džeroski, in Proceedings of the 19th International Conference on Machine Learning. Inducing process models (Morgan Kaufmann, Sydney, 2002), pp. 347–354

  50. L. Ljung, System Identification—Theory For the User, 2nd edn. (Prentice Hall, Upper Saddle River, NJ, 1999)

    Google Scholar 

  51. E.N. Lorenz, Deterministic non-periodic flow. J. Atmos. Sci. 20(2), 130–141 (1963)

    Article  MathSciNet  Google Scholar 

  52. G. Mamani, J. Becedas, V.F. Batlle, H. Sira-Ramírez, Algebraic observer to estimate unmeasured state variables of DC motors. Eng. Lett. 16(2), 248–255 (2008)

    Google Scholar 

  53. D.J. Montana, Strongly typed genetic programming. Evol. Comput. 3(2), 199–230 (1995)

    Article  Google Scholar 

  54. B.S. Mulloy, R.L. Riolo, R.S. Savit, in Proceedings of the GECCO 1996. Dynamics of genetic programming and chaotic time series prediction (MIT Press, Cambridge, MA, 1996), pp. 166–174

  55. C. Neely, P. Weller, R. Dittmara, Is technical analysis in the foreign exchange market profitable? A genetic programming approach. J. Financ. Quant. Anal. 32, 405–426 (1997)

    Article  Google Scholar 

  56. H. Oakley, Two scientific applications of genetic programming: stack filters and non-linear equation fitting to chaotic data, in Advances Genetic Programming, ed. by K.E. Kinnear Jr (MIT Press, Cambridge, MA, 1994), pp. 369–389

    Google Scholar 

  57. N.H. Packard, J.P. Crutchfield, J.D. Farmer, R.S. Shaw, Geometry from a time series. Phys. Rev. Lett. 45(9), 712–716 (1980)

    Article  Google Scholar 

  58. S.N. Pnevmatikos (ed.), Singularities & Dynamical Systems (Elsevier, Amsterdam, 1985)

    MATH  Google Scholar 

  59. W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes: The Art of Scientific Computing, 3rd edn. (Cambridge University Press, Cambridge, 2007)

    MATH  Google Scholar 

  60. L. Qian, H. Wang, E.R. Dougherty, Inference of noisy nonlinear differential equation models for gene regulatory networks using genetic programming and Kalman filtering. IEEE Trans. Signal Process. 56(7), 3327–3339 (2008)

    Article  MathSciNet  Google Scholar 

  61. A. Raue, V. Becker, U. Klingmüller, J. Timmer, Identifiability and observability analysis for experimental design in nonlinear dynamical models. Chaos 20, 045105 (2010)

    Article  Google Scholar 

  62. K. Rodriguez-Vázquez, P.J. Fleming, Evolution of mathematical models of chaotic systems based on multiobjective genetic programming. Knowl. Inf. Syst. 8(2), 235–256 (2005)

    Article  Google Scholar 

  63. K. Routray, G. Deo, Kinetic parameter estimation for a multiresponse nonlinear reaction model. AIChE J. 51(6), 1733–1746 (2005)

    Article  Google Scholar 

  64. S. Russell, P. Norvig, Artificial Intelligence: A Modern Approach, 3rd edn. (Prentice Hall, Upper Saddle River, NJ, 2009)

    Google Scholar 

  65. C. Ryan, J.J. Collins, M. O’Neill, in Proceedings of the EuroGP 1998, ed. by W. Banzhaf, R. Poli, M. Schoenauer, T.C. Fogarty. Grammatical evolution: evolving programs for an arbitrary language (Springer, London, 1998), pp. 83–96

  66. E. Sakamoto, H. Iba, in Proceedings of the CEC 2001. Inferring a system of differential equations for a gene regulatory network by using genetic programming (2001), pp. 720–726

  67. M.D. Schmidt, H. Lipson, Coevolution of fitness predictors. IEEE Trans. Evol. Comput. 12(6), 736–749 (2008)

    Article  Google Scholar 

  68. M.D. Schmidt, H. Lipson, Distilling free-form natural laws from experimental data. Science 324(5923), 81–85 (2009)

    Article  Google Scholar 

  69. M.D. Schmidt, H. Lipson, in Proceedings of the GECCO 2010. Age-fitness Pareto optimization (2010), pp. 543–544

  70. M.D. Schmidt, H. Lipson, Age-fitness Pareto optimization. Genet. Program. Theory Pract. 8, 129–146 (2010)

    Google Scholar 

  71. M. Schwabacher, P. Langley, in Proceedings of the ICML 2001, ed. by C.E. Brodley, A.P. Danyluk. Discovering communicable scientific knowledge from spatio-temporal data (Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001), pp. 489–496

  72. K.V. Sharp, R.J. Adrian, Transition from laminar to turbulent flow in liquid filled microtubes. Exp. Fluids 36(5), 741–747 (2004)

    Article  Google Scholar 

  73. J. Sjöberg, Q. Zhang, L. Ljung, A. Benveniste, B. Delyon, P.-Y. Glorennec, H. Hjalmarsson, A. Juditsky, Nonlinear black-box modeling in system identification: a unified overview. Automatica 31(12), 1691–1724 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  74. W.E. Stewart, M. Caracotsios, J.P. Sørensen, Parameter estimation from multiresponse data. AIChE J. 38(5), 641–650 (1992)

    Article  Google Scholar 

  75. S.H. Strogatz, Exploring complex networks. Nature 410, 268–276 (2001)

    Article  Google Scholar 

  76. A. Szalay, J. Gray, 2020 Computing: science in an exponential world. Nature 440, 413–414 (2006)

    Article  Google Scholar 

  77. F. Takens, Detecting strange attractors in turbulence. Lect. Notes Math. 898, 366–381 (1981)

    Article  MathSciNet  Google Scholar 

  78. L. Todorovski, S. Džeroski, in Proceedings of the ICML 1997. Declarative bias in equation discovery (Kaufmann, Nashville, 1997), pp 376–384

  79. S. Vajda, H. Rabitz, Identifiability and distinguishability of general reaction systems. J. Phys. Chem. 98, 5265–5271 (1994)

    Article  Google Scholar 

  80. H.U. Voss, J. Timmer, J. Kurths, Nonlinear dynamical system identification from uncertain and indirect measurements. Int. J. Bifurcat. Chaos 14(6), 1905–1933 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  81. É. Walter, L. Pronzato, Identification of Parametric Models From Experimental Data (Springer, New York, 1997)

    MATH  Google Scholar 

  82. D. Waltz, B.G. Buchanan, Automating science. Science 324(5923), 43–44 (2009)

    Article  Google Scholar 

  83. T. Washio, H. Motoda, Y. Niwa, in Proceedings of the ICML 2000. Enhancing the plausibility of law equation discovery (2000), pp. 1127–1134

  84. H.-L. Wei, S.A. Billings, Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information. Int. J. Model. Identif. Control 3(4), 341–356 (2008)

    Article  Google Scholar 

  85. P.A. Whigham, in Proceedings of the Workshop on Genetic Programming: From Theory to Real-World Applications, ed. by J.P. Rosca. Grammatically-based genetic programming (Tahoe City, California, 1995), pp. 33–41

  86. S. Winkler, M. Affenzeller, S. Wagner, New methods for the identification of nonlinear model structures based upon genetic programming techniques. J. Syst. Sci. 31, 5–14 (2005)

    Google Scholar 

  87. T. Yoshida, L.E. Jones, S.P. Ellner, G.F. Fussmann, N.G. Hairston Jr, Rapid evolution drives ecological dynamics in a predator-prey system. Nature 424, 303–306 (2003)

    Article  Google Scholar 

Download references

Acknowledgments

Support was provided by the Tri-Institutional Training Program in Computational Biology and Medicine, National Science Foundation grant ECCS 0941561 on Cyber-enabled Discovery and Innovation (CDI), National Institutes of Health NIDA grant RC2 DA028981, and the U.S. Defense Threat Reduction Agency (DTRA) grant HDTRA 1-09-1-0013.

Author information

Authors and Affiliations

  1. Biological Statistics and Computational Biology, Cornell University, 1198 Comstock Hall, Ithaca, NY, 14853-2601, USA

    Theodore W. Cornforth

  2. Mechanical and Aerospace Engineering, Cornell University, 239 Upson Hall, Ithaca, NY, 14853-7501, USA

    Hod Lipson

Authors
  1. Theodore W. Cornforth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hod Lipson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Theodore W. Cornforth.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cornforth, T.W., Lipson, H. Inference of hidden variables in systems of differential equations with genetic programming. Genet Program Evolvable Mach 14, 155–190 (2013). https://doi.org/10.1007/s10710-012-9175-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-012-9175-4

Keywords

Search

Navigation