Genetic Programming and Evolvable Machines

, Volume 14, Issue 2, pp 155–190 | Cite as

Inference of hidden variables in systems of differential equations with genetic programming



The data-driven modeling of dynamical systems is an important scientific activity, and many studies have applied genetic programming (GP) to the task of automatically constructing such models in the form of systems of ordinary differential equations (ODEs). These previous studies assumed that data measurements were available for all variables in the system, whereas in real-world settings, it is typically the case that one or more variables are unmeasured or “hidden.” Here, we investigate the prospect of automatically constructing ODE models of dynamical systems from time series data with GP in the presence of hidden variables. Several examples with both synthetic and physical systems demonstrate the unique challenges of this problem and the circumstances under which it is possible to reverse-engineer both the form and parameters of ODE models with hidden variables.


Genetic programming Ordinary differential equations Hidden variables Modeling Symbolic identification 


  1. 1.
    F. Adachi, T. Washio, H. Motoda, Scientific discovery of dynamic models based on scale-type constraints. IPSJ Digit. Cour. 2, 607–619 (2006)CrossRefGoogle Scholar
  2. 2.
    L.A. Aguirre, U.S. Freitas, C. Letellier, J. Maquet, Structure-selection techniques applied to continuous-time nonlinear models. Physica D 158, 1–18 (2001)MATHCrossRefGoogle Scholar
  3. 3.
    L.A. Aguirre, C. Letellier, Modeling nonlinear dynamics and chaos: a review. Math. Probl. Eng. (2009). doi:10.1155/2009/238960 MathSciNetGoogle Scholar
  4. 4.
    D.P. Ahalpara, A. Sen, in Proceedings of the EuroGP 2011, ed. by S. Silva, J.A. Foster, M. Nicolau, P. Machado, M. Giacobini. A sniffer technique for an efficient deduction of model dynamical equations using genetic programming (Springer, Berlin, 2011), pp. 1–12Google Scholar
  5. 5.
    S. Ando, E. Sakamoto, H. Iba, Evolutionary modeling and inference of gene network. Inf. Sci. 145(3–4), 237–259 (2002)MathSciNetCrossRefGoogle Scholar
  6. 6.
    H. Andrew, in Proceedings of the 2nd International Conference on Adaptive Computing in Engineering Design and Control. System identification using genetic programming (University of Plymouth, Plymouth, 1996), pp. 57–62Google Scholar
  7. 7.
    P.J. Angeline, D.B. Fogel, in Proceedings of the SPIE. An evolutionary program for the identification of dynamical systems (1997), pp. 409–417Google Scholar
  8. 8.
    B. Argemí, J. Saurina, Study of the degradation of 5-azacytidine as a model of unstable drugs using a stopped-flow method and further data analysis with multivariate curve resolution. Talanta 74, 176–182 (2007)CrossRefGoogle Scholar
  9. 9.
    K.J. Åström, P. Eykhoff, System identification—a survey. Automatica 7(2), 123–162 (1971)MATHCrossRefGoogle Scholar
  10. 10.
    E. Baake, M. Baake, H.G. Bock, K.M. Briggs, Fitting ordinary differential equations to chaotic data. Phys. Rev. A 45(8), 5524–5529 (1992)CrossRefGoogle Scholar
  11. 11.
    V. Babovic, M. Keijzer, in Proceedings of the 4th International Conference on Hydroinformatics. Evolutionary algorithms approach to induction of differential equations (Iowa City, USA, 2000), pp. 251–258Google Scholar
  12. 12.
    R. Bakker, J.C. Schouten, C.L. Giles, F. Takens, C.M. van den Bleek, Learning chaotic attractors. Neural Comput. 12(10), 2355–2383 (2000)CrossRefGoogle Scholar
  13. 13.
    W. Banzhaf, P. Nordin, R.E. Keller, F.D. Francone, Genetic Programming: An Introduction on the Automatic Evolution of Computer Programs and Its Applications (Morgan Kaufmann, San Francisco, 1997)Google Scholar
  14. 14.
    J.A. Beisler, Isolation, characterization, and properties of a labile hydrolysis product of the antitumor nucleoside, 5-azacytidine. J. Med. Chem. 21(2), 204–208 (1978)CrossRefGoogle Scholar
  15. 15.
    H.S. Bernardino, H.J.C. Barbosa, Inferring systems of ordinary differential equations via grammar-based immune programming. LNCS 6825, 198–211 (2011)Google Scholar
  16. 16.
    C.M. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006)MATHGoogle Scholar
  17. 17.
    S. Boccaletti, The Synchronized Dynamics of Complex Systems (Elsevier, Amsterdam, 2008)MATHGoogle Scholar
  18. 18.
    J. Bongard, H. Lipson, Automated reverse engineering of nonlinear dynamical systems. PNAS 104(24), 9943–9948 (2007)MATHCrossRefGoogle Scholar
  19. 19.
    E. Bradley, M. Easley, R. Stolle, Reasoning about nonlinear system identification. Artif. Intell. 133, 139–188 (2001)MATHCrossRefGoogle Scholar
  20. 20.
    M. Brameier, W. Banzhaf, Linear Genetic Programming (Springer, New York, 2007)MATHGoogle Scholar
  21. 21.
    J.L. Breeden, A. Hübler, Reconstructing equations of motion from experimental data with unobserved variables. Phys. Rev. A 42(10), 5817–5826 (1990)MathSciNetCrossRefGoogle Scholar
  22. 22.
    W. Bridewell, P. Langley, L. Todorovski, S. Dzeroski, Inductive process modeling. Mach. Learn. 71, 1–32 (2008)CrossRefGoogle Scholar
  23. 23.
    H. Cao, L. Kang, Y. Chen, J. Yu, Evolutionary modeling of systems of ordinary differential equations with genetic programming. Genet. Program Evolvable Mach. 1(4), 309–337 (2000)MATHCrossRefGoogle Scholar
  24. 24.
    O.-T. Chis, J.R. Banga, E. Balsa-Canto, Structural identifiability of systems biology models: a critical comparison of methods. PLoS ONE 6(11), e27755 (2011)CrossRefGoogle Scholar
  25. 25.
    K.-H. Cho, S.-Y. Shin, H.-W. Kim, O. Wolkenhauer, B. McFerran, W. Kolch, Mathematical modeling of the influence of RKIP on the ERK signaling pathway. LNCS 2602, 127–141 (2003)Google Scholar
  26. 26.
    D. Clery, D. Voss, All for one and one for all. Science 308, 809 (2005)CrossRefGoogle Scholar
  27. 27.
    J. Cremers, A. Hübler, Construction of differential equations from experimental data. Zeitschrift für Naturforschung A 42(8), 797–802 (1987)Google Scholar
  28. 28.
    J.P. Crutchfield, B.S. McNamara, Equations of motion from a data series. Complex Syst. 1(3), 417–452 (1987)MathSciNetMATHGoogle Scholar
  29. 29.
    S. Džeroski, L. Todorovski, in Proceedings of the 10th International Conference on Machine Learning. Discovering dynamics (Kaufmann, Amherst, 1993), pp. 97–103Google Scholar
  30. 30.
    J. Evans, A. Rzhetsky, Machine science. Science 329, 3994400 (2010)CrossRefGoogle Scholar
  31. 31.
    G. Gouesbet, Reconstruction of the vector fields of continuous dynamical systems from numerical scalar time series. Phys. Rev. A 43(10), 5321–5331 (1991)MathSciNetCrossRefGoogle Scholar
  32. 32.
    P. Grassberger, I. Procaccia, Measuring the strangeness of strange attractors. Physica D 9, 189–208 (1983)MathSciNetMATHCrossRefGoogle Scholar
  33. 33.
    G.J. Gray, D.J. Murray-Smith, Y. Li, K.C. Sharman, in Proceedings of the 2nd International Conference on Genetic Algorithms in Engineering Systems: Innovations and Applications. Nonlinear model structure identification using genetic programming (Inst of Electrical Engineers, London, 1997), pp. 308–313Google Scholar
  34. 34.
    G.J. Gray, D.J. Murray-Smith, Y. Li, K.C. Sharman, T. Weinbrenner, Nonlinear model structure identification using genetic programming. Control Eng. Pract. 6, 1341–1352 (1998)CrossRefGoogle Scholar
  35. 35.
    M.S. Grewal, K. Glover, Identifiability of linear and nonlinear dynamical systems. IEEE Trans. Automat. Control 21(6), 833–837 (1976)MathSciNetMATHCrossRefGoogle Scholar
  36. 36.
    S. Hengl, C. Kreutz, J. Timmer, T. Maiwald, Data-based identifiability analysis of non-linear dynamical models. Bioinformatics 23(19), 2612–2618 (2007)CrossRefGoogle Scholar
  37. 37.
    A.L. Hodgkin, A.F. Huxley, A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Phys. 117, 500–544 (1952)Google Scholar
  38. 38.
    X. Hong, R.J. Mitchell, S. Chen, C.J. Harris, K. Li, G.W. Irwin, Model selection approaches for non-linear system identification: a review. Int. J. Syst. Sci. 39(10), 925–946 (2008)MathSciNetMATHCrossRefGoogle Scholar
  39. 39.
    H. Iba, H. de Garis, T. Sato, in Proceedings of the PPSN III, ed. by Y. Davidor, H.-P. Schwefel, R. Männer. Genetic programming with local hill-climbing (Springer, London, 1994), pp. 302–311Google Scholar
  40. 40.
    H. Iba, Inference of differential equation models by genetic programming. Inf. Sci. 178(23), 4453–4468 (2008)CrossRefGoogle Scholar
  41. 41.
    S.J. Julier, J.K. Uhlmann, in Proceedings of the International Symposium on Aerospace/Defense Sensing, Simulation, and Controls. A new extension of the Kalman filter to nonlinear systems (1997), pp. 182–193Google Scholar
  42. 42.
    M.A. Kaboudan, Genetic programming prediction of stock prices. Comput. Econ. 16(3), 207–236 (2000)MATHCrossRefGoogle Scholar
  43. 43.
    M.B. Kennel, R. Brown, H.D. Abarbanel, Determining embedding dimension for phase-space reconstruction using a geometrical construction. Phys. Rev. A 45(6), 3403–3411 (1992)CrossRefGoogle Scholar
  44. 44.
    R.D. King, K.E. Whelan, F.M. Jones, P.G.K. Reiser, C.H. Bryant, S.H. Muggleton, D.B. Kell, S.G. Oliver, Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427, 247–252 (2004)CrossRefGoogle Scholar
  45. 45.
    L.D. Kissinger, N.L. Stemm, Determination of the antileukemia agents cytarabine and azacitidine and their respective degradation products by high-performance liquid chromatography. J. Chromatogr. 353, 309–318 (1986)CrossRefGoogle Scholar
  46. 46.
    J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, Cambridge, 1992)MATHGoogle Scholar
  47. 47.
    P. Langley, Data-driven discovery of physical laws. Cognitive Sci. 5, 31–54 (1981)CrossRefGoogle Scholar
  48. 48.
    P. Langley, H.A. Simon, G.L. Bradshaw, J.M. Zytkow, Scientific Discovery: Computational Explorations of the Creative Processes (MIT Press, Cambridge, 1987)Google Scholar
  49. 49.
    P. Langley, J. Sanchez, L. Todorovski, S. Džeroski, in Proceedings of the 19th International Conference on Machine Learning. Inducing process models (Morgan Kaufmann, Sydney, 2002), pp. 347–354Google Scholar
  50. 50.
    L. Ljung, System Identification—Theory For the User, 2nd edn. (Prentice Hall, Upper Saddle River, NJ, 1999)Google Scholar
  51. 51.
    E.N. Lorenz, Deterministic non-periodic flow. J. Atmos. Sci. 20(2), 130–141 (1963)MathSciNetCrossRefGoogle Scholar
  52. 52.
    G. Mamani, J. Becedas, V.F. Batlle, H. Sira-Ramírez, Algebraic observer to estimate unmeasured state variables of DC motors. Eng. Lett. 16(2), 248–255 (2008)Google Scholar
  53. 53.
    D.J. Montana, Strongly typed genetic programming. Evol. Comput. 3(2), 199–230 (1995)CrossRefGoogle Scholar
  54. 54.
    B.S. Mulloy, R.L. Riolo, R.S. Savit, in Proceedings of the GECCO 1996. Dynamics of genetic programming and chaotic time series prediction (MIT Press, Cambridge, MA, 1996), pp. 166–174Google Scholar
  55. 55.
    C. Neely, P. Weller, R. Dittmara, Is technical analysis in the foreign exchange market profitable? A genetic programming approach. J. Financ. Quant. Anal. 32, 405–426 (1997)CrossRefGoogle Scholar
  56. 56.
    H. Oakley, Two scientific applications of genetic programming: stack filters and non-linear equation fitting to chaotic data, in Advances Genetic Programming, ed. by K.E. Kinnear Jr (MIT Press, Cambridge, MA, 1994), pp. 369–389Google Scholar
  57. 57.
    N.H. Packard, J.P. Crutchfield, J.D. Farmer, R.S. Shaw, Geometry from a time series. Phys. Rev. Lett. 45(9), 712–716 (1980)CrossRefGoogle Scholar
  58. 58.
    S.N. Pnevmatikos (ed.), Singularities & Dynamical Systems (Elsevier, Amsterdam, 1985)MATHGoogle Scholar
  59. 59.
    W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes: The Art of Scientific Computing, 3rd edn. (Cambridge University Press, Cambridge, 2007)MATHGoogle Scholar
  60. 60.
    L. Qian, H. Wang, E.R. Dougherty, Inference of noisy nonlinear differential equation models for gene regulatory networks using genetic programming and Kalman filtering. IEEE Trans. Signal Process. 56(7), 3327–3339 (2008)MathSciNetCrossRefGoogle Scholar
  61. 61.
    A. Raue, V. Becker, U. Klingmüller, J. Timmer, Identifiability and observability analysis for experimental design in nonlinear dynamical models. Chaos 20, 045105 (2010)CrossRefGoogle Scholar
  62. 62.
    K. Rodriguez-Vázquez, P.J. Fleming, Evolution of mathematical models of chaotic systems based on multiobjective genetic programming. Knowl. Inf. Syst. 8(2), 235–256 (2005)CrossRefGoogle Scholar
  63. 63.
    K. Routray, G. Deo, Kinetic parameter estimation for a multiresponse nonlinear reaction model. AIChE J. 51(6), 1733–1746 (2005)CrossRefGoogle Scholar
  64. 64.
    S. Russell, P. Norvig, Artificial Intelligence: A Modern Approach, 3rd edn. (Prentice Hall, Upper Saddle River, NJ, 2009)Google Scholar
  65. 65.
    C. Ryan, J.J. Collins, M. O’Neill, in Proceedings of the EuroGP 1998, ed. by W. Banzhaf, R. Poli, M. Schoenauer, T.C. Fogarty. Grammatical evolution: evolving programs for an arbitrary language (Springer, London, 1998), pp. 83–96Google Scholar
  66. 66.
    E. Sakamoto, H. Iba, in Proceedings of the CEC 2001. Inferring a system of differential equations for a gene regulatory network by using genetic programming (2001), pp. 720–726Google Scholar
  67. 67.
    M.D. Schmidt, H. Lipson, Coevolution of fitness predictors. IEEE Trans. Evol. Comput. 12(6), 736–749 (2008)CrossRefGoogle Scholar
  68. 68.
    M.D. Schmidt, H. Lipson, Distilling free-form natural laws from experimental data. Science 324(5923), 81–85 (2009)CrossRefGoogle Scholar
  69. 69.
    M.D. Schmidt, H. Lipson, in Proceedings of the GECCO 2010. Age-fitness Pareto optimization (2010), pp. 543–544Google Scholar
  70. 70.
    M.D. Schmidt, H. Lipson, Age-fitness Pareto optimization. Genet. Program. Theory Pract. 8, 129–146 (2010)Google Scholar
  71. 71.
    M. Schwabacher, P. Langley, in Proceedings of the ICML 2001, ed. by C.E. Brodley, A.P. Danyluk. Discovering communicable scientific knowledge from spatio-temporal data (Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001), pp. 489–496Google Scholar
  72. 72.
    K.V. Sharp, R.J. Adrian, Transition from laminar to turbulent flow in liquid filled microtubes. Exp. Fluids 36(5), 741–747 (2004)CrossRefGoogle Scholar
  73. 73.
    J. Sjöberg, Q. Zhang, L. Ljung, A. Benveniste, B. Delyon, P.-Y. Glorennec, H. Hjalmarsson, A. Juditsky, Nonlinear black-box modeling in system identification: a unified overview. Automatica 31(12), 1691–1724 (1995)MathSciNetMATHCrossRefGoogle Scholar
  74. 74.
    W.E. Stewart, M. Caracotsios, J.P. Sørensen, Parameter estimation from multiresponse data. AIChE J. 38(5), 641–650 (1992)CrossRefGoogle Scholar
  75. 75.
    S.H. Strogatz, Exploring complex networks. Nature 410, 268–276 (2001)CrossRefGoogle Scholar
  76. 76.
    A. Szalay, J. Gray, 2020 Computing: science in an exponential world. Nature 440, 413–414 (2006)CrossRefGoogle Scholar
  77. 77.
    F. Takens, Detecting strange attractors in turbulence. Lect. Notes Math. 898, 366–381 (1981)MathSciNetCrossRefGoogle Scholar
  78. 78.
    L. Todorovski, S. Džeroski, in Proceedings of the ICML 1997. Declarative bias in equation discovery (Kaufmann, Nashville, 1997), pp 376–384Google Scholar
  79. 79.
    S. Vajda, H. Rabitz, Identifiability and distinguishability of general reaction systems. J. Phys. Chem. 98, 5265–5271 (1994)CrossRefGoogle Scholar
  80. 80.
    H.U. Voss, J. Timmer, J. Kurths, Nonlinear dynamical system identification from uncertain and indirect measurements. Int. J. Bifurcat. Chaos 14(6), 1905–1933 (2004)MathSciNetMATHCrossRefGoogle Scholar
  81. 81.
    É. Walter, L. Pronzato, Identification of Parametric Models From Experimental Data (Springer, New York, 1997)MATHGoogle Scholar
  82. 82.
    D. Waltz, B.G. Buchanan, Automating science. Science 324(5923), 43–44 (2009)CrossRefGoogle Scholar
  83. 83.
    T. Washio, H. Motoda, Y. Niwa, in Proceedings of the ICML 2000. Enhancing the plausibility of law equation discovery (2000), pp. 1127–1134Google Scholar
  84. 84.
    H.-L. Wei, S.A. Billings, Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information. Int. J. Model. Identif. Control 3(4), 341–356 (2008)CrossRefGoogle Scholar
  85. 85.
    P.A. Whigham, in Proceedings of the Workshop on Genetic Programming: From Theory to Real-World Applications, ed. by J.P. Rosca. Grammatically-based genetic programming (Tahoe City, California, 1995), pp. 33–41Google Scholar
  86. 86.
    S. Winkler, M. Affenzeller, S. Wagner, New methods for the identification of nonlinear model structures based upon genetic programming techniques. J. Syst. Sci. 31, 5–14 (2005)Google Scholar
  87. 87.
    T. Yoshida, L.E. Jones, S.P. Ellner, G.F. Fussmann, N.G. Hairston Jr, Rapid evolution drives ecological dynamics in a predator-prey system. Nature 424, 303–306 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.Biological Statistics and Computational BiologyCornell UniversityIthacaUSA
  2. 2.Mechanical and Aerospace EngineeringCornell UniversityIthacaUSA

Personalised recommendations