Inference of hidden variables in systems of differential equations with genetic programming
 Theodore W. Cornforth,
 Hod Lipson
 … show all 2 hide
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Get AccessAbstract
The datadriven modeling of dynamical systems is an important scientific activity, and many studies have applied genetic programming (GP) to the task of automatically constructing such models in the form of systems of ordinary differential equations (ODEs). These previous studies assumed that data measurements were available for all variables in the system, whereas in realworld settings, it is typically the case that one or more variables are unmeasured or “hidden.” Here, we investigate the prospect of automatically constructing ODE models of dynamical systems from time series data with GP in the presence of hidden variables. Several examples with both synthetic and physical systems demonstrate the unique challenges of this problem and the circumstances under which it is possible to reverseengineer both the form and parameters of ODE models with hidden variables.
Inside
Within this Article
 Introduction
 Related work
 Methods
 Results
 Discussion
 References
 References
Other actions
 F. Adachi, T. Washio, H. Motoda, Scientific discovery of dynamic models based on scaletype constraints. IPSJ Digit. Cour. 2, 607–619 (2006) CrossRef
 L.A. Aguirre, U.S. Freitas, C. Letellier, J. Maquet, Structureselection techniques applied to continuoustime nonlinear models. Physica D 158, 1–18 (2001) CrossRef
 L.A. Aguirre, C. Letellier, Modeling nonlinear dynamics and chaos: a review. Math. Probl. Eng. (2009). doi:10.1155/2009/238960
 D.P. Ahalpara, A. Sen, in Proceedings of the EuroGP 2011, ed. by S. Silva, J.A. Foster, M. Nicolau, P. Machado, M. Giacobini. A sniffer technique for an efficient deduction of model dynamical equations using genetic programming (Springer, Berlin, 2011), pp. 1–12
 S. Ando, E. Sakamoto, H. Iba, Evolutionary modeling and inference of gene network. Inf. Sci. 145(3–4), 237–259 (2002) CrossRef
 H. Andrew, in Proceedings of the 2nd International Conference on Adaptive Computing in Engineering Design and Control. System identification using genetic programming (University of Plymouth, Plymouth, 1996), pp. 57–62
 P.J. Angeline, D.B. Fogel, in Proceedings of the SPIE. An evolutionary program for the identification of dynamical systems (1997), pp. 409–417
 B. Argemí, J. Saurina, Study of the degradation of 5azacytidine as a model of unstable drugs using a stoppedflow method and further data analysis with multivariate curve resolution. Talanta 74, 176–182 (2007) CrossRef
 K.J. Åström, P. Eykhoff, System identification—a survey. Automatica 7(2), 123–162 (1971) CrossRef
 E. Baake, M. Baake, H.G. Bock, K.M. Briggs, Fitting ordinary differential equations to chaotic data. Phys. Rev. A 45(8), 5524–5529 (1992) CrossRef
 V. Babovic, M. Keijzer, in Proceedings of the 4th International Conference on Hydroinformatics. Evolutionary algorithms approach to induction of differential equations (Iowa City, USA, 2000), pp. 251–258
 R. Bakker, J.C. Schouten, C.L. Giles, F. Takens, C.M. van den Bleek, Learning chaotic attractors. Neural Comput. 12(10), 2355–2383 (2000) CrossRef
 W. Banzhaf, P. Nordin, R.E. Keller, F.D. Francone, Genetic Programming: An Introduction on the Automatic Evolution of Computer Programs and Its Applications (Morgan Kaufmann, San Francisco, 1997)
 J.A. Beisler, Isolation, characterization, and properties of a labile hydrolysis product of the antitumor nucleoside, 5azacytidine. J. Med. Chem. 21(2), 204–208 (1978) CrossRef
 H.S. Bernardino, H.J.C. Barbosa, Inferring systems of ordinary differential equations via grammarbased immune programming. LNCS 6825, 198–211 (2011)
 C.M. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006)
 S. Boccaletti, The Synchronized Dynamics of Complex Systems (Elsevier, Amsterdam, 2008)
 J. Bongard, H. Lipson, Automated reverse engineering of nonlinear dynamical systems. PNAS 104(24), 9943–9948 (2007) CrossRef
 E. Bradley, M. Easley, R. Stolle, Reasoning about nonlinear system identification. Artif. Intell. 133, 139–188 (2001) CrossRef
 M. Brameier, W. Banzhaf, Linear Genetic Programming (Springer, New York, 2007)
 J.L. Breeden, A. Hübler, Reconstructing equations of motion from experimental data with unobserved variables. Phys. Rev. A 42(10), 5817–5826 (1990) CrossRef
 W. Bridewell, P. Langley, L. Todorovski, S. Dzeroski, Inductive process modeling. Mach. Learn. 71, 1–32 (2008) CrossRef
 H. Cao, L. Kang, Y. Chen, J. Yu, Evolutionary modeling of systems of ordinary differential equations with genetic programming. Genet. Program Evolvable Mach. 1(4), 309–337 (2000) CrossRef
 O.T. Chis, J.R. Banga, E. BalsaCanto, Structural identifiability of systems biology models: a critical comparison of methods. PLoS ONE 6(11), e27755 (2011) CrossRef
 K.H. Cho, S.Y. Shin, H.W. Kim, O. Wolkenhauer, B. McFerran, W. Kolch, Mathematical modeling of the influence of RKIP on the ERK signaling pathway. LNCS 2602, 127–141 (2003)
 D. Clery, D. Voss, All for one and one for all. Science 308, 809 (2005) CrossRef
 J. Cremers, A. Hübler, Construction of differential equations from experimental data. Zeitschrift für Naturforschung A 42(8), 797–802 (1987)
 J.P. Crutchfield, B.S. McNamara, Equations of motion from a data series. Complex Syst. 1(3), 417–452 (1987)
 S. Džeroski, L. Todorovski, in Proceedings of the 10th International Conference on Machine Learning. Discovering dynamics (Kaufmann, Amherst, 1993), pp. 97–103
 J. Evans, A. Rzhetsky, Machine science. Science 329, 3994400 (2010) CrossRef
 G. Gouesbet, Reconstruction of the vector fields of continuous dynamical systems from numerical scalar time series. Phys. Rev. A 43(10), 5321–5331 (1991) CrossRef
 P. Grassberger, I. Procaccia, Measuring the strangeness of strange attractors. Physica D 9, 189–208 (1983) CrossRef
 G.J. Gray, D.J. MurraySmith, Y. Li, K.C. Sharman, in Proceedings of the 2nd International Conference on Genetic Algorithms in Engineering Systems: Innovations and Applications. Nonlinear model structure identification using genetic programming (Inst of Electrical Engineers, London, 1997), pp. 308–313
 G.J. Gray, D.J. MurraySmith, Y. Li, K.C. Sharman, T. Weinbrenner, Nonlinear model structure identification using genetic programming. Control Eng. Pract. 6, 1341–1352 (1998) CrossRef
 M.S. Grewal, K. Glover, Identifiability of linear and nonlinear dynamical systems. IEEE Trans. Automat. Control 21(6), 833–837 (1976) CrossRef
 S. Hengl, C. Kreutz, J. Timmer, T. Maiwald, Databased identifiability analysis of nonlinear dynamical models. Bioinformatics 23(19), 2612–2618 (2007) CrossRef
 A.L. Hodgkin, A.F. Huxley, A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Phys. 117, 500–544 (1952)
 X. Hong, R.J. Mitchell, S. Chen, C.J. Harris, K. Li, G.W. Irwin, Model selection approaches for nonlinear system identification: a review. Int. J. Syst. Sci. 39(10), 925–946 (2008) CrossRef
 H. Iba, H. de Garis, T. Sato, in Proceedings of the PPSN III, ed. by Y. Davidor, H.P. Schwefel, R. Männer. Genetic programming with local hillclimbing (Springer, London, 1994), pp. 302–311
 H. Iba, Inference of differential equation models by genetic programming. Inf. Sci. 178(23), 4453–4468 (2008) CrossRef
 S.J. Julier, J.K. Uhlmann, in Proceedings of the International Symposium on Aerospace/Defense Sensing, Simulation, and Controls. A new extension of the Kalman filter to nonlinear systems (1997), pp. 182–193
 M.A. Kaboudan, Genetic programming prediction of stock prices. Comput. Econ. 16(3), 207–236 (2000) CrossRef
 M.B. Kennel, R. Brown, H.D. Abarbanel, Determining embedding dimension for phasespace reconstruction using a geometrical construction. Phys. Rev. A 45(6), 3403–3411 (1992) CrossRef
 R.D. King, K.E. Whelan, F.M. Jones, P.G.K. Reiser, C.H. Bryant, S.H. Muggleton, D.B. Kell, S.G. Oliver, Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427, 247–252 (2004) CrossRef
 L.D. Kissinger, N.L. Stemm, Determination of the antileukemia agents cytarabine and azacitidine and their respective degradation products by highperformance liquid chromatography. J. Chromatogr. 353, 309–318 (1986) CrossRef
 J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, Cambridge, 1992)
 P. Langley, Datadriven discovery of physical laws. Cognitive Sci. 5, 31–54 (1981) CrossRef
 P. Langley, H.A. Simon, G.L. Bradshaw, J.M. Zytkow, Scientific Discovery: Computational Explorations of the Creative Processes (MIT Press, Cambridge, 1987)
 P. Langley, J. Sanchez, L. Todorovski, S. Džeroski, in Proceedings of the 19th International Conference on Machine Learning. Inducing process models (Morgan Kaufmann, Sydney, 2002), pp. 347–354
 L. Ljung, System Identification—Theory For the User, 2nd edn. (Prentice Hall, Upper Saddle River, NJ, 1999)
 E.N. Lorenz, Deterministic nonperiodic flow. J. Atmos. Sci. 20(2), 130–141 (1963) CrossRef
 G. Mamani, J. Becedas, V.F. Batlle, H. SiraRamírez, Algebraic observer to estimate unmeasured state variables of DC motors. Eng. Lett. 16(2), 248–255 (2008)
 D.J. Montana, Strongly typed genetic programming. Evol. Comput. 3(2), 199–230 (1995) CrossRef
 B.S. Mulloy, R.L. Riolo, R.S. Savit, in Proceedings of the GECCO 1996. Dynamics of genetic programming and chaotic time series prediction (MIT Press, Cambridge, MA, 1996), pp. 166–174
 C. Neely, P. Weller, R. Dittmara, Is technical analysis in the foreign exchange market profitable? A genetic programming approach. J. Financ. Quant. Anal. 32, 405–426 (1997) CrossRef
 H. Oakley, Two scientific applications of genetic programming: stack filters and nonlinear equation fitting to chaotic data, in Advances Genetic Programming, ed. by K.E. Kinnear Jr (MIT Press, Cambridge, MA, 1994), pp. 369–389
 N.H. Packard, J.P. Crutchfield, J.D. Farmer, R.S. Shaw, Geometry from a time series. Phys. Rev. Lett. 45(9), 712–716 (1980) CrossRef
 S.N. Pnevmatikos (ed.), Singularities & Dynamical Systems (Elsevier, Amsterdam, 1985)
 W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes: The Art of Scientific Computing, 3rd edn. (Cambridge University Press, Cambridge, 2007)
 L. Qian, H. Wang, E.R. Dougherty, Inference of noisy nonlinear differential equation models for gene regulatory networks using genetic programming and Kalman filtering. IEEE Trans. Signal Process. 56(7), 3327–3339 (2008) CrossRef
 A. Raue, V. Becker, U. Klingmüller, J. Timmer, Identifiability and observability analysis for experimental design in nonlinear dynamical models. Chaos 20, 045105 (2010) CrossRef
 K. RodriguezVázquez, P.J. Fleming, Evolution of mathematical models of chaotic systems based on multiobjective genetic programming. Knowl. Inf. Syst. 8(2), 235–256 (2005) CrossRef
 K. Routray, G. Deo, Kinetic parameter estimation for a multiresponse nonlinear reaction model. AIChE J. 51(6), 1733–1746 (2005) CrossRef
 S. Russell, P. Norvig, Artificial Intelligence: A Modern Approach, 3rd edn. (Prentice Hall, Upper Saddle River, NJ, 2009)
 C. Ryan, J.J. Collins, M. O’Neill, in Proceedings of the EuroGP 1998, ed. by W. Banzhaf, R. Poli, M. Schoenauer, T.C. Fogarty. Grammatical evolution: evolving programs for an arbitrary language (Springer, London, 1998), pp. 83–96
 E. Sakamoto, H. Iba, in Proceedings of the CEC 2001. Inferring a system of differential equations for a gene regulatory network by using genetic programming (2001), pp. 720–726
 M.D. Schmidt, H. Lipson, Coevolution of fitness predictors. IEEE Trans. Evol. Comput. 12(6), 736–749 (2008) CrossRef
 M.D. Schmidt, H. Lipson, Distilling freeform natural laws from experimental data. Science 324(5923), 81–85 (2009) CrossRef
 M.D. Schmidt, H. Lipson, in Proceedings of the GECCO 2010. Agefitness Pareto optimization (2010), pp. 543–544
 M.D. Schmidt, H. Lipson, Agefitness Pareto optimization. Genet. Program. Theory Pract. 8, 129–146 (2010)
 M. Schwabacher, P. Langley, in Proceedings of the ICML 2001, ed. by C.E. Brodley, A.P. Danyluk. Discovering communicable scientific knowledge from spatiotemporal data (Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001), pp. 489–496
 K.V. Sharp, R.J. Adrian, Transition from laminar to turbulent flow in liquid filled microtubes. Exp. Fluids 36(5), 741–747 (2004) CrossRef
 J. Sjöberg, Q. Zhang, L. Ljung, A. Benveniste, B. Delyon, P.Y. Glorennec, H. Hjalmarsson, A. Juditsky, Nonlinear blackbox modeling in system identification: a unified overview. Automatica 31(12), 1691–1724 (1995) CrossRef
 W.E. Stewart, M. Caracotsios, J.P. Sørensen, Parameter estimation from multiresponse data. AIChE J. 38(5), 641–650 (1992) CrossRef
 S.H. Strogatz, Exploring complex networks. Nature 410, 268–276 (2001) CrossRef
 A. Szalay, J. Gray, 2020 Computing: science in an exponential world. Nature 440, 413–414 (2006) CrossRef
 F. Takens, Detecting strange attractors in turbulence. Lect. Notes Math. 898, 366–381 (1981) CrossRef
 L. Todorovski, S. Džeroski, in Proceedings of the ICML 1997. Declarative bias in equation discovery (Kaufmann, Nashville, 1997), pp 376–384
 S. Vajda, H. Rabitz, Identifiability and distinguishability of general reaction systems. J. Phys. Chem. 98, 5265–5271 (1994) CrossRef
 H.U. Voss, J. Timmer, J. Kurths, Nonlinear dynamical system identification from uncertain and indirect measurements. Int. J. Bifurcat. Chaos 14(6), 1905–1933 (2004) CrossRef
 É. Walter, L. Pronzato, Identification of Parametric Models From Experimental Data (Springer, New York, 1997)
 D. Waltz, B.G. Buchanan, Automating science. Science 324(5923), 43–44 (2009) CrossRef
 T. Washio, H. Motoda, Y. Niwa, in Proceedings of the ICML 2000. Enhancing the plausibility of law equation discovery (2000), pp. 1127–1134
 H.L. Wei, S.A. Billings, Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information. Int. J. Model. Identif. Control 3(4), 341–356 (2008) CrossRef
 P.A. Whigham, in Proceedings of the Workshop on Genetic Programming: From Theory to RealWorld Applications, ed. by J.P. Rosca. Grammaticallybased genetic programming (Tahoe City, California, 1995), pp. 33–41
 S. Winkler, M. Affenzeller, S. Wagner, New methods for the identification of nonlinear model structures based upon genetic programming techniques. J. Syst. Sci. 31, 5–14 (2005)
 T. Yoshida, L.E. Jones, S.P. Ellner, G.F. Fussmann, N.G. Hairston Jr, Rapid evolution drives ecological dynamics in a predatorprey system. Nature 424, 303–306 (2003) CrossRef
 Title
 Inference of hidden variables in systems of differential equations with genetic programming
 Journal

Genetic Programming and Evolvable Machines
Volume 14, Issue 2 , pp 155190
 Cover Date
 20130601
 DOI
 10.1007/s1071001291754
 Print ISSN
 13892576
 Online ISSN
 15737632
 Publisher
 Springer US
 Additional Links
 Topics
 Keywords

 Genetic programming
 Ordinary differential equations
 Hidden variables
 Modeling
 Symbolic identification
 Authors

 Theodore W. Cornforth ^{(1)}
 Hod Lipson ^{(2)}
 Author Affiliations

 1. Biological Statistics and Computational Biology, Cornell University, 1198 Comstock Hall, Ithaca, NY, 148532601, USA
 2. Mechanical and Aerospace Engineering, Cornell University, 239 Upson Hall, Ithaca, NY, 148537501, USA