Skip to main content
SpringerLink
Log in

Optimal control of molecular dynamics using Markov state models

  • Full Length Paper
  • Series B
  • Published:
Mathematical Programming Submit manuscript

Abstract

A numerical scheme for solving high-dimensional stochastic control problems on an infinite time horizon that appear relevant in the context of molecular dynamics is outlined. The scheme rests on the interpretation of the corresponding Hamilton–Jacobi–Bellman equation as a nonlinear eigenvalue problem that, using a logarithmic transformation, can be recast as a linear eigenvalue problem, for which the principal eigenvalue and its eigenfunction are sought. The latter can be computed efficiently by approximating the underlying stochastic process with a coarse-grained Markov state model for the dominant metastable sets. We illustrate our method with two numerical examples, one of which involves the task of maximizing the population of α-helices in an ensemble of small biomolecules (alanine dipeptide), and discuss the relation to the large deviation principle of Donsker and Varadhan.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fleming W.H., Soner H.M.: Controlled Markov Processes and Viscosity Solutions. Springer, New York (2006)

    MATH  Google Scholar 

  2. Kushner H.J., Dupuis P.G.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, New York (2001)

    MATH  Google Scholar 

  3. Barles G., Souganidis P.: Convergence of approximation schemes for fully nonlinear equations. Asymptot. Anal. 4, 271–283 (1991)

    MathSciNet  MATH  Google Scholar 

  4. Barles G., Jakobsen E.: Error bounds for monotone approximation schemes for parabolic Hamilton–Jacobi–Bellman equations. Math. Comput. 76, 1861–1893 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  5. Carlini E., Falcone M., Ferretti R.: An efficient algorithm for Hamilton Jacobi equations in high dimension. Comput. Vis. Sci. 7, 15–29 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  6. Kappen H.J.: Path integrals and symmetry breaking for optimal control theory. J. Stat. Mech. Theor. Exp. 2005(11), P11011 (2005)

    Article  MathSciNet  Google Scholar 

  7. Isralewitz B., Gao M., Schulten K.: Steered molecular dynamics and mechanical functions of proteins. Curr. Opin. Struct. Biol. 11, 224–230 (2001)

    Article  Google Scholar 

  8. Schlitter J., Engels M., Krüger P., Jacoby E., Wollmer A.: Targeted molecular dynamics. Mol. Simul. 10, 291–309 (1993)

    Article  Google Scholar 

  9. Rabitz H., de Vivie-Riedle R., Motzkus M., Kompa K.: Whither the future of controlling quantum phenomena?. Science 288(5467), 824–828 (2000)

    Article  Google Scholar 

  10. Kühn O., Wöste L.: Analysis and control of ultrafast photoinduced reactions. In: Kühn, O., Wöste, L.(eds) Series in Chemical Physics, vol. 87, Springer, New York (2007)

    Google Scholar 

  11. Engel A., Müller D.J.: Observing single biomolecules at work with the atomic force microscope. Nat. Struct. Mol. Biol. 7, 715–718 (2000)

    Article  Google Scholar 

  12. Fleming W.H., McEneaney W.M.: Risk-sensitive control on an infinite time horizon. SIAM J. Control Opt. 33, 1881–1915 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  13. Fleming W.H., Sheu S.J.: Asymptotics for the principal eigenvalue and eigenfunction of a nearly first-order operator with large potential. Ann. Probab. 25, 1953–1994 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  14. Freidlin M., Wentzell A.D.: Random Perturbations of Dynamical Systems. Springer, New York (1998)

    Book  MATH  Google Scholar 

  15. Jäger M., Zhang Y., Bieschke J., Nguyen H., Dendle M., Bowman M.E., Noel J.P., Gruebele M., Kelly J.W.: Structure-function-folding relationship in a WW domain. Proc. Natl. Acad. Sci. USA. 103, 10648–10653 (2006)

    Article  Google Scholar 

  16. Kobitski A.Y., Nierth A., Helm M., Jäschke A., Nienhaus G.U.: Mg2+ dependent folding of a Diels-Alderase ribozyme probed by single-molecule FRET analysis. Nucleic Acids Res. 35, 2047–2059 (2007)

    Article  Google Scholar 

  17. Fischer S., Windshuegel B., Horak D., Holmes K.C., Smith J.C.: Structural mechanism of the recovery stroke in the myosin molecular motor. Proc. Natl. Acad. Sci. USA. 102, 6873–6878 (2005)

    Article  Google Scholar 

  18. Noé F., Krachtus D., Smith J.C., Fischer S.: Transition networks for the comprehensive characterization of complex conformational change in proteins. J. Chem. Theor. Comp. 2, 840–857 (2006)

    Article  Google Scholar 

  19. Ostermann A., Waschipky R., Parak F.G., Nienhaus U.G.: Ligand binding and conformational motions in myoglobin. Nature 404, 205–208 (2000)

    Article  Google Scholar 

  20. Schaeffer D.D., Fersht A., Daggett V.: Combining experiment and simulation in protein folding: closing the gap for small model systems. Curr. Opin. Struct. Biol. 18(1), 4–9 (2008)

    Article  Google Scholar 

  21. Noé F., Schütte C., Vanden-Eijnden E., Reich L., Weikl T.R.: Constructing the full ensemble of folding pathways from short off-equilibrium simulations. Proc. Natl. Acad. Sci. USA 106, 19011–19016 (2009)

    Article  Google Scholar 

  22. van Gunsteren W., Dolenc J., Mark A.: Molecular simulation as an aid to experimentalists. Curr. Opin. Struct. Biol. 18(2), 149–153 (2008)

    Article  Google Scholar 

  23. Schütte, C.: Conformational dynamics: modelling, theory, algorithm, and applications to biomolecules. Habilitation thesis, Fachbereich Mathematik und Informatik, FU Berlin (1999)

  24. Deuflhard P., Huisinga W., Fischer A., Schütte C.: Identification of almost invariant aggregates in reversible nearly uncoupled Markov chains. Linear Algebra Appl. 315, 39–59 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  25. Schütte C., Fischer A., Huisinga W., Deuflhard P.: A direct approach to conformational dynamics based on hybrid Monte Carlo. J. Comp. Phys. (Special issue on Comput. Biophys.) 151, 146–168 (1999)

    MATH  Google Scholar 

  26. Schütte C., Huisinga W.: Biomolecular conformations can be identified as metastable sets of molecular dynamics. In: Le Bris, C.(ed) Handbook of Numerical Analysis, pp. 699–744. Elsevier, New York (2003)

    Google Scholar 

  27. Bovier A., Eckhoff M., Gayrard V., Klein M.: Metastability and low lying spectra in reversible markov chains. Commun. Math. Phys. 228, 219–255 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  28. Noé F., Horenko I., Schütte C., Smith J.: Hierarchical analysis of conformational dynamics in biomolecules: transition networks of metastable states. J. Chem. Phys. 126, 155102 (2007)

    Article  Google Scholar 

  29. Chodera J., Singhal N., Pande V.S., Dill K., Swope W.: Automatic discovery of metastable states for the construction of Markov models of macromolecular conformational dynamics. J. Chem. Phys. 126, 155101 (2007)

    Article  Google Scholar 

  30. Buchete N.V., Hummer G.: Coarse master equations for peptide folding dynamics. J. Phys. Chem. B. 112, 6057–6069 (2008)

    Article  Google Scholar 

  31. Pan A.C., Roux B.: Building Markov state models along pathways to determine free energies and rates of transitions. J. Chem. Phys. 129, 064107 (2008)

    Article  Google Scholar 

  32. Sarich M., Noé F., Schütte C.: On the approximation quality of Markov state models. Multiscale Model. Simul. 8(4), 1154–1177 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  33. Voter A.: Introduction to the kinetic Monte Carlo method. In: Sickafus, K., Kotomin, E., Uberuaga, B.(eds) Radiation Effects in Solids, Springer, NATO Publishing Unit, Dordrecht (2005)

    Google Scholar 

  34. Huisinga, W.: Metastability of markovian systems a transfer operator based approach in application to molecular dynamics. Ph.D. thesis, Fachbereich Mathematik und Informatik, FU Berlin (2001)

  35. Schütte C., Noe F., Lu J., Sarich M., Vanden-Eijnden E.: Markov state models based on milestoning. J. Chem. Phys. 134, 204105–204120 (2011)

    Article  Google Scholar 

  36. Borkar V.S., Ghosh M.K.: Ergodic control of multidimensional diffusions, II: adaptive control. Appl. Math. Opt. 21, 191–220 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  37. Bensoussan A., Nagai H.: An ergodic control problem arising from the principal eigenvalue of an elliptic operator. J. Math. Soc. Jpn. 43, 49–65 (1990)

    Article  MathSciNet  Google Scholar 

  38. Huisinga W., Meyn S.P., Schütte C.: Phase transitions & metastability in Markovian and molecular systems. Ann. Appl. Probab. 14, 419–458 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  39. James M.: Asymptotic analysis of nonlinear stochastic risk-sensitive control and differential games. Math. Control Signals Syst. 5, 401–417 (1992)

    Article  MATH  Google Scholar 

  40. Holland C.J.: A minimum principle for the principal eigenvalue for second-order linear elliptic equations with natural boundary conditions. Commun. Pure Appl. Math. 31, 509–519 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  41. Whittle P.: Risk-sensitivity, large deviations and stochastic control. Eur. J. Oper. Res. 73, 295–303 (1994)

    Article  MATH  Google Scholar 

  42. Donsker M.D., Varadhan S.S.: Asymptotic evaluation of certain Markov process expectations for large time, i. Commun. Pure Appl. Math. 28, 1–47 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  43. Ellis R.S.: Entropy, Large Deviations and Statistical Mechanics. Springer, Berlin (1985)

    Book  MATH  Google Scholar 

  44. Kurchan J.: Fluctuation theorem for stochastic dynamics. J. Phys. A. Math. Gen. 31, 3719–3729 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  45. Lebowitz J.L., Spohn H.: A Gallavotti–Cohen-type symmetry in the large deviation functional for stochastic dynamics. J. Stat. Phys. 95, 333–365 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  46. Pham H.: Continuous-time Stochastic Control and Optimization with Financial Applications. Springer, Berlin (2009)

    Book  MATH  Google Scholar 

  47. Robin M.: Long-term average cost control problems for continuous time Markov processes: a survey. Acta Appl. Math. 1, 281–299 (1983)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut für Mathematik, Freie Universität Berlin, Arnimallee 6, 14195, Berlin, Germany

    Christof Schütte, Stefanie Winkelmann & Carsten Hartmann

Authors
  1. Christof Schütte
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefanie Winkelmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carsten Hartmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christof Schütte.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schütte, C., Winkelmann, S. & Hartmann, C. Optimal control of molecular dynamics using Markov state models. Math. Program. 134, 259–282 (2012). https://doi.org/10.1007/s10107-012-0547-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-012-0547-6

Mathematics Subject Classification

Search

Navigation