Mobile Health pp 361-387 | Cite as

Learning Continuous-Time Hidden Markov Models for Event Data

  • Yu-Ying Liu
  • Alexander MorenoEmail author
  • Shuang Li
  • Fuxin Li
  • Le Song
  • James M. Rehg


The Continuous-Time Hidden Markov Model (CT-HMM) is an attractive modeling tool for mHealth data that takes the form of events occurring at irregularly-distributed continuous time points. However, the lack of an efficient parameter learning algorithm for CT-HMM has prevented its widespread use, necessitating the use of very small models or unrealistic constraints on the state transitions. In this paper, we describe recent advances in the development of efficient EM-based learning methods for CT-HMM models. We first review the structure of the learning problem, demonstrating that it consists of two challenges: (1) the estimation of posterior state probabilities and (2) the computation of end-state conditioned expectations. The first challenge can be addressed by reformulating the estimation problem in terms of an equivalent discrete time-inhomogeneous hidden Markov model. The second challenge is addressed by exploiting computational methods traditionally used for continuous-time Markov chains and adapting them to the CT-HMM domain. We describe three computational approaches and analyze the tradeoffs between them. We evaluate the resulting parameter learning methods in simulation and demonstrate the use of models with more than 100 states to analyze disease progression using glaucoma and Alzheimer’s Disease datasets.



Portions of this work were supported in part by NIH R01 EY13178-15 and by grant U54EB020404 awarded by the National Institute of Biomedical Imaging and Bioengineering through funds provided by the Big Data to Knowledge (BD2K) initiative ( The research was also supported in part by NSF/NIH BIGDATA 1R01GM108341, ONR N00014-15-1-2340, NSF IIS-1218749, NSF CAREER IIS-1350983, and funding from the Georgia Tech Executive Vice President of Research Office and the Center for Computational Health.


  1. 1.
    Al-Mohy, A.H., Higham, N.J.: Computing the Fréchet derivative of the matrix exponential, with an application to condition number estimation. SIAM Journal on Matrix Analysis and Applications 30(4), 1639–1657 (2009)CrossRefzbMATHGoogle Scholar
  2. 2.
    Al-Mohy, A.H., Higham, N.J.: Computing the action of the matrix exponential, with an application to exponential integrators. SIAM Journal on Scientific Computing 33(2), 488–511 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bartolomeo, N., Trerotoli, P., Serio, G.: Progression of liver cirrhosis to HCC: An application of hidden Markov model. BMC Medical Research Methodology 11(38) (2011)Google Scholar
  4. 4.
    Bauer, F.L., Fike, C.T.: Norms and exclusion theorems. Numerische Mathematik 2(1), 137–141 (1960)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bindel, D., Goodman, J.: Principles of scientific computing (2009)Google Scholar
  6. 6.
    Bladt, M., Sørensen, M.: Statistical inference for discretely observed Markov jump processes. J. R. Statist. Soc. B 39(3), 395–410 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Cox, D.R., Miller, H.D.: The Theory of Stochastic Processes. Chapman and Hall, London (1965)zbMATHGoogle Scholar
  8. 8.
    Fagan, A.M., Head, D., Shah, A.R., et. al: Decreased CSF A beta 42 correlates with brain atrophy in cognitively normal elderly. Ann Neurol. 65(2), 176–183 (2009)Google Scholar
  9. 9.
    Golub, G.H., Van Loan, C.F.: Matrix computations, vol. 3. JHU Press (2012)Google Scholar
  10. 10.
    Higham, N.: Functions of Matrices: Theory and Computation. SIAM Press (2008)Google Scholar
  11. 11.
    Hobolth, A., Jensen, J.L.: Statistical inference in evolutionary models of DNA sequences via the EM algorithm. Statistical Applications in Genetics and Molecular Biology 4(1) (2005)Google Scholar
  12. 12.
    Hobolth, A., Jensen, J.L.: Summary statistics for endpoint-conditioned continuous-time Markov chains. Journal of Applied Probability 48(4), 911–924 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Jackson, C.H.: Multi-state models for panel data: The MSM package for R. Journal of Statistical Software 38(8) (2011)Google Scholar
  14. 14.
    Jensen, A.: Markoff chains as an aid in the study of Markoff processes. Skand. Aktuarietidskr 36, 87–91 (1953)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Kingman, S.: Glaucoma is second leading cause of blindness globally. Bulletin of the World Health Organization 82(11) (2004)Google Scholar
  16. 16.
    Leiva-Murillo, J.M., Rodrguez, A.A., Baca-Garca, E.: Visualization and prediction of disease interactions with continuous-time hidden Markov models. In: Advances in Neural Information Processing Systems (2011)Google Scholar
  17. 17.
    Liu, Y., Ishikawa, H., Chen, M., et al.: Longitudinal modeling of glaucoma progression using 2-dimensional continuous-time hidden Markov model. In: Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 444–451 (2013)Google Scholar
  18. 18.
    Liu, Y.Y., Li, S., Li, F., Song, L., Rehg, J.M.: Efficient learning of continuous-time hidden Markov models for disease progression. In: Proc. Twenty-Ninth Annual Conference on Neural Information Processing Systems (NIPS 15). Montreal, Canada (2015)Google Scholar
  19. 19.
    McGibbon, R.T., Pande, V.S.: Efficient maximum likelihood parameterization of continuous-time Markov processes. The Journal of Chemical Physics 143(3), 034,109 (2015)CrossRefGoogle Scholar
  20. 20.
    Metzner, P., Horenko, I., Schütte, C.: Generator estimation of Markov jump processes. Journal of Computational Physics 227, 353–375 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Metzner, P., Horenko, I., Schütte, C.: Generator estimation of Markov jump processes based on incomplete observations nonequidistant in time. Physical Review E 76(066702) (2007)Google Scholar
  22. 22.
    Moler, C., Van Loan, C.: Nineteen dubious ways to compute the exponential of a matrix. SIAM Review 20(4), 801–836 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Moler, C., Van Loan, C.: Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM Review 45(1), 3–49 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Nodelman, U., Shelton, C.R., Koller, D.: Expectation maximization and complex duration distributions for continuous time Bayesian networks. In: Proc. Uncertainty in AI (UAI 05) (2005)Google Scholar
  25. 25.
    Osborne, E.: On pre-conditioning of matrices. Journal of the ACM (JACM) 7(4), 338–345 (1960)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Rabinar, L.R.: A tutorial on Hidden Markov Models and selected applications in speech recognition. Proceedings of the IEEE 77(2) (1989)Google Scholar
  27. 27.
    Ross, S.M.: Stochastic Processes. John Wiley, New York (1983)zbMATHGoogle Scholar
  28. 28.
    Tataru, P., Hobolth, A.: Comparison of methods for calculating conditional expectations of sufficient statistics for continuous time Markov chains. BMC Bioinformatics 12(465) (2011)Google Scholar
  29. 29.
    The Alzheimer’s Disease Neuroimaging Initiative:
  30. 30.
    Van Loan, C.: Computing integrals involving the matrix exponential. IEEE Trans. Automatic Control 23, 395–404 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Wang, X., Sontag, D., Wang, F.: Unsupervised learning of disease progression models. Proceeding KDD 4(1), 85–94 (2014)Google Scholar
  32. 32.
    Wollstein, G., Kagemann, L., Bilonick, R., et al.: Retinal nerve fibre layer and visual function loss in glaucoma: the tipping point. Br J Ophthalmol 96(1), 47–52 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yu-Ying Liu
    • 1
  • Alexander Moreno
    • 1
    Email author
  • Shuang Li
    • 1
  • Fuxin Li
    • 2
  • Le Song
    • 1
  • James M. Rehg
    • 1
  1. 1.Georgia Institute of TechnologyAtlantaUSA
  2. 2.2077 Kelley Engineering CenterOregon State UniversityCorvallisUSA

Personalised recommendations