Skip to main content

Hidden Markov Models with mixtures as emission distributions

Statistics and Computing volume 24, pages 493–504 (2014)Cite this article

Abstract

In unsupervised classification, Hidden Markov Models (HMM) are used to account for a neighborhood structure between observations. The emission distributions are often supposed to belong to some parametric family. In this paper, a semiparametric model where the emission distributions are a mixture of parametric distributions is proposed to get a higher flexibility. We show that the standard EM algorithm can be adapted to infer the model parameters. For the initialization step, starting from a large number of components, a hierarchical method to combine them into the hidden states is proposed. Three likelihood-based criteria to select the components to be combined are discussed. To estimate the number of hidden states, BIC-like criteria are derived. A simulation study is carried out both to determine the best combination between the combining criteria and the model selection criteria and to evaluate the accuracy of classification. The proposed method is also illustrated using a biological dataset from the model plant Arabidopsis thaliana. A R package HMMmix is freely available on the CRAN.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  • Baudry, J.P., Raftery, A.E., Celeux, G., Lo, K., Gottardo, R.: Combining mixture components for clustering. J. Comput. Graph. Stat. 9(2), 332–353 (2010). doi:10.1198/jcgs.2010.08111

    Article  MathSciNet  Google Scholar 

  • Bérard, C., Martin-Magniette, M.-L., Robin, S.: Mixture model approach to compare two samples of tiling array data: chip-chip and transcriptome. Stat. Appl. Genet. Mol. Biol. 10(1), 1–22 (2011). doi:10.2202/1544-6115.1692

    MathSciNet  Google Scholar 

  • Biernacki, C., Celeux, G., Govaert, G.: Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Trans. Pattern Anal. Mach. Intell. 22, 719–725 (2000). ISSN 0162-8828

    Article  Google Scholar 

  • Cappé, O., Moulines, E., Ryden, T.: Inference in Hidden Markov Models. Springer, Berlin (2010)

    Google Scholar 

  • Celeux, G., Durand, J.B.: Selecting hidden Markov model state number with cross-validated likelihood. In: Computational Statistics, pp. 541–564 (2008)

    Google Scholar 

  • Chatzis, S.P.: Hidden Markov models with nonelliptically contoured state densities. IEEE Trans. Pattern Anal. Mach. Intell. 32, 2297–2304 (2010). ISSN 0162-8828

    Article  Google Scholar 

  • Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. R. Stat. Soc. B 39(1), 1–38 (1977)

    MATH  MathSciNet  Google Scholar 

  • Durbin, R., Eddy, S.R., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998)

    Book  MATH  Google Scholar 

  • Fraley, C., Raftery, A.E.: Mclust: software for model-based cluster analysis. J. Classif. 16, 297–306 (1999)

    Article  MATH  Google Scholar 

  • Hennig, C.: Methods for merging Gaussian mixture components. Adv. Data Anal. Classif. 4, 3–34 (2010). doi:10.1007/s11634-010-0058-3

    Article  MathSciNet  Google Scholar 

  • Keribin, C.: Consistent estimation of the order of mixture models. Sankhya, Ser. A 62, 49–66 (2000). doi:10.2307/25051289

    MATH  MathSciNet  Google Scholar 

  • Li, J.: Clustering based on a multilayer mixture model. J. Comput. Graph. Stat. 14(3), 547–568 (2005). doi:10.1198/106186005X59586

    Article  Google Scholar 

  • Li, W., Meyer, A., Liu, X.S.: A hidden Markov model for analyzing chip-chip experiments on genome tiling arrays and its application to p53 binding sequences. Bioinformatics 211, 274–282 (2005)

    Article  Google Scholar 

  • Lippman, Z., Gendreland, A.-V., Blackand, M., Vaughn, M.W., et al.: Role of transposable elements in heterochromatin and epigenetic control. Nature 430, 471–476 (2004)

    Article  Google Scholar 

  • McLachlan, G.J., Peel, D.: In: Finite Mixture Models (2000)

    Chapter  Google Scholar 

  • Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989). doi:10.1109/5.18626

    Article  Google Scholar 

  • Roudier, F., Ahmed, I., Bérard, C., Sarazin, A., Mary-Huard, T., et al.: Integrative epigenomic mapping defines four main chromatin states in arabidopsis. EMBO J. 30, 1928–1938 (2011)

    Article  Google Scholar 

  • Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 461–464 (1978)

    Article  MATH  Google Scholar 

  • Sun, W., Cai, T.: Large-scale multiple testing under dependence. J. R. Stat. Soc. 71, 393–424 (2009)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INRA, UMR 518 MIA, 75231, Paris, France

    Stevenn Volant, Caroline Bérard, Marie-Laure Martin-Magniette & Stéphane Robin

  2. AgroParisTech, UMR MIA, 75231, Paris, France

    Stevenn Volant, Caroline Bérard, Marie-Laure Martin-Magniette & Stéphane Robin

  3. INRA, UMR1165 URGV, 91057, Evry, France

    Marie-Laure Martin-Magniette

  4. UEVE, UMR URGV, 91057, Evry, France

    Marie-Laure Martin-Magniette

  5. CNRS, ERL8196 UMR URGV, 91057, Evry, France

    Marie-Laure Martin-Magniette

Authors
  1. Stevenn Volant
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caroline Bérard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marie-Laure Martin-Magniette
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stéphane Robin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stevenn Volant.

Appendix

Appendix

A.1 Algorithm

We present in this appendix the algorithm we proposed for combining components of an HMM. This algorithm has been written with respect to the results we obtained in Sect. 4.1.2. However, this algorithm can easily be written for other criteria.

  1. 1.

    Fit an HMM with K components.

  2. 2.

    From G=K,K−1,…,1

    • Select the clusters k and l to be combined as:

    • Update the parameters with a few steps of the EM algorithm to get closer to a local optimum.

  3. 3.

    Selection of the number of groups \(\widehat{D}\):

A.2 Mean and variance of the Gaussian distributions for the simulation study (Sect. 4.1)

and,

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Volant, S., Bérard, C., Martin-Magniette, ML. et al. Hidden Markov Models with mixtures as emission distributions. Stat Comput 24, 493–504 (2014). https://doi.org/10.1007/s11222-013-9383-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-013-9383-7

Keywords

  • Hidden Markov models
  • Model-based clustering
  • Mixture model
  • Hierarchical algorithm