Skip to main content

A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants

  • Chapter
Learning in Graphical Models

Part of the book series: NATO ASI Series ((ASID,volume 89))

Abstract

The EM algorithm performs maximum likelihood estimation for data in which some variables are unobserved. We present a function that resembles negative free energy and show that the M step maximizes this function with respect to the model parameters and the E step maximizes it with respect to the distribution over the unobserved variables. From this perspective, it is easy to justify an incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step. This variant is shown empirically to give faster convergence in a mixture estimation problem. A variant of the algorithm that exploits sparse conditional distributions is also described, and a wide range of other variant algorithms are also seen to be possible.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  • Csiszàr I. and Tusnàdy, G. (1984) “Information geometry and alternating minimization procedures”, in E. J. Dudewicz, et al(editors) Recent Results in Estimation Theory and Related Topics(Statistics and Decisions, Supplement Issue No. 1, 1984 ).

    Google Scholar 

  • Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977) “Maximum likelihood from incomplete data via the EM algorithm” (with discussion), Journal of the Royal Statistical Society B, vol. 39, pp. 1–38.

    MathSciNet  MATH  Google Scholar 

  • Hathaway, R. J. (1986) “Another interpretation of the EM algorithm for mixture distributions”, Statistics and Probability Letters, vol. 4, pp. 53–56.

    Article  MathSciNet  MATH  Google Scholar 

  • McLachlan, G. J. and Krishnan, T. (1997) The EM Algorithm and Extensions, New York: Wiley.

    MATH  Google Scholar 

  • Meng, X. L. and Rubin, D. B. (1992) “Recent extensions of the EM algorithm (with discussion) ”, in J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith (editors), Bayesian Statistics4, Oxford: Clarendon Press.

    Google Scholar 

  • Meng, X. L. and van Dyk, D. (1997) “The EM algorithm — an old folksong sung to a fast new tune” (with discussion), Journal of the Royal Statistical Society B, vol. 59, pp. 511–567.

    Article  MATH  Google Scholar 

  • Nowlan, S. J. (1991) Soft Competitive Adaptation: Neural Network Learning Algorithms based on Fitting Statistical Mixtures, Ph. D. thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Neal, R.M., Hinton, G.E. (1998). A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants. In: Jordan, M.I. (eds) Learning in Graphical Models. NATO ASI Series, vol 89. Springer, Dordrecht. https://doi.org/10.1007/978-94-011-5014-9_12

Download citation

  • DOI: https://doi.org/10.1007/978-94-011-5014-9_12

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-010-6104-9

  • Online ISBN: 978-94-011-5014-9

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics