EM Algorithms from a Non-stochastic Perspective

  • Charles Byrne
Living reference work entry


The EM algorithm is not a single algorithm, but a template for the construction of iterative algorithms. While it is always presented in stochastic language, relying on conditional expectations to obtain a method for estimating parameters in statistics, the essence of the EM algorithm is not stochastic. The conventional formulation of the EM algorithm given in many texts and papers on the subject is inadequate. A new formulation is given here based on the notion of acceptable data.


Probability Density Function Expectation Maximization Expectation Maximization Algorithm Continuous Case Discrete Case 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



I wish to thank Professor Paul Eggermont of the University of Delaware for helpful discussions on these matters.


  1. 1.
    McLachlan, G., Krishnan, T.: The EM Algorithm and Extensions. Wiley, New York (1997)zbMATHGoogle Scholar
  2. 2.
    Meng, X., Pedlow, S.: EM: a bibliographic review with missing articles. In: Proceedings of the Statistical Computing Section, American Statistical Association. American Statistical Association, Alexandria (1992)Google Scholar
  3. 3.
    Meng, X., van Dyk, D.: The EM algorithm- an old folk-song sung to a fast new tune. J. R. Stat. Soc. B 59(3), 511–567 (1997)CrossRefzbMATHGoogle Scholar
  4. 4.
    Becker, M., Yang, I., Lange, K.: EM algorithms without missing data. Stat. Methods Med. Res. 6, 38–54 (1997)CrossRefGoogle Scholar
  5. 5.
    Byrne, C.: Iterative Optimization in Inverse Problems. Taylor and Francis, Boca Raton (2014)zbMATHGoogle Scholar
  6. 6.
    Byrne, C.: Non-stochastic EM algorithms in optimization. J. Nonlinear Convex Anal. (to appear, 2015)Google Scholar
  7. 7.
    Kullback, S., Leibler, R.: On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Hogg, R., McKean, J., Craig, A.: Introduction to Mathematical Statistics, 6th edn. Prentice Hall, Englewood Cliffs (2004)Google Scholar
  9. 9.
    Byrne, C., Eggermont, P.: EM algorithms. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging. Springer, New York (2010)Google Scholar
  10. 10.
    Fessler, J., Ficaro, E., Clinthorne, N., Lange, K.: Grouped-coordinate ascent algorithms for penalized-likelihood transmission image reconstruction. IEEE Trans. Med. Imaging 16(2), 166–175 (1997)CrossRefGoogle Scholar
  11. 11.
    Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 37, 1–38 (1977)MathSciNetGoogle Scholar
  12. 12.
    Csiszár, I., Tusnády, G.: Information geometry and alternating minimization procedures. Stat. Decis. (Suppl. 1), R. Oldenbourg Verlag, München, 205–237 (1984)Google Scholar
  13. 13.
    Byrne, C.: Alternating and sequential unconstrained minimization algorithms. J. Optim. Theory Appl. Electron. 154(3), doi:10.1007/s1090134-2; hardcopy 156(2) (2012)Google Scholar
  14. 14.
    Eggermont, P., LaRiccia, V.: Smoothed maximum likelihood density estimation for inverse problems. Ann. Stat. 23, 199–220 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Eggermont, P., LaRiccia, V.: Maximum Penalized Likelihood Estimation, Volume I: Density Estimation. Springer, New York (2001)Google Scholar
  16. 16.
    Bauschke, H., Combettes, P., Noll, D.: Joint minimization with alternating Bregman proximity operators. Pac. J. Opt. 2, 401–424 (2006)zbMATHMathSciNetGoogle Scholar
  17. 17.
    Wernick, M., Aarsvold, J. (eds.): Emission Tomography: The Fundamentals of PET and SPECT. Elsevier Academic, San Diego (2004)Google Scholar
  18. 18.
    Rockmore, A., Macovski, A.: A maximum likelihood approach to emission image reconstruction from projections. IEEE Trans. Nucl. Sc. NS-23, 1428–1432 (1976)CrossRefGoogle Scholar
  19. 19.
    Shepp, L., Vardi, Y.: Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imaging MI-1, 113–122 (1982)CrossRefGoogle Scholar
  20. 20.
    Byrne, C.: Iterative image reconstruction algorithms based on cross-entropy minimization. IEEE Trans. Image Proc. IP-2, 96–103 (1993)CrossRefGoogle Scholar
  21. 21.
    Byrne, C.: Block-iterative methods for image reconstruction from projections. IEEE Trans. Image Proc. IP-5, 792–794 (1996)CrossRefGoogle Scholar
  22. 22.
    Byrne, C.: Convergent block-iterative algorithms for image reconstruction from inconsistent data. IEEE Trans. Image Proc. IP-6, 1296–1304 (1997)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Byrne, C.: Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative (RBI) methods. IEEE Trans. Image Proc. IP-7, 100–109 (1998)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Gordon, R., Bender, R., Herman, G.T.: Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and x-ray photography. J. Theor. Biol. 29, 471–481 (1970)CrossRefGoogle Scholar
  25. 25.
    Darroch, J., Ratcliff, D.: Generalized iterative scaling for log-linear models. Ann. Math. Stat. 43, 1470–1480 (1972)CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Schmidlin, P.: Iterative separation of sections in tomographic scintigrams. Nucl. Med. 15(1), 1–16 (1972)Google Scholar
  27. 27.
    Censor, Y., Segman, J.: On block-iterative maximization. J. Inf. Opt. Sci. 8, 275–291 (1987)zbMATHMathSciNetGoogle Scholar
  28. 28.
    Byrne, C.: Iterative reconstruction algorithms based on cross-entropy minimization. In: Shepp, L., Levinson, S.E. (eds.) Image Models (and their Speech Model Cousins). IMA Volumes in Mathematics and its Applications vol. 80, pp. 1–11. Springer, New York (1996)CrossRefGoogle Scholar
  29. 29.
    Narayanan, M., Byrne, C., King, M.: An interior point iterative maximum-likelihood reconstruction algorithm incorporating upper and lower bounds with application to SPECT transmission imaging. IEEE Trans. Med. Imaging TMI-20(4), 342–353 (2001)CrossRefGoogle Scholar
  30. 30.
    Byrne, C.: Iterative algorithms for deblurring and deconvolution with constraints. Inverse Probl. 14, 1455–1467 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Everitt, B., Hand, D.: Finite Mixture Distributions. Chapman and Hall, London (1981)CrossRefzbMATHGoogle Scholar
  32. 32.
    Redner, R., Walker, H.: Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev. 26(2), 195–239 (1984)CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Byrne, C.: Likelihood maximization for list-mode emission tomographic image reconstruction. IEEE Trans. Med. Imaging 20(10), 1084–1092 (2001)CrossRefGoogle Scholar
  34. 34.
    Wu, C.F.J.: On the convergence properties of the EM algorithm. Ann. Stat. 11, 95–103 (1983)CrossRefzbMATHGoogle Scholar
  35. 35.
    Boyles, R.: On the convergence of the EM algorithm. J. R. Stat. Soc. B 45, 47–50 (1983)zbMATHMathSciNetGoogle Scholar
  36. 36.
    Lange, K., Carson, R.: EM reconstruction algorithms for emission and transmission tomography. J. Comput. Assist. Tomogr. 8, 306–316 (1984)Google Scholar
  37. 37.
    Vardi, Y., Shepp, L.A., Kaufman, L.: A statistical model for positron emission tomography. J. Am. Stat. Assoc. 80, 8–20 (1985)CrossRefzbMATHMathSciNetGoogle Scholar
  38. 38.
    Lange, K., Bahn, M., Little, R.: A theoretical study of some maximum likelihood algorithms for emission and transmission tomography. IEEE Trans. Med. Imaging MI-6(2), 106–114 (1987)CrossRefGoogle Scholar
  39. 39.
    Barrett, H., White, T., Parra, L.: List-mode likelihood. J. Opt. Soc. Am. A 14, 2914–2923 (1997)CrossRefGoogle Scholar
  40. 40.
    Parra, L., Barrett, H.: List-mode likelihood: EM algorithm and image quality estimation demonstrated on 2-D PET. IEEE Trans. Med. Imaging 17, 228–235 (1998)CrossRefGoogle Scholar
  41. 41.
    Huesman, R., Klein, G., Moses, W., Qi, J., Ruetter, B., Virador, P.: List-mode maximum likelihood reconstruction applied to positron emission mammography (PEM) with irregular sampling. IEEE Trans. Med. Imaging 19(5), 532–537 (2000)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Mathematical Sciences University of Massachusetts LowellLowellUSA

Personalised recommendations