Statistics and Computing

, Volume 18, Issue 1, pp 47–57 | Cite as

Particle methods for maximum likelihood estimation in latent variable models

  • Adam M. JohansenEmail author
  • Arnaud Doucet
  • Manuel Davy


Standard methods for maximum likelihood parameter estimation in latent variable models rely on the Expectation-Maximization algorithm and its Monte Carlo variants. Our approach is different and motivated by similar considerations to simulated annealing; that is we build a sequence of artificial distributions whose support concentrates itself on the set of maximum likelihood estimates. We sample from these distributions using a sequential Monte Carlo approach. We demonstrate state-of-the-art performance for several applications of the proposed approach.


Latent variable models Markov chain Monte Carlo Maximum likelihood Sequential Monte Carlo Simulated annealing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Amzal, B., Bois, F.Y., Parent, E., Robert, C.P.: Bayesian optimal design via interacting particle systems. J. Am. Stat. Assoc. 101(474), 773–785 (2006) zbMATHCrossRefMathSciNetGoogle Scholar
  2. Chopin, N.: Central limit theorem for sequential Monte Carlo methods and its applications to Bayesian inference. Ann. Stat. 32(6), 2385–2411 (2004) zbMATHCrossRefMathSciNetGoogle Scholar
  3. Del Moral, P.: Feynman-Kac Formulae: Genealogical and Interacting Particle Systems with Applications. Probability and its Applications. Springer, New York (2004) zbMATHGoogle Scholar
  4. Del Moral, P., Doucet, A., Jasra, A.: Sequential Monte Carlo samplers. J. R. Stat. Soc. B 63(3), 411–436 (2006) CrossRefMathSciNetGoogle Scholar
  5. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM Algorithm. J. R. Stat. Soc. B 39, 2–38 (1977) MathSciNetGoogle Scholar
  6. Doucet, A., de Freitas, N., Gordon, N. (eds.): Sequential Monte Carlo Methods in Practice. Statistics for Engineering and Information Science. Springer, New York (2001) zbMATHGoogle Scholar
  7. Doucet, A., Godsill, S.J., Robert, C.P.: Marginal maximum a posteriori estimation using Markov chain Monte Carlo. Stat. Comput. 12, 77–84 (2002) CrossRefMathSciNetGoogle Scholar
  8. Doucet, A., Briers, M., Sénécal, S.: Efficient block sampling strategies for sequential Monte Carlo methods. J. Comput. Graph. Stat. 15(3), 693–711 (2006) CrossRefGoogle Scholar
  9. Escobar, M.D., West, M.: Bayesian density estimation and inference using mixtures. J. Am. Stat. Assoc. 90(430), 577–588 (1995) zbMATHCrossRefMathSciNetGoogle Scholar
  10. Gaetan, C., Yao, J.-F.: A multiple-imputation Metropolis version of the EM algorithm. Biometrika 90(3), 643–654 (2003) CrossRefMathSciNetGoogle Scholar
  11. Hwang, C.-R.: Laplace’s method revisited: weak convergence of probability measures. Ann. Probab. 8(6), 1177–1182 (1980) zbMATHMathSciNetGoogle Scholar
  12. Jacquier, E., Johannes, M., Polson, N.: MCMC maximum likelihood for latent state models. J. Econom. 137(2), 615–640 (2007) CrossRefGoogle Scholar
  13. Johansen, A.M.: Some non-standard sequential Monte Carlo methods with applications, Ph.D. thesis. University of Cambridge, Department of Engineering (2006) Google Scholar
  14. Liu, J.S., Chen, R.: Sequential Monte Carlo methods for dynamic systems. J. Am. Stat. Assoc. 93(443), 1032–1044 (1998) zbMATHCrossRefGoogle Scholar
  15. Müller, P., Sansó, B., de Iorio, M.: Optimum Bayesian design by inhomogeneous Markov chain simulation. J. Am. Stat. Assoc. 99(467), 788–798 (2004) zbMATHCrossRefGoogle Scholar
  16. Robert, C.P., Casella, G.: Monte Carlo Statistical Methods, 2nd edn. Springer, New York (2004) zbMATHGoogle Scholar
  17. Robert, C.P., Titterington, D.M.: Reparameterization strategies for hidden Markov models and Bayesian approaches to maximum likelihood estimation. Stat. Comput. 8, 145–158 (1998) CrossRefGoogle Scholar
  18. Roeder, K.: Density estimation with cofidence sets exemplified by superclusters and voids in galaxies. J. Am. Stat. Assoc. 85(411), 617–624 (1990) zbMATHCrossRefGoogle Scholar
  19. Roeder, K., Wasserman, L.: Practical Bayesian density estimation using mixtures of normals. J. Am. Stat. Assoc. 92(439), 894–902 (1997) zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Mathematics, University WalkUniversity of BristolBristolUK
  2. 2.Department of Statistics & Department of Computer ScienceUniversity of British ColumbiaVancouverCanada
  3. 3.LAGIS UMR 8146Villeneuve d’Ascq CedexFrance

Personalised recommendations