Statistics and Computing

, Volume 26, Issue 1–2, pp 333–348 | Cite as

Explaining the behavior of joint and marginal Monte Carlo estimators in latent variable models with independence assumptions

  • Silia Vitoratou
  • Ioannis Ntzoufras
  • Irini Moustaki


In latent variable models parameter estimation can be implemented by using the joint or the marginal likelihood, based on independence or conditional independence assumptions. The same dilemma occurs within the Bayesian framework with respect to the estimation of the Bayesian marginal (or integrated) likelihood, which is the main tool for model comparison and averaging. In most cases, the Bayesian marginal likelihood is a high dimensional integral that cannot be computed analytically and a plethora of methods based on Monte Carlo integration (MCI) are used for its estimation. In this work, it is shown that the joint MCI approach makes subtle use of the properties of the adopted model, leading to increased error and bias in finite settings. The sources and the components of the error associated with estimators under the two approaches are identified here and provided in exact forms. Additionally, the effect of the sample covariation on the Monte Carlo estimators is examined. In particular, even under independence assumptions the sample covariance will be close to (but not exactly) zero which surprisingly has a severe effect on the estimated values and their variability. To address this problem, an index of the sample’s divergence from independence is introduced as a multivariate extension of covariance. The implications addressed here are important in the majority of practical problems appearing in Bayesian inference of multi-parameter models with analogous structures.


Marginal likelihood Bayes factor  Monte Carlo integration 


  1. Aguilar, O., West, M.: Bayesian dynamic factor models and portfolio allocation. J. Bus. Econ. Stat. 18, 338–357 (2000)Google Scholar
  2. Baker, F.: An investigation of the item parameter recovery characteristics of a Gibbs sampling procedure. Appl. Psychol. Meas. 22, 153–169 (1998)CrossRefGoogle Scholar
  3. Bartholomew, D., Knott, M., Moustaki, I.: Latent Variable Models and Factor Analysis: A Unified Approach. Wiley Series on Probability and Statistics, 3rd edn. Wiley, London (2011)CrossRefGoogle Scholar
  4. Bock, R., Aitkin, M.: Marginal maximum likelihood estimation of item parameters: application of an EM algorithm. Psychometrika 46, 443–459 (1981)MathSciNetCrossRefGoogle Scholar
  5. Bock, R.D., Lieberman, M.: Fitting a response model for n dichotomously scored items. Psychometrika 35, 179–197 (1970)CrossRefGoogle Scholar
  6. Bratley, P., Fox, B.L., Schrage, L.: A Guide to Simulation, 2nd edn. Springer, Berlin (1987)CrossRefGoogle Scholar
  7. Carlin, B.P., Louis, T.A.: Bayes and Empirical Bayes Methods for Data Analysis, 2nd edn. Chapman & Hall/CRC, London (2000)zbMATHCrossRefGoogle Scholar
  8. Chib, S., Jeliazkov, I.: Marginal likelihood from the Metropolis–Hastings output. J. Am. Stat. Assoc. 96, 270–281 (2001)zbMATHMathSciNetCrossRefGoogle Scholar
  9. Congdon, P.: Applied Bayesian Hierarchical Methods. Chapman and Hall/CRC, London (2010)zbMATHCrossRefGoogle Scholar
  10. DiCiccio, T.J., Kass, R.E., Raftery, A., Wasserman, L.: Computing Bayes factors by combining simulation and asymptotic approximations. J. Am. Stat. Assoc. 92(439), 903–915 (1997)zbMATHMathSciNetCrossRefGoogle Scholar
  11. Flegal, J., Jones, G.: Batch means and spectral variance estimators in markov chain monte carlo. Ann. Stat. 38, 1034–1070 (2010)zbMATHMathSciNetCrossRefGoogle Scholar
  12. Fouskakis, D., Ntzoufras, I., Draper, D.: Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care. Ann. Appl. Stat. 3, 663–690 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  13. Friel, N., Pettitt, N.: Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. Ser. (Stat. Methodol.) 70(3), 589–607 (2008)zbMATHMathSciNetCrossRefGoogle Scholar
  14. Gelfand, A.E., Dey, D.K.: Bayesian model choice: asymptotics and exact calculations. J. R. Stat. Soc. Ser. B (Methodol.) 56(3), 501–514 (1994)zbMATHMathSciNetGoogle Scholar
  15. Gelman, A., Meng, X.-L.: Simulating normalizing constants: from importance sampling to bridge sampling to path sampling. Stat. Sci. 13(2), 163–185 (1998)zbMATHMathSciNetCrossRefGoogle Scholar
  16. Geweke, J., Zhou, G.: Measuring the pricing error of the arbitrage pricing theory. Rev. Financ. Stud. 9, 557–587 (1996)CrossRefGoogle Scholar
  17. Gifford, J.A., Swaminathan, H.: Bias and the effect of priors in Bayesian estimation of parameters of item response models. Appl. Psychol. Meas. 14, 33–43 (1990)CrossRefGoogle Scholar
  18. Goodman, L.A.: The variance of the product of K random variables. J. Am. Stat. Assoc. 57, 54–60 (1962)zbMATHGoogle Scholar
  19. Huber, P., Ronchetti, E., Victoria-Feser, M.-P.: Estimation of generalized linear latent variable models. J. R. Stat. Soc. Ser. B 66, 893–908 (2004)zbMATHMathSciNetCrossRefGoogle Scholar
  20. Jones, G., Haran, M., Caffo, B., Neath, R.: Fixed-width output analysis for Markov chain Monte Carlo. J. Am. Stat. Assoc. 101, 1537–1547 (2006)zbMATHMathSciNetCrossRefGoogle Scholar
  21. Kang, T., Cohen, A.S.: Irt model selection methods for dichotomous items. Appl. Psychol. Meas. 31(4), 331358 (2007)MathSciNetCrossRefGoogle Scholar
  22. Kass, R., Raftery, A.: Bayes factors. J. Am. Stat. Assoc. 90, 773–795 (1995)zbMATHCrossRefGoogle Scholar
  23. Kim, S.-H., Cohen, A.S., Baker, F.B., Subkoviak, M.J., Leonard, T.: An investigation of hierarchical Bayes procedures in item response theory. Psychometrika 59(3), 405–421 (1994)zbMATHCrossRefGoogle Scholar
  24. Koehler, E., Brown, E., Haneuse, S.J.-P.A.: On the assessment of Monte Carlo error in simulation-based statistical analyses. Am. Stat. 63(2), 155–162 (2009)MathSciNetCrossRefGoogle Scholar
  25. Lewis, S., Raftery, A.: Estimating Bayes factors via posterior simulation with the Laplace Metropolis estimator. J. Am. Stat. Assoc. 92, 648–655 (1997)zbMATHMathSciNetGoogle Scholar
  26. Lopes, H.F., West, M.: Bayesian model assessment in factor analysis. Stat. Sin. 14, 4167 (2004)MathSciNetGoogle Scholar
  27. Lord, F.M.: Applications of Item Response Theory to Practical Testing Problems. Erlbaum Associates, Hillsdale (1980)Google Scholar
  28. Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley, Oxford (1968)zbMATHGoogle Scholar
  29. Meketon, M.S., Schmeiser, B.W. Overlapping batch means: something for nothing?” In: Proceedings of the 1984 Winter Simulation Conference, pp. 227–230. Institute of Electrical and Electronics Engineers Inc., Piscataway (1984) Google Scholar
  30. Meng, X.-L., Schilling, S.: Warp bridge sampling. J. Comput. Graph. Stat. 11(3), 552–586 (2002)MathSciNetCrossRefGoogle Scholar
  31. Meng, X.-L., Wong, W.-H.: Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Stat. Sin. 6, 831–860 (1996)zbMATHMathSciNetGoogle Scholar
  32. Mislevy, R.: Bayes modal estimation in item response models. Psychometrika 51, 177–195 (1986)zbMATHMathSciNetCrossRefGoogle Scholar
  33. Moustaki, I., Knott, M.: Generalized latent trait models. Psychometrika 65, 391–411 (2000)zbMATHMathSciNetCrossRefGoogle Scholar
  34. Ntzoufras, I., Dellaportas, P., Forster, J.: Bayesian variable and link determination for generalised linear models. J. Stat. Plan. Inference 111(1–2), 165–180 (2003)zbMATHMathSciNetCrossRefGoogle Scholar
  35. Patz, R., Junker, B.: A straightforward approach to Markov chain Monte Carlo methods for item response models. J. Educ. Behav. Stat. 24, 146–178 (1999)CrossRefGoogle Scholar
  36. Rabe-Hesketh, S., Skrondal, A., Pickles, A.: Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. J. Econom. 128, 301–323 (2005)MathSciNetCrossRefGoogle Scholar
  37. Schilling, S., Bock, R.: High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika 70, 533–555 (2005)zbMATHMathSciNetGoogle Scholar
  38. Schmeiser, B.W.: Batch size effects in the analysis of simulation output. Oper. Res. 30, 556–568 (1982)zbMATHMathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Silia Vitoratou
    • 1
  • Ioannis Ntzoufras
    • 2
  • Irini Moustaki
    • 3
  1. 1.Department of BiostatisticsInstitute of Psychiatry, King’s CollegeLondonUK
  2. 2.Department of StatisticsAthens University of Economics and BusinessAthensGreece
  3. 3.Department of StatisticsLondon School of EconomicsLondonUK

Personalised recommendations