Using Hamiltonian Monte Carlo to estimate the log-linear cognitive diagnosis model via Stan

Article

Abstract

The Bayesian literature has shown that the Hamiltonian Monte Carlo (HMC) algorithm is powerful and efficient for statistical model estimation, especially for complicated models. Stan, a software program built upon HMC, has been introduced as a means of psychometric modeling estimation. However, there are no systemic guidelines for implementing Stan with the log-linear cognitive diagnosis model (LCDM), which is the saturated version of many cognitive diagnostic model (CDM) variants. This article bridges the gap between Stan application and Bayesian LCDM estimation: Both the modeling procedures and Stan code are demonstrated in detail, such that this strategy can be extended to other CDMs straightforwardly.

Keywords

Markov chain Monte Carlo (MCMC) Bayesian Cognitive diagnostic model LCDM Stan Hamiltonian Monte Carlo (HMC) 

References

  1. Almond, R. (2014). Comparison of two MCMC algorithms for hierarchical mixture models. In Bayesian Modeling Application Workshop at the Uncertainty in Artificial Intelligence Conference (pp. 1–19). Corvallis, OR: AUAI Press.Google Scholar
  2. Annis, J., Miller, B. J., & Palmeri, T. J. (2017). Bayesian inference with Stan: A tutorial on adding custom distributions. Behavior Research Methods, 49, 863–886. doi: https://doi.org/10.3758/s13428-016-0746-9 CrossRefPubMedPubMedCentralGoogle Scholar
  3. Betancourt, M. J., Byrne, S., & Girolami, M. (2014). Optimizing the integrator step size for Hamiltonian Monte Carlo. arXiv:1411.6669Google Scholar
  4. Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., … Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 20, 1–37.Google Scholar
  5. Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40, 454–476.CrossRefGoogle Scholar
  6. Dai, S., Svetina, D., & Chen, C. (2018). Investigation of missing responses in Q-matrix validation. Applied Psychological Measurement. Advance online publication. doi: https://doi.org/10.1177/0146621618762742
  7. de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.CrossRefGoogle Scholar
  8. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.CrossRefGoogle Scholar
  9. de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.CrossRefGoogle Scholar
  10. DeCarlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36, 447–468.CrossRefGoogle Scholar
  11. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EMalgorithm. Journal of the Royal Statistical Society, 39, 1–38.Google Scholar
  12. Gelman, A., Lee, D., & Guo, J. (2015). Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics, 40, 530–543.CrossRefGoogle Scholar
  13. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. doi: https://doi.org/10.2307/2246093 CrossRefGoogle Scholar
  14. Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6, 721–741. doi: https://doi.org/10.1109/TPAMI.1984.4767596 CrossRefGoogle Scholar
  15. George, A. C., Robitzsch, A., Kiefer, T., Groß, J., & Ünlü, A. (2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1–24.CrossRefGoogle Scholar
  16. Gilks, W. R. (1998). Full conditional distributions. In W. R. Gilks, S. Richardson, & D. J. Spiegelhalter (Eds.), Markov chain Monte Carlo in practice (pp. 75–88). Boca Raton, FL: Chapman & Hall.Google Scholar
  17. Girolami, M., & Calderhead, B. (2011). Riemann manifold Langevin and Hamiltonian Monte Carlo methods. Journal of the Royal Statistical Society: Series B, 73, 123–214.CrossRefGoogle Scholar
  18. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.CrossRefGoogle Scholar
  19. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Doctoral dissertation), University of Illinois at Urbana-Champaign, IL.Google Scholar
  20. Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.CrossRefGoogle Scholar
  21. Hoffman, M. D., & Gelman, A. (2014). The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593–1623.Google Scholar
  22. Ishwaran, H., & Zarepour, M. (2002). Dirichlet prior sieves in finite normal mixtures. Statistica Sinica, 941–963.Google Scholar
  23. Jiang, S., Wang, C., & Weiss, D. J. (2016). Sample size requirements for estimation of item parameters in the multidimensional graded response model. Frontiers in Psychology, 7, 109. doi: https://doi.org/10.3389/fpsyg.2016.00109 PubMedPubMedCentralGoogle Scholar
  24. Jiang, Z., & Skorupski, W. (2017). A Bayesian approach to estimating variance components within a multivariate generalizability theory framework. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-017-0986-3Google Scholar
  25. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
  26. Knott, M., & Bartholomew, D. J. (1999). Latent variable models and factor analysis (No. 7). Edward Arnold.Google Scholar
  27. Lao, H., & Templin, J. (2016, April). Estimation of diagnostic classification models without constraints: Issues with class label switching. Paper presented at Annual Meeting of the National Council on Measurement in Education, Washington, DC.Google Scholar
  28. Lee, M. D., & Wagenmakers, E. J. (2014). Bayesian cognitive modeling: A practical course. New York, NY: Cambridge University Press.Google Scholar
  29. Lee, S. T. (2016, November 21). DINA model with independent attributes. Retrieved from http://mc-stan.org/documentation/case-studies/dina_independent.html.
  30. Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100, 1989–2001.CrossRefGoogle Scholar
  31. Liu, R. (2017). Misspecification of attribute structure in diagnostic measurement. Educational and Psychological Measurement.  https://doi.org/10.1177/0013164417702458.
  32. Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS—A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.CrossRefGoogle Scholar
  33. Luo, Y., & Jiao, H. (2017). Using the Stan program for Bayesian item response theory. Educational and Psychological Measurement, 78, 384–408. doi: https://doi.org/10.1177/0013164417693666 CrossRefGoogle Scholar
  34. Ma, W., & de la Torre, J. (2016). GDINA: The Generalized DINA model framework (R package version 0.13.0). Available online at http://CRAN. R-project.org/package=GDINA.
  35. Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99–120.Google Scholar
  36. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.CrossRefGoogle Scholar
  37. Merkle, E. C., & Wang, T. (2018). Bayesian latent variable models for the analysis of experimental psychology data. Psychonomic Bulletin & Review, 25, 256–270. doi: https://doi.org/10.3758/s13423-016-1016-7 CrossRefGoogle Scholar
  38. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953). Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21, 1087–1092.CrossRefGoogle Scholar
  39. Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi: https://doi.org/10.1037/a0026802 CrossRefPubMedGoogle Scholar
  40. Muthén, L. K., & Muthén, B. O. (2013). Mplus user’s guide (Version 7.1) [Computer software and manual]. Los Angeles, CA: Muthén & Muthén.Google Scholar
  41. Neal, R. M. (2011). MCMC using Hamiltonian dynamics. In S. Brooks (Ed.), Handbook of Markov Chain Monte Carlo (pp. 113–162). Boca Raton, FL: CRC Press/Taylor & Francis.Google Scholar
  42. Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Paper presented at the 3rd International Workshop on Distributed Statistical Computing, Vienna, Austria.Google Scholar
  43. Plummer, M. (2008). Penalized loss functions for Bayesian model comparison. Biostatistics, 9, 523–539CrossRefPubMedGoogle Scholar
  44. R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from www.Rproject.org/
  45. Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.Google Scholar
  46. Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219–262.Google Scholar
  47. Shi, D., Song, H., Liao, X., Terry, R., & Snyder, L. A. (2017). Bayesian SEM for specification search problems in testing factorial invariance. Multivariate Behavioral Research, 52, 430–444.CrossRefPubMedGoogle Scholar
  48. da Silva, M. A., de Oliveira, E. S. B., von Davier, A. A., & Bazán, J. L. (2017). Estimating the DINA model parameters using the No-U-Turn Sampler. Biometrical Journal. Advance online publication. doi: https://doi.org/10.1002/bimj.201600225
  49. Sorensen, T., Hohenstein, S., & Vasishth, S. (2016). Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists. Quantitative Methods for Psychology, 12, 175–200.CrossRefGoogle Scholar
  50. Stan Development Team. (2016a). rstan: R interface to Stan (R package version 2.0.3). Retrieved from http://mc-stan.org
  51. Stan Development Team. (2016b). Stan: A C++ library for probability and sampling (Version 2.8.0). Retrieved from http://mc-stan.org
  52. Stephens, M. (2000). Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B, 62, 795–809.CrossRefGoogle Scholar
  53. Templin, J., & Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317–339.CrossRefPubMedGoogle Scholar
  54. Templin, J., & Hoffman, L. (2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement: Issues and Practice, 32, 37–50.CrossRefGoogle Scholar
  55. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.  https://doi.org/10.1037/1082-989X.11.3.287 CrossRefPubMedGoogle Scholar
  56. van der Linde, A. (2005). DIC in variable selection. Statistica Neerlandica, 59, 45–56.CrossRefGoogle Scholar
  57. Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432.CrossRefGoogle Scholar
  58. von Davier, M. (2009). Some notes on the reinvention of latent structure models as diagnostic classification models. Measurement: Interdisciplinary Research and Perspectives, 7, 67–74.Google Scholar
  59. Zhan, P. (2017). Using JAGS for Bayesian cognitive diagnosis models: A tutorial. arXiv:1708.02632Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.University of AlabamaTuscaloosaUSA
  2. 2.University of WyomingLaramieUSA

Personalised recommendations