Abstract
Multinomial processing tree (MPT) modeling has been widely and successfully applied as a statistical methodology for measuring hypothesized latent cognitive processes in selected experimental paradigms. In this article, we address the problem of selecting the best MPT model from a set of scientifically plausible MPT models, given observed data. We introduce a minimum description length (MDL) based model-selection approach that overcomes the limitations of existing methods such as the G 2-based likelihood ratio test, the Akaike information criterion, and the Bayesian information criterion. To help ease the computational burden of implementing MDL, we provide a computer program in MATLAB that performs MDL-based model selection for any MPT model, with or without inequality constraints. Finally, we discuss applications of the MDL approach to well-studied MPT models with real data sets collected in two different experimental paradigms: source monitoring and pair clustering. The aforementioned MATLAB program may be downloaded from http://pbr.psychonomic-journals.org/content/supplemental.
Article PDF
Similar content being viewed by others
References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Csaki (Eds.), Proceedings from the Second Symposium on Information Theory (pp. 267–281). Budapest: Akadémiai Kiadó.
Batchelder, W. H. (2009). Cognitive psychometrics: Using multinomial processing tree models as measurement tools. In S. E. Embretson (Ed.), Measuring psychological constructs: Advances in model-based approaches (pp. 71–93). Washington, DC: American Psychological Association.
Batchelder, W. H., & Riefer, D. M. (1980). Separation of storage and retrieval factors in free recall of clusterable pairs. Psychological Review, 87, 375–397.
Batchelder, W. H., & Riefer, D. M. (1986). The statistical analysis of a model for storage and retrieval processes in human memory. British Journal of Mathematical & Statistical Psychology, 39, 129–149.
Batchelder, W. H., & Riefer, D. M. (1990). Multinomial processing models of source monitoring. Psychological Review, 97, 548–564.
Batchelder, W. H., & Riefer, D. M. (1999). Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review, 6, 57–86.
Batchelder, W. H., & Riefer, D. M. (2007). Using multinomial processing tree models to measure cognitive deficits in clinical populations. In R. W. J. Neufeld (Ed.), Advances in clinical cognitive science: Formal modeling of processes and symptoms (pp. 19–50). Washington, DC: American Psychological Association.
Bishara, A. J., & Payne, B. K. (2008). Multinomial process tree models of control and automaticity in weapon misidentification. Journal of Experimental Social Psychology, 45, 524–534.
Casella, G., & Berger, R. L. (2001). Statistical inference (2nd ed.). Belmont, CA: Duxbury Press.
Chechile, R. A. (1998). A new method for estimating model parameters for multinomial data. Journal of Mathematical Psychology, 42, 432–471.
Chechile, R. A. (2004). New multinomial models for the Chechile-Meyer task. Journal of Mathematical Psychology, 48, 364–384.
Chechile, R. A. (2007). A model-based storage-retrieval analysis of developmental dyslexia. In R. W. J. Neufeld (Ed.), Advances in clinical cognitive sciences: Formal modeling of processes and symptoms (pp. 51–80). Washington, DC: American Psychological Association.
Erdfelder, E., Auer, T.-S., Hilbig, B. E., Aßfalg, A., Moshagen, M., & Nadarevic, L. (2009). Multinomial processing tree models: A review of the literature. Zeitschrift für Psychologie, 217, 108–124.
Grünwald, P. [D.] (2000). Model selection based on minimum description length. Journal of Mathematical Psychology, 44, 133–152.
Grünwald, P. D. (2007). The minimum description length principle. Cambridge, MA: MIT Press.
Grünwald, P. D., Myung, I. J., & Pitt, M. A. (Eds.) (2005). Advances in minimum description length: Theory and applications. Cambridge, MA: MIT Press.
Hu, X., & Batchelder, W. H. (1994). The statistical analysis of general processing tree models with the EM algorithm. Psychometrika, 59, 21–47.
Hu, X., & Phillips, G. A. (1999). GPT.EXE: A powerful tool for the visualization and analysis of general processing tree models. Behavior Research Methods, Instruments, & Computers, 31, 220–234.
Klauer, K. C. (2010). Hierarchical multinomial processing tree models: A latent-trait approach. Psychometrika, 75, 70–98. doi:10.1007/ s11336-009-9141-0
Knapp, B. R., & Batchelder, W. H. (2004). Representing parametric order constraints in multitrial applications of multinomial processing tree models. Journal of Mathematical Psychology, 48, 215–229.
Lee, M. D. (2001). On the complexity of additive clustering models. Journal of Mathematical Psychology, 45, 131–148.
Lee, M. D., & Pope, K. J. (2006). Model selection for the rate problem: A comparison of significance testing, Bayesian, and minimum description length statistical inference. Journal of Mathematical Psychology, 50, 193–202.
Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190–204.
Myung, I. J., Forster, M. R., & Browne, M. W. (2000). Guest editors’ introduction: Special issue on model selection. Journal of Mathematical Psychology, 44, 1–2.
Myung, J. I., Navarro, D. J., & Pitt, M. A. (2006). Model selection by normalized maximum likelihood. Journal of Mathematical Psychology, 50, 167–179.
Myung, I. J., & Pitt, M. A. (1997). Applying Occam’s razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin & Review, 4, 79–95.
Myung, J. I., Pitt, M. A., & Navarro, D. J. (2007). Does response scaling cause the generalized context model to mimic a prototype model? Psychonomic Bulletin & Review, 14, 1043–1050.
Navarro, D. J., & Lee, M. D. (2004). Common and distinctive features in stimulus similarity: A modified version of the contrast model. Psychonomic Bulletin & Review, 11, 961–974.
Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472–491.
Purdy, B. P., & Batchelder, W. H. (2009). A context-free language for binary multinomial processing tree models. Journal of Mathematical Psychology, 53, 547–561.
Raftery, A. E. (1999). Bayes factors and BIC: Comment on “A critique of the Bayesian information criterion for model selection.” Sociological Methods & Research, 27, 411–427.
Read, T. R. C., & Cressie, N. A. C. (1988). Goodness-of-fit statistics for discrete multivariate data. New York: Springer.
Riefer, D. M., & Batchelder, W. H. (1988). Multinomial modeling and the measurement of cognitive processes. Psychological Review, 95, 318–339.
Riefer, D. M., & Batchelder, W. H. (1991). Statistical inference for multinomial processing tree models. In J.-P. Doignon & J.-C. Falmagne (Eds.), Mathematical psychology: Current developments (pp. 313–335). New York: Springer.
Riefer, D. M., & Batchelder, W. H. (1995). A multinomial modeling analysis of the recognition-failure paradigm. Memory & Cognition, 23, 611–630.
Riefer, D. M., Hu, X., & Batchelder, W. H. (1994). Response strategies in source monitoring. Journal of Experimental Psychology: Learning, Memory, & Cognition, 20, 680–693.
Riefer, D. M., Knapp, B. R., Batchelder, W. H., Bamber, D., & Manifold, V. (2002). Cognitive psychometrics: Assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychological Assessment, 14, 184–201.
Rissanen, J. J. (1996). Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 42, 40–47.
Rissanen, J. J. (2001). Strong optimality of the normalized ML models as universal codes and information in data. IEEE Transactions on Information Theory, 47, 1712–1717.
Rose, R. G., Rose, P. R., King, N., & Perez, A. (1975). Bilingual memory for related and unrelated sentences. Journal of Experimental Psychology: Human Learning & Memory, 1, 599–606.
Saegert, J., Hamayan, E., & Ahmar, H. (1975). Memory for language of input in polyglots. Journal of Experimental Psychology: Human Learning & Memory, 1, 607–613.
Schwartz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
Schweickert, R. (1993). A multinomial processing tree model for degradation and redintegration in immediate recall. Memory & Cognition, 21, 168–175.
Schweickert, R., & Chen, S. (2008). Tree inference with factors selectively influencing processes in a processing tree. Journal of Mathematical Psychology, 52, 158–183.
Shapiro, A. (1988). Towards a unified theory of inequality constrained testing in multivariate analysis. International Statistical Review, 56, 49–62.
Smith, J. B., & Batchelder, W. H. (2010). Beta-MPT: Multinomial processing tree models for addressing individual differences. Journal of Mathematical Psychology, 54, 167–183.
Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.
Wagenmakers, E.-J., & Waldorp, L. (2006). Editors’ introduction. Journal of Mathematical Psychology, 50, 99–100.
Weakliem, D. L. (1999). A critique of the Bayesian information criterion for model selection. Sociological Methods & Research, 27, 359–397.
Wu, H., Myung, J. I., & Batchelder, W. H. (2010). On the minimum description length complexity of multinomial processing tree models. Journal of Mathematical Psychology, 54, 291–303.
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is based on Hao Wu’s MA thesis submitted to Ohio State University in July 2006. It was supported in part by National Institutes of Health Grant R01-MH57472 to J.I.M. Work on this article by the third author was supported in part by Grant SES-0616657 to Xiangen Hu and W.H.B. from the National Science Foundation.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Wu, H., Myung, J.I. & Batchelder, W.H. Minimum description length model selection of multinomial processing tree models. Psychonomic Bulletin & Review 17, 275–286 (2010). https://doi.org/10.3758/PBR.17.3.275
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/PBR.17.3.275