Flexibility versus generalizability in model selection
Which quantitative method should be used to choose among competing mathematical models of cognition? Massaro, Cohen, Campbell, and Rodriguez (2001) favor root mean squared deviation (RMSD), choosing the model that provides the best fit to the data. Their simulation results appear to legitimize its use for comparing two models of information integration because it performed just as well as Bayesian model selection (BMS), which had previously been shown by Myung and Pitt (1997) to be a superior alternative selection method because it considers a model’s complexity in addition to its fit. In the present study, after contrasting the theoretical approaches to model selection espoused by Massaro et al. and Myung and Pitt, we discuss the cause of the inconsistencies by expanding on the simulations of Massaro et al. Findings demonstrate that the results from model recovery simulations can be misleading if they are not interpreted relative to the data on which they were evaluated, and that BMS is a more robust selection method.
- Gilks, W. R., Richardson, S., &Spiegelhalter, D. J. (1996).Markov chain Monte Carlo in practice. New York: Chapman & Hall.
- Grunwald, P. (in press).Minimum description length and maximum probability. Heidelberg: Springer-Verlag.
- Hansen, M. H., &Yu, B. (2001). Model selection and the principle of minimum description length.Journal of the American Statistical Association,96, 746–774. CrossRef
- Linhart, H., &Zucchini, W. (1986).Model selection. New York: Wiley.
- Massaro, D. W. (1988). Some criticisms of connectionist models of human performance.Journal of Memory & Language,27, 213–234. CrossRef
- Massaro, D. W. (1998).Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
- Massaro, D. W., &Cohen, M. M. (1983). Phonological context in speech perception.Perception & Psychophysics,34, 338–348. CrossRef
- Massaro, D. W., Cohen, M. M., Campbell, C. S., &Rodriguez, T. (2001). Bayes factor of model selection validates FLMP.Psychonomic Bulletin & Review,8, 1–17. CrossRef
- Myung, I. J., Balasubramanian, V., &Pitt, M. A. (2000). Counting probability distributions: Differential geometry and model selection.Proceedings of the National Academy of Sciences,97, 11170–11175. CrossRef
- Myung, I. J., Kim, C., &Pitt, M. A. (2000). Toward an explanation of the power law artifact: Insights from response surface analysis.Memory & Cognition,28, 832–840. CrossRef
- Myung, I. J., &Pitt, M. A. (1997). Applying Occam’s razor in modeling cognition: A Bayesian approach.Psychonomic Bulletin & Review,4, 79–95. CrossRef
- Pitt, M. A. (1995). The locus of the lexical shift in phoneme identification.Journal of Experimental Psychology: Learning, Memory, & Cognition,21, 1037–1052. CrossRef
- Pitt, M. A., Myung, I. J., &Zhang, S. (2002). Toward a method of selecting among computational models of cognition.Psychological Review,109, 472–491. CrossRef
- Rissanen, J. (1983). A universal prior for integers and estimation by minimum description length.Annals of Statistics,11, 416–431. CrossRef
- Vitanyi, P., &Li, M. (2000). Minimum description length, Bayesianism and Kolmogorov complexity.IEEE Transactions on Information Theory,46, 446–464. CrossRef
- Flexibility versus generalizability in model selection
Psychonomic Bulletin & Review
Volume 10, Issue 1 , pp 29-44
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links