Abstract
Which quantitative method should be used to choose among competing mathematical models of cognition? Massaro, Cohen, Campbell, and Rodriguez (2001) favor root mean squared deviation (RMSD), choosing the model that provides the best fit to the data. Their simulation results appear to legitimize its use for comparing two models of information integration because it performed just as well as Bayesian model selection (BMS), which had previously been shown by Myung and Pitt (1997) to be a superior alternative selection method because it considers a model’s complexity in addition to its fit. In the present study, after contrasting the theoretical approaches to model selection espoused by Massaro et al. and Myung and Pitt, we discuss the cause of the inconsistencies by expanding on the simulations of Massaro et al. Findings demonstrate that the results from model recovery simulations can be misleading if they are not interpreted relative to the data on which they were evaluated, and that BMS is a more robust selection method.
Article PDF
Similar content being viewed by others
References
Gilks, W. R., Richardson, S., &Spiegelhalter, D. J. (1996).Markov chain Monte Carlo in practice. New York: Chapman & Hall.
Grunwald, P. (in press).Minimum description length and maximum probability. Heidelberg: Springer-Verlag.
Hansen, M. H., &Yu, B. (2001). Model selection and the principle of minimum description length.Journal of the American Statistical Association,96, 746–774.
Linhart, H., &Zucchini, W. (1986).Model selection. New York: Wiley.
Massaro, D. W. (1988). Some criticisms of connectionist models of human performance.Journal of Memory & Language,27, 213–234.
Massaro, D. W. (1998).Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
Massaro, D. W., &Cohen, M. M. (1983). Phonological context in speech perception.Perception & Psychophysics,34, 338–348.
Massaro, D. W., Cohen, M. M., Campbell, C. S., &Rodriguez, T. (2001). Bayes factor of model selection validates FLMP.Psychonomic Bulletin & Review,8, 1–17.
Myung, I. J., Balasubramanian, V., &Pitt, M. A. (2000). Counting probability distributions: Differential geometry and model selection.Proceedings of the National Academy of Sciences,97, 11170–11175.
Myung, I. J., Kim, C., &Pitt, M. A. (2000). Toward an explanation of the power law artifact: Insights from response surface analysis.Memory & Cognition,28, 832–840.
Myung, I. J., &Pitt, M. A. (1997). Applying Occam’s razor in modeling cognition: A Bayesian approach.Psychonomic Bulletin & Review,4, 79–95.
Pitt, M. A. (1995). The locus of the lexical shift in phoneme identification.Journal of Experimental Psychology: Learning, Memory, & Cognition,21, 1037–1052.
Pitt, M. A., Myung, I. J., &Zhang, S. (2002). Toward a method of selecting among computational models of cognition.Psychological Review,109, 472–491.
Rissanen, J. (1983). A universal prior for integers and estimation by minimum description length.Annals of Statistics,11, 416–431.
Vitanyi, P., &Li, M. (2000). Minimum description length, Bayesianism and Kolmogorov complexity.IEEE Transactions on Information Theory,46, 446–464.
Author information
Authors and Affiliations
Corresponding authors
Additional information
This work was supported by Research Grant R01 MH57472 from the National Institute of Mental Health.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Pitt, M.A., Kim, W. & Myung, I.J. Flexibility versus generalizability in model selection. Psychonomic Bulletin & Review 10, 29–44 (2003). https://doi.org/10.3758/BF03196467
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03196467