Abstract
The target article, “Robust Modeling in Cognitive Science,” proposes a number of recommended practices in computational modeling in response to the growing “crisis of confidence” facing many scientific disciplines, including psychology and neuroscience. Those of us who do modeling, write about modeling, teach modeling, and mentor modelers worry deeply about best practices and any new suggestions for making modeling more transparent, trusted, and robust are welcome. Many of the recommendations seem uncontroversial. My commentary focuses on forms of preregistration and postregistration, which constitute three of the four key ideas highlighted as take-home recommendations at the conclusion of the target article. I have chosen to consider these recommendations by reflecting on my own past experiences developing new models and modeling approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
An early example of “adversarial” collaboration (Kahneman and Klein 2009) in cognitive modeling.
We have certainly evolved over the years to using more robust modeling methods (Farrell and Lewandowsky 2018), from minimizing sum-squared-error (SSE) in these early publications, to minimizing chi-squared or maximizing likelihood, to using Bayesian estimation and model comparison when possible (e.g., Annis and Palmeri 2018, 2019).
“Exploratory” is such an unfortunate word since it is so often hedged in science in ways that connote “merely exploratory.” Creating a model that for the first time instantiates a new set of theoretical principles, or accounts for a new type of phenomenon, or establishes links between brain and behavior in a new way is a deeply exploratory process. Whereas fitting an existing model might take a few weeks for well-mentored member of a laboratory, creating a new model or modeling approach, at least in my experience, can take many months if not years of deep, scientific exploration by a team of collaborators.
And my intent is not to rail.
References
Adam, D. (2019). A solution to psychology’s reproducibility problem just failed its first test. Science. Retrieved from https://www.sciencemag.org/news/2019/05/solutionpsychology-s-reproducibility-problem-just-failed-its-first-test. Accessed 23 May 2019.
Anderson, J. R. (1990). The adaptive character of thought. Hillsdale: Erlbaum.
Annese, J., Schenker-Ahmed, N. M., Bartsch, H., Maechler, P., Sheh, C., Thomas, N., ... & Klaming, R. (2014). Postmortem examination of patient H.M.’s brain based on histological sectioning and digital 3D reconstruction. Nature Communications, 5, 3122.
Annis, J., & Palmeri, T.J. (2018). Bayesian statistical approaches to evaluating cognitive models. Wiley Interdisciplinary Reviews in Cognitive Science.
Annis, J., & Palmeri, T. J. (2019). Modeling memory dynamics in visual expertise. Journal of Experimental Psychology: Learning, Memory, and Cognition.
Boucher, L., Palmeri, T. J., Logan, G. D., & Schall, J. D. (2007). Inhibitory control in mind and brain: an interactive race model of countermanding saccades. Psychological Review, 114, 376–397.
Farrell, S., & Lewandowsky, S. (2018). Computational modeling of cognition and behavior. Cambridge University Press.
Gluck, M. A., & Bower, G. H. (1988). Evaluating an adaptive network model of human learning. Journal of Memory & Language, 27, 166–195.
Hintzman, D. L. (1990). Human learning and memory: connections and dissociations. Annual Review of Psychology, 41, 109–139.
Johansen, M. K., & Palmeri, T. J. (2002). Are there representational shifts during category learning? Cognitive Psychology, 45, 482–553.
Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: a failure to disagree. American Psychologist, 64(6), 515–526.
Kruschke, J. K. (1992). ALCOVE: an exemplar-based connectionist model of category learning. Psychological Review, 99, 22–44.
Logan, G. D. (1988). Toward an instance theory of automatization. Psychological Review, 95, 492–527.
Medin, D. L., & Schaffer, M. M. (1978). Context theory of classification learning. Psychological Review, 85, 207–238.
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387.
Noble, W. S. (2009). A quick guide to organizing computational biology projects. PLoS Computational Biology, 5(7), e1000424.
Nosofsky, R. M. (1986). Attention, similarity, and the identification–categorization relationship. Journal of Experimental Psychology: General, 115, 39–57.
Nosofsky, R. M., Gluck, M., Palmeri, T. J., McKinley, S. C., & Glauthier, P. (1994a). Comparing models of rule-based classification learning: a replication and extension of Shepard, Hovland, and Jenkins (1961). Memory & Cognition, 22, 352–369.
Nosofsky, R. M., Palmeri, T. J., & McKinley, S. C. (1994b). Rule-plus-exception model of classification learning. Psychological Review, 101, 53–79.
Nosofsky, R.M., & Palmeri, T.J.(1997). An exemplar-based random walk model of speeded classification. Psychological Review, 104, 266–300
Palmeri, T. J. (1997). Exemplar similarity and the development of automaticity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 324–354.
Palmeri, T. J. (1999). Learning hierarchically structured categories: a comparison of category learning models. Psychonomic Bulletin & Review, 6, 495–503.
Palmeri, T. J., Love, B. C., & Turner, B. M. (2017). Model-based cognitive neuroscience. Journal of Mathematical Psychology, 76, 59–64.
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science, 7, 528–530.
Pitt, M. A., Kim, W., Navarro, D. J., & Myung, J. I. (2006). Global model analysis by parameter space partitioning. Psychological Review, 113, 57–83.
Purcell, B. A., Heitz, R. P., Cohen, J. Y., Schall, J. D., Logan, G. D., & Palmeri, T. J. (2010). Neurallyconstrained modeling of perceptual decision making. Psychological Review, 117, 1113–1143.
Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358–367.
Rouder, J., Haaf, J. M., & Snyder, H. K. (2019). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Sciences, 2(1), 3–11. https://doi.org/10.1177/2515245918801915.
Turner, B. M., Forstmann, B. U., Love, B., Palmeri, T. J., & Van Maanen, L. (2017). Approaches to analysis in model-based cognitive neuroscience. Journal of Mathematical Psychology, 76, 65–79.
Funding
TJP is supported by NSF grant SMA 1640681 and NEI grant R01 EY021833.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Palmeri, T.J. On Testing and Developing Cognitive Models. Comput Brain Behav 2, 193–196 (2019). https://doi.org/10.1007/s42113-019-00041-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42113-019-00041-2