Robust Modeling in Cognitive Science


In an attempt to increase the reliability of empirical findings, psychological scientists have recently proposed a number of changes in the practice of experimental psychology. Most current reform efforts have focused on the analysis of data and the reporting of findings for empirical studies. However, a large contingent of psychologists build models that explain psychological processes and test psychological theories using formal psychological models. Some, but not all, recommendations borne out of the broader reform movement bear upon the practice of behavioral or cognitive modeling. In this article, we consider which aspects of the current reform movement are relevant to psychological modelers, and we propose a number of techniques and practices aimed at making psychological modeling more transparent, trusted, and robust.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.


  1. 1.

    Note that registered reports involve more than preregistration: They also involve a journal’s guarantee that a paper will be published regardless of how the data turn out.

  2. 2.

  3. 3.


  1. Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école Américaine. Econometrica, 21, 503–546.

  2. Allais, M. (1979). The foundations of a positive theory of choice involving risk and a criticism of the postulates and axioms of the American School. In Allais, M., & Hagen, O. (Eds.) Expected utility hypothesis and the Allais paradox (pp. 27–145). Dordrecht: Riedel.

  3. Alogna, V., Attaya, M.K., Aucoin, P., Bahník, Š., Birch, S., Birt, A.R., et al. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.

  4. Batchelder, W.H. (2010). Cognitive psychometrics: using multinomial processing tree models as measurement tools. In Embretson, S. (Ed.) Measuring psychological constructs: advances in model-based approaches. Washington, DC: American Psychological Association Books.

  5. Baumeister, A.A., & Kellas, G. (1968). Reaction time and mental retardation. International Review of Research in Mental Retardation, 3, 163–193.

  6. Baumgaertner, B., Devezer, B., Buzbas, E.O., Nardin, L.G. (2018). A model-centric analysis of openness, replication, and reproducibility. arXiv:1811.04525.

  7. Beck, K. (2003). Test-driven development: by example. Addison-Wesley Professional.

  8. Bell, R.M., Koren, Y., Volinsky, C. (2010). All together now: a perspective on the Netflix prize. Chance, 23, 24–29.

  9. Birnbaum, M.H., & Quispe-Torreblanca, E.G. (2018). TEMAP2.R: true and error model analysis program in R. Judgment and Decision Making, 13, 428–440.

  10. Bones, A.K. (2012). We knew the future all along: scientific hypothesizing is much more accurate than other forms of precognition—a satire in one part. Perspectives on Psychological Science, 7, 307–309.

  11. Brown, G.D.A., Neath, I., Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114(1), 539–576.

  12. Brown, S.D., Marley, A.A.J., Donkin, C., Heathcote, A. (2008). An integrated model of choices and response times in absolute identification. Psychological Review, 115, 396–425.

  13. Busemeyer, J.R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171–189.

  14. Cavagnaro, D.R., Pitt, M.A., Gonzalez, R., Myung, I.J. (2013). Discriminating among probability weighting functions with adaptive design optimization. Journal of Risk and Uncertainty, 47, 255–289.

  15. Chambers, C.D. (2013). Registered reports: a new publishing initiative at cortex. Cortex, 49, 609–610.

  16. Chambers, C.D., Dienes, Z., McIntosh, R.D., Rotshtein, P., Willmes, K. (2015). Registered reports: realigning incentives in scientific publishing. Cortex, 66, A1–A2.

  17. Cook, S.R., Gelman, A., Rubin, D.B. (2006). Validation of software for Bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics, 15(3), 675–692.

  18. Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2011). Output interference in recognition memory. Journal of Memory and Language, 64(4), 316–326.

  19. Dutilh, G., Vandekerckhove, J., Ly, A., Matzke, D., Pedroni, A., Frey, R., Wagenmakers, E.-J. (2017). A test of the diffusion model explanation for the worst performance rule using preregistration and blinding. Attention, Perception, & Psychophysics, 79, 713–725.

  20. Dutilh, G., Annis, J., Brown, S.D., Cassey, P., Evans, N.J., Grasman, R.P.P.P., Donkin, C. (2018). The quality of response time data inference: a blinded, collaborative assessment of the validity of cognitive models. Psychonomic Bulletin & Review.

  21. Epper, T., & Fehr-Duda, H. (2018). The missing link: unifying risk taking and time discounting (Tech. Rep. Nos. Department of Economics Discussion Paper 2018–12). St. Gallen: University of St. Gallen.

  22. Evans, N.J., Holmes, W.R., Trueblood, J.S. (2018). Response time data provides critical constraints on dynamic models of multi-alternative, multi-attribute choice. (Manuscript submitted for publication).

  23. Farrell, S., & Lewandowsky, S. (2018). Computational modeling of cognition and behavior. Cambridge University Press.

  24. Finkbeiner, S.D., Briscoe, A.D., Reed, R.D. (2014). Warning signals are seductive: relative contributions of color and pattern to predator avoidance and mate attraction in Heliconius butterflies. Evolution, 68, 3410–3420.

  25. Garner, W.R. (1953). An informational analysis of absolute judgments of loudness. Journal of Experimental Psychology, 46, 373–380.

  26. Gerrein, J.R., & Chechile, R.A. (1977). Storage and retrieval processes of alcohol-induced amnesia. Journal of Abnormal Psychology, 86(3), 285.

  27. Gould, S.J. (1996). The mismeasure of man. WW Norton & Company.

  28. Guan, M., Lee, M.D., Vandekerckhove, J. (2015). A hierarchical cognitive threshold model of human decision making on different length optimal stopping problems. In Dale, R. et al. (Eds.) Proceedings of the 37th annual conference of the cognitive science society. Austin: Cognitive Science Society.

  29. Hardwicke, T.E., & Ioannidis, J. (2018). Mapping the universe of registered reports. BITSS.,

  30. Heathcote, A., Brown, S.D., Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In Forstmann, B.U., & Agenmakers, E.-J. (Eds.) An introduction to model-based cognitive neuroscience (pp. 25–48): Springer.

  31. Holland, M.K., & Lockhead, G.R. (1968). Sequential effects in absolute judgments of loudness. Perception & Psychophysics, 3, 409–414.

  32. Huber, J., Payne, J.W., Puto, C. (1982). Adding asymmetrically dominated alternatives: violations of regularity and the similarity hypothesis. Journal of Consumer Research, 9(1), 90–98.

  33. Jensen, A.R. (2006). Clocking the mind: mental chronometry and individual differences. Elsevier.

  34. Kass, R.E., & Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377–395.

  35. Kerr, N.L. (1998). Harking: hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.

  36. Kılıč, A., Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2017). Models that allow us to perceive the world more accurately also allow us to remember past events more accurately via differentiation. Cognitive psychology, 92, 65–86.

  37. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B.J., Bahník, S., Bernstein, M.J., et al. (2014). Investigating variation in replicability: a ”many labs” replication project. Social Psychology, 45, 142–152.

  38. Latty, T., & Beekman, M. (2010). Irrational decision-making in an amoeboid organism: transitivity and context-dependent preferences. Proceedings of the Royal Society B: Biological Sciences, 278(1703), 307–312.

  39. Lee, M.D. (2018). Bayesian methods in cognitive modeling. In Wixted, J., & Wagenmakers, E.-J. (Eds.) The Stevens’ handbook of experimental psychology and cognitive neuroscience methodology. 4th edn., Vol. 5: Wiley.

  40. Lee, M.D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: a practical course. Cambridge University Press.

  41. Liew, S.X., Howe, P.D., Little, D.R. (2016). The appropriacy of averaging in the study of context effects. Psychonomic Bulletin & Review, 23(5), 1639–1646.

  42. Luce, R.D. (1959). Individual choice behavior: a theoretical analysis. New York: Wiley.

  43. MacCoun, R.J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Lilienfeld, S., & Waldman, I. (Eds.) Psychological science under scrutiny: recent challenges and proposed solutions (pp. 297–322): Wiley.

  44. Marsh, H.W., Morin, A.J., Parker, P.D., Kaur, G. (2014). Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110.

  45. Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H.A., van der Molen, M.W., Wagenmakers, E.-J. (2015). The effect of horizontal eye movements on free recall: a preregistered adversarial collaboration. Journal of Experimental Psychology: General, 144, e1–e15.

  46. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., et al. (2014). Promoting transparency in social science research. Science, 343, 30–31.

  47. Munafò, M. R., Nosek, B.A., Bishop, D.V., Button, K.S., Chambers, C.D., du Sert, N.P., Ioannidis, J.P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

  48. Murdock, B.B. (1960). The distinctiveness of stimuli. Psychological Review, 67, 16–31.

  49. Murdock, B.B. (1962). The serial position effect in free recall. Journal of Experimental Psychology, 64, 482–488.

  50. Myung, I.J., Forster, M.R., Browne, M.W. (2000). A special issue on model selection. Journal of Mathematical Psychology, 44, 1–2.

  51. Navarro, D.J. (in press). Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection. Computational Brain & Behavior.

  52. Noble, W.S. (2009). A quick guide to organizing computational biology projects. PLOS Computational Biology, 5(7), 1–5.

  53. Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., et al. (2015). Promoting an open research culture. Science, 348, 1422–1425.

  54. Nosek, B.A., Ebersole, C.R., DeHaven, A.C., Mellor, D.T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 201708274.

  55. Oberauer, K., Lewandowsky, S., Avh, E., Brown, G.D., Conway, A., Covan, N., et al. (2018). Benchmarks for models of short term and working memory. Psychological Bulletin.

  56. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.

  57. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

  58. Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science, 7, 528– 530.

  59. Pitt, M.A., Kim, W., Navarro, D.J., Myung, J.I. (2006). Global model analysis by parameter space partitioning. Psychological Review, 113(1), 57.

  60. Platt, J.R. (1964). Strong inference. Science, 146(3642), 347–353.

  61. Popper, K.R. (1959). The logic of scientific discovery. London: Routledge.

  62. Ratcliff, R., Schmiedek, F., McKoon, G. (2008). A diffusion model explanation of the worst performance rule for reaction time and IQ. Intelligence, 36, 10–17.

  63. Roe, R.M., Busemeyer, J.R., Townsend, J.T. (2001). Multialternative decision field theory: a dynamic connectionst model of decision making. Psychological Review, 108(2), 370.

  64. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.

  65. Rouder, J., Haaf, J.M., Snyder, H.K. (in press). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Science,

  66. Rubin, D.C., Hinton, S., Wenzel, A. (1999). The precise time course of retention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 1161–1176.

  67. Shanks, D.R., Newell, B.R., Lee, E.H., Balakrishnan, D., Ekelund, L., Cenac, Z., Moore, C. (2013). Priming intelligent behavior: an elusive phenomenon. PloS ONE, 8, e56515.

  68. Shiffrin, R.M., Lee, M.D., Kim, W.-J., Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.

  69. Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1, 337–356.

  70. Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

  71. Simonsohn, U. (2013). Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888.

  72. Simonson, I. (1989). Choice based on reasons: the case of attraction and compromise effects. Journal of Consumer Research, 16(2), 158–174.

  73. Spellman, B.A. (2015). A short (personal) future history of revolution 2.0. Perspectives on Psychological Science, 10, 886–899.

  74. Stewart, N., Brown, G.D.A., Chater, N. (2005). Absolute identification by relative judgment. Psychological Review, 112, 881–911.

  75. Sun, R. (2008). The Cambridge handbook of computational psychology. New York: Cambridge University Press.

  76. Trueblood, J.S., Brown, S.D., Heathcote, A. (2015). The fragile nature of contextual preference reversals: reply to tsetsos, chater, and usher (2015). Psychological Review, 122(4), 848–853.

  77. Turner, B.M., Schley, D.R., Muller, C., Tsetsos, K. (2017). Competing theories of multialternative, multiattribute preferential choice. Psychological Review, 125(3), 329–362.

  78. Tversky, A. (1972). Elimination by aspects: a theory of choice. Psychological Review, 79(4), 281.

  79. Unsworth, N., Redick, T.S., Lakey, C.E., Young, D.L. (2010). Lapses in sustained attention and their relation to executive control and fluid abilities: an individual differences investigation. Intelligence, 38, 111–122.

  80. Usher, M., & McClelland, J.L. (2004). Loss aversion and inhibition in dynamical models of multialternative choice. Psychological Review, 111(3), 757.

  81. Vandekerckhove, J., Matzke, D., Wagenmakers, E.-J. (2015). Model comparison and the principle of parsimony. In Busemeyer, J. R., Wang, Z., Townsend, J. T., Eidels, A. (Eds.) Oxford handbook of computational and mathematical psychology (pp. 300–317). New York: Oxford University Press.

  82. Vanpaemel, W. (2010). Prior sensitivity in theory testing: an apologia for the Bayes factor. Journal of Mathematical Psychology, 54, 491–498.

  83. Vanpaemel, W., Vermorgen, M., Deriemaecker, L., Storms, G. (2015). Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra, 1, 3.

  84. Voss, A., Rothermund, K., Voss, J. (2004). Interpreting the parameters of the diffusion model: an empirical validation. Memory & Cognition, 32, 1206–1220.

  85. Wagenmakers, E.-J. (2012). A year of horrors. De Psychonoom, 27, 12–13.

  86. Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11(1), 192–196.

  87. Wagenmakers, E.-J., Grünwald, P., Steyvers, M. (2006). Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149–166.

  88. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L., Kievit, R.A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

  89. Watts, D.J. (2017). Should social science be more solution-oriented? Nature Human Behaviour, 1, 1–5.

  90. Wicherts, J.M., Borsboom, D., Kats, J., Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726–728.

  91. Zhang, S., & Lee, M.D. (2010). Optimal experimental design for a class of bandit problems. Journal of Mathematical Psychology, 54, 499–508.

  92. Zwaan, R.A., Etz, A., Lucas, R.E., Donnellan, M.B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41.

Download references


This article is the product of the Workshop on Robust Social Science held in St. Petersburg, FL, in June 2018.

Funding Information

The workshop was made possible by generous funding from the National Science Foundation (grant no. BCS-1754205) to Joachim Vandekerckhove and Michael Lee of the University of California, Irvine. Alexander Etz was supported by NSF GRFP no. DGE-1321846. Berna Devezer was supported by NIGMS of the NIH under award no. P20GM104420. Dora Matzke was supported by a Veni grant (no. 451-15-010) from the Netherlands Organization of Scientific Research (NWO). Jennifer Trueblood was supported by NSF no. SES-1556325.

Author information

Correspondence to Joachim Vandekerckhove.

Ethics declarations


The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the National Science Foundation and Netherlands Organization of Scientific Research.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, M.D., Criss, A.H., Devezer, B. et al. Robust Modeling in Cognitive Science. Comput Brain Behav 2, 141–153 (2019).

Download citation


  • Robustness
  • Cognitive Modeling
  • Open Science
  • Reproducibility