Skip to main content

Robust Modeling in Cognitive Science

Abstract

In an attempt to increase the reliability of empirical findings, psychological scientists have recently proposed a number of changes in the practice of experimental psychology. Most current reform efforts have focused on the analysis of data and the reporting of findings for empirical studies. However, a large contingent of psychologists build models that explain psychological processes and test psychological theories using formal psychological models. Some, but not all, recommendations borne out of the broader reform movement bear upon the practice of behavioral or cognitive modeling. In this article, we consider which aspects of the current reform movement are relevant to psychological modelers, and we propose a number of techniques and practices aimed at making psychological modeling more transparent, trusted, and robust.

This is a preview of subscription content, access via your institution.

Notes

  1. 1.

    Note that registered reports involve more than preregistration: They also involve a journal’s guarantee that a paper will be published regardless of how the data turn out.

  2. 2.

    https://www.kaggle.com/competitions

  3. 3.

    https://cpc-18.com/

References

  1. Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école Américaine. Econometrica, 21, 503–546.

    Article  Google Scholar 

  2. Allais, M. (1979). The foundations of a positive theory of choice involving risk and a criticism of the postulates and axioms of the American School. In Allais, M., & Hagen, O. (Eds.) Expected utility hypothesis and the Allais paradox (pp. 27–145). Dordrecht: Riedel.

  3. Alogna, V., Attaya, M.K., Aucoin, P., Bahník, Š., Birch, S., Birt, A.R., et al. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.

    Article  Google Scholar 

  4. Batchelder, W.H. (2010). Cognitive psychometrics: using multinomial processing tree models as measurement tools. In Embretson, S. (Ed.) Measuring psychological constructs: advances in model-based approaches. Washington, DC: American Psychological Association Books.

  5. Baumeister, A.A., & Kellas, G. (1968). Reaction time and mental retardation. International Review of Research in Mental Retardation, 3, 163–193.

    Article  Google Scholar 

  6. Baumgaertner, B., Devezer, B., Buzbas, E.O., Nardin, L.G. (2018). A model-centric analysis of openness, replication, and reproducibility. arXiv:1811.04525.

  7. Beck, K. (2003). Test-driven development: by example. Addison-Wesley Professional.

  8. Bell, R.M., Koren, Y., Volinsky, C. (2010). All together now: a perspective on the Netflix prize. Chance, 23, 24–29.

    Article  Google Scholar 

  9. Birnbaum, M.H., & Quispe-Torreblanca, E.G. (2018). TEMAP2.R: true and error model analysis program in R. Judgment and Decision Making, 13, 428–440.

    Google Scholar 

  10. Bones, A.K. (2012). We knew the future all along: scientific hypothesizing is much more accurate than other forms of precognition—a satire in one part. Perspectives on Psychological Science, 7, 307–309.

    Article  Google Scholar 

  11. Brown, G.D.A., Neath, I., Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114(1), 539–576.

    Article  Google Scholar 

  12. Brown, S.D., Marley, A.A.J., Donkin, C., Heathcote, A. (2008). An integrated model of choices and response times in absolute identification. Psychological Review, 115, 396–425.

    Article  Google Scholar 

  13. Busemeyer, J.R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171–189.

    Article  Google Scholar 

  14. Cavagnaro, D.R., Pitt, M.A., Gonzalez, R., Myung, I.J. (2013). Discriminating among probability weighting functions with adaptive design optimization. Journal of Risk and Uncertainty, 47, 255–289.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Chambers, C.D. (2013). Registered reports: a new publishing initiative at cortex. Cortex, 49, 609–610.

    Article  Google Scholar 

  16. Chambers, C.D., Dienes, Z., McIntosh, R.D., Rotshtein, P., Willmes, K. (2015). Registered reports: realigning incentives in scientific publishing. Cortex, 66, A1–A2.

    Article  Google Scholar 

  17. Cook, S.R., Gelman, A., Rubin, D.B. (2006). Validation of software for Bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics, 15(3), 675–692.

    Article  Google Scholar 

  18. Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2011). Output interference in recognition memory. Journal of Memory and Language, 64(4), 316–326.

    Article  Google Scholar 

  19. Dutilh, G., Vandekerckhove, J., Ly, A., Matzke, D., Pedroni, A., Frey, R., Wagenmakers, E.-J. (2017). A test of the diffusion model explanation for the worst performance rule using preregistration and blinding. Attention, Perception, & Psychophysics, 79, 713–725.

    Article  Google Scholar 

  20. Dutilh, G., Annis, J., Brown, S.D., Cassey, P., Evans, N.J., Grasman, R.P.P.P., Donkin, C. (2018). The quality of response time data inference: a blinded, collaborative assessment of the validity of cognitive models. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-017-1417-2.

  21. Epper, T., & Fehr-Duda, H. (2018). The missing link: unifying risk taking and time discounting (Tech. Rep. Nos. Department of Economics Discussion Paper 2018–12). St. Gallen: University of St. Gallen.

    Google Scholar 

  22. Evans, N.J., Holmes, W.R., Trueblood, J.S. (2018). Response time data provides critical constraints on dynamic models of multi-alternative, multi-attribute choice. https://osf.io/h7e6v/ (Manuscript submitted for publication).

  23. Farrell, S., & Lewandowsky, S. (2018). Computational modeling of cognition and behavior. Cambridge University Press.

  24. Finkbeiner, S.D., Briscoe, A.D., Reed, R.D. (2014). Warning signals are seductive: relative contributions of color and pattern to predator avoidance and mate attraction in Heliconius butterflies. Evolution, 68, 3410–3420.

    Article  Google Scholar 

  25. Garner, W.R. (1953). An informational analysis of absolute judgments of loudness. Journal of Experimental Psychology, 46, 373–380.

    Article  Google Scholar 

  26. Gerrein, J.R., & Chechile, R.A. (1977). Storage and retrieval processes of alcohol-induced amnesia. Journal of Abnormal Psychology, 86(3), 285.

    Article  Google Scholar 

  27. Gould, S.J. (1996). The mismeasure of man. WW Norton & Company.

  28. Guan, M., Lee, M.D., Vandekerckhove, J. (2015). A hierarchical cognitive threshold model of human decision making on different length optimal stopping problems. In Dale, R. et al. (Eds.) Proceedings of the 37th annual conference of the cognitive science society. Austin: Cognitive Science Society.

  29. Hardwicke, T.E., & Ioannidis, J. (2018). Mapping the universe of registered reports. BITSS. https://doi.org/10.17605/OSF.IO/FZPCY, http://osf.io/preprints/bitss/fzpcy.

  30. Heathcote, A., Brown, S.D., Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In Forstmann, B.U., & Agenmakers, E.-J. (Eds.) An introduction to model-based cognitive neuroscience (pp. 25–48): Springer.

  31. Holland, M.K., & Lockhead, G.R. (1968). Sequential effects in absolute judgments of loudness. Perception & Psychophysics, 3, 409–414.

    Article  Google Scholar 

  32. Huber, J., Payne, J.W., Puto, C. (1982). Adding asymmetrically dominated alternatives: violations of regularity and the similarity hypothesis. Journal of Consumer Research, 9(1), 90–98.

    Article  Google Scholar 

  33. Jensen, A.R. (2006). Clocking the mind: mental chronometry and individual differences. Elsevier.

  34. Kass, R.E., & Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377–395.

    Google Scholar 

  35. Kerr, N.L. (1998). Harking: hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.

    Article  Google Scholar 

  36. Kılıč, A., Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2017). Models that allow us to perceive the world more accurately also allow us to remember past events more accurately via differentiation. Cognitive psychology, 92, 65–86.

    Article  Google Scholar 

  37. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B.J., Bahník, S., Bernstein, M.J., et al. (2014). Investigating variation in replicability: a ”many labs” replication project. Social Psychology, 45, 142–152.

    Article  Google Scholar 

  38. Latty, T., & Beekman, M. (2010). Irrational decision-making in an amoeboid organism: transitivity and context-dependent preferences. Proceedings of the Royal Society B: Biological Sciences, 278(1703), 307–312.

    Article  Google Scholar 

  39. Lee, M.D. (2018). Bayesian methods in cognitive modeling. In Wixted, J., & Wagenmakers, E.-J. (Eds.) The Stevens’ handbook of experimental psychology and cognitive neuroscience methodology. 4th edn., Vol. 5: Wiley.

  40. Lee, M.D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: a practical course. Cambridge University Press.

  41. Liew, S.X., Howe, P.D., Little, D.R. (2016). The appropriacy of averaging in the study of context effects. Psychonomic Bulletin & Review, 23(5), 1639–1646.

    Article  Google Scholar 

  42. Luce, R.D. (1959). Individual choice behavior: a theoretical analysis. New York: Wiley.

    Google Scholar 

  43. MacCoun, R.J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Lilienfeld, S., & Waldman, I. (Eds.) Psychological science under scrutiny: recent challenges and proposed solutions (pp. 297–322): Wiley.

  44. Marsh, H.W., Morin, A.J., Parker, P.D., Kaur, G. (2014). Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110.

    Article  Google Scholar 

  45. Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H.A., van der Molen, M.W., Wagenmakers, E.-J. (2015). The effect of horizontal eye movements on free recall: a preregistered adversarial collaboration. Journal of Experimental Psychology: General, 144, e1–e15.

    Article  Google Scholar 

  46. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., et al. (2014). Promoting transparency in social science research. Science, 343, 30–31.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Munafò, M. R., Nosek, B.A., Bishop, D.V., Button, K.S., Chambers, C.D., du Sert, N.P., Ioannidis, J.P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

    Article  Google Scholar 

  48. Murdock, B.B. (1960). The distinctiveness of stimuli. Psychological Review, 67, 16–31.

    Article  Google Scholar 

  49. Murdock, B.B. (1962). The serial position effect in free recall. Journal of Experimental Psychology, 64, 482–488.

    Article  Google Scholar 

  50. Myung, I.J., Forster, M.R., Browne, M.W. (2000). A special issue on model selection. Journal of Mathematical Psychology, 44, 1–2.

    Article  Google Scholar 

  51. Navarro, D.J. (in press). Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection. Computational Brain & Behavior.

  52. Noble, W.S. (2009). A quick guide to organizing computational biology projects. PLOS Computational Biology, 5(7), 1–5. https://doi.org/10.1371/journal.pcbi.1000424.

    Article  Google Scholar 

  53. Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., et al. (2015). Promoting an open research culture. Science, 348, 1422–1425.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Nosek, B.A., Ebersole, C.R., DeHaven, A.C., Mellor, D.T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 201708274.

  55. Oberauer, K., Lewandowsky, S., Avh, E., Brown, G.D., Conway, A., Covan, N., et al. (2018). Benchmarks for models of short term and working memory. Psychological Bulletin.

  56. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.

    Article  Google Scholar 

  57. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

    Article  Google Scholar 

  58. Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science, 7, 528– 530.

    Article  Google Scholar 

  59. Pitt, M.A., Kim, W., Navarro, D.J., Myung, J.I. (2006). Global model analysis by parameter space partitioning. Psychological Review, 113(1), 57.

    Article  Google Scholar 

  60. Platt, J.R. (1964). Strong inference. Science, 146(3642), 347–353.

    Article  Google Scholar 

  61. Popper, K.R. (1959). The logic of scientific discovery. London: Routledge.

    Google Scholar 

  62. Ratcliff, R., Schmiedek, F., McKoon, G. (2008). A diffusion model explanation of the worst performance rule for reaction time and IQ. Intelligence, 36, 10–17.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Roe, R.M., Busemeyer, J.R., Townsend, J.T. (2001). Multialternative decision field theory: a dynamic connectionst model of decision making. Psychological Review, 108(2), 370.

    Article  Google Scholar 

  64. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.

    Article  Google Scholar 

  65. Rouder, J., Haaf, J.M., Snyder, H.K. (in press). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Science, https://psyarxiv.com/gxcy5/.

  66. Rubin, D.C., Hinton, S., Wenzel, A. (1999). The precise time course of retention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 1161–1176.

    Google Scholar 

  67. Shanks, D.R., Newell, B.R., Lee, E.H., Balakrishnan, D., Ekelund, L., Cenac, Z., Moore, C. (2013). Priming intelligent behavior: an elusive phenomenon. PloS ONE, 8, e56515.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Shiffrin, R.M., Lee, M.D., Kim, W.-J., Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.

    Article  Google Scholar 

  69. Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1, 337–356.

    Article  Google Scholar 

  70. Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  Google Scholar 

  71. Simonsohn, U. (2013). Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888.

    Article  Google Scholar 

  72. Simonson, I. (1989). Choice based on reasons: the case of attraction and compromise effects. Journal of Consumer Research, 16(2), 158–174.

    Article  Google Scholar 

  73. Spellman, B.A. (2015). A short (personal) future history of revolution 2.0. Perspectives on Psychological Science, 10, 886–899.

    Article  Google Scholar 

  74. Stewart, N., Brown, G.D.A., Chater, N. (2005). Absolute identification by relative judgment. Psychological Review, 112, 881–911.

    Article  Google Scholar 

  75. Sun, R. (2008). The Cambridge handbook of computational psychology. New York: Cambridge University Press.

    Book  Google Scholar 

  76. Trueblood, J.S., Brown, S.D., Heathcote, A. (2015). The fragile nature of contextual preference reversals: reply to tsetsos, chater, and usher (2015). Psychological Review, 122(4), 848–853. https://doi.org/10.1037/a0039656.

    Article  PubMed  Google Scholar 

  77. Turner, B.M., Schley, D.R., Muller, C., Tsetsos, K. (2017). Competing theories of multialternative, multiattribute preferential choice. Psychological Review, 125(3), 329–362. https://doi.org/10.1037/rev0000089.

    Article  PubMed  Google Scholar 

  78. Tversky, A. (1972). Elimination by aspects: a theory of choice. Psychological Review, 79(4), 281.

    Article  Google Scholar 

  79. Unsworth, N., Redick, T.S., Lakey, C.E., Young, D.L. (2010). Lapses in sustained attention and their relation to executive control and fluid abilities: an individual differences investigation. Intelligence, 38, 111–122.

    Article  Google Scholar 

  80. Usher, M., & McClelland, J.L. (2004). Loss aversion and inhibition in dynamical models of multialternative choice. Psychological Review, 111(3), 757.

    Article  Google Scholar 

  81. Vandekerckhove, J., Matzke, D., Wagenmakers, E.-J. (2015). Model comparison and the principle of parsimony. In Busemeyer, J. R., Wang, Z., Townsend, J. T., Eidels, A. (Eds.) Oxford handbook of computational and mathematical psychology (pp. 300–317). New York: Oxford University Press.

  82. Vanpaemel, W. (2010). Prior sensitivity in theory testing: an apologia for the Bayes factor. Journal of Mathematical Psychology, 54, 491–498.

    Article  Google Scholar 

  83. Vanpaemel, W., Vermorgen, M., Deriemaecker, L., Storms, G. (2015). Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra, 1, 3. https://doi.org/10.1525/collabra.13.

    Article  Google Scholar 

  84. Voss, A., Rothermund, K., Voss, J. (2004). Interpreting the parameters of the diffusion model: an empirical validation. Memory & Cognition, 32, 1206–1220.

    Article  Google Scholar 

  85. Wagenmakers, E.-J. (2012). A year of horrors. De Psychonoom, 27, 12–13.

    Google Scholar 

  86. Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11(1), 192–196. https://doi.org/10.3758/BF03206482.

    Article  Google Scholar 

  87. Wagenmakers, E.-J., Grünwald, P., Steyvers, M. (2006). Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149–166.

    Article  Google Scholar 

  88. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L., Kievit, R.A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

    Article  Google Scholar 

  89. Watts, D.J. (2017). Should social science be more solution-oriented? Nature Human Behaviour, 1, 1–5.

    Article  Google Scholar 

  90. Wicherts, J.M., Borsboom, D., Kats, J., Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726–728.

    Article  Google Scholar 

  91. Zhang, S., & Lee, M.D. (2010). Optimal experimental design for a class of bandit problems. Journal of Mathematical Psychology, 54, 499–508.

    Article  Google Scholar 

  92. Zwaan, R.A., Etz, A., Lucas, R.E., Donnellan, M.B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41.

Download references

Acknowledgments

This article is the product of the Workshop on Robust Social Science held in St. Petersburg, FL, in June 2018.

Funding

The workshop was made possible by generous funding from the National Science Foundation (grant no. BCS-1754205) to Joachim Vandekerckhove and Michael Lee of the University of California, Irvine. Alexander Etz was supported by NSF GRFP no. DGE-1321846. Berna Devezer was supported by NIGMS of the NIH under award no. P20GM104420. Dora Matzke was supported by a Veni grant (no. 451-15-010) from the Netherlands Organization of Scientific Research (NWO). Jennifer Trueblood was supported by NSF no. SES-1556325.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Joachim Vandekerckhove.

Ethics declarations

Disclaimer

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the National Science Foundation and Netherlands Organization of Scientific Research.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, M.D., Criss, A.H., Devezer, B. et al. Robust Modeling in Cognitive Science. Comput Brain Behav 2, 141–153 (2019). https://doi.org/10.1007/s42113-019-00029-y

Download citation

Keywords

  • Robustness
  • Cognitive Modeling
  • Open Science
  • Reproducibility