An Introduction to Good Practices in Cognitive Modeling

  • Andrew HeathcoteEmail author
  • Scott D. Brown
  • Eric-Jan Wagenmakers


Cognitive modeling can provide important insights into the underlying causes of behavior, but the validity of those insights rests on careful model development and checking. We provide guidelines on five important aspects of the practice of cognitive modeling: parameter recovery, testing selective influence of experimental manipulations on model parameters, quantifying uncertainty in parameter estimates, testing and displaying model fit, and selecting among different model parameterizations and types of models. Each aspect is illustrated with examples.


Cognition Theory Quantitative Model Simulation study Parameter estimation Model selection 


  1. 1.
    Brown SD, Heathcote AJ (2008) The simplest complete model of choice reaction time: linear ballistic accumulation. Cognit Psychol 57:153–178Google Scholar
  2. 2.
    Ratcliff R, Thapar A, McKoon G (2006) Aging, practice, and perceptual tasks: a diffusion model analysis. Psychol Aging 21:353–371Google Scholar
  3. 3.
    Riefer DM, Knapp BR, Batchelder WH, Bamber D, Manifold V. (2002) Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychol Assess 14:184–201Google Scholar
  4. 4.
    Mulder M., Wagenmakers EJ, Ratcliff R, Boekel W, Forstmann BU (2012) Bias in the brain: a diffusion model analysis of prior probability and potential payoff. J Neurosci 32:2335–2343Google Scholar
  5. 5.
    Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85:59–108Google Scholar
  6. 6.
    Ratcliff R, Gomez P, McKoon G (2004) Diffusion model account of lexical decision. Psychol Rev 111:159–182Google Scholar
  7. 7.
    Ratcliff R, Starns JJ (2009) Modeling confidence and response time in recognition memory. Psychol Rev 116:59–83Google Scholar
  8. 8.
    Ratcliff R, van Dongen HPA (2009) Sleep deprivation affects multiple distinct cognitive processes. Psychon Bull Rev 16:742–751Google Scholar
  9. 9.
    Smith PL, Ratcliff R (2004) The psychology and neurobiology of simple decisions. Trends Neurosci 27:161–168Google Scholar
  10. 10.
    Ratcliff R, Tuerlinckx F (2002) Estimating parameters of the diffusion model: approaches to dealing with contaminant reaction times and parameter variability. Psychon Bull Rev 9:438–481Google Scholar
  11. 11.
    Wagenmakers EJ, van der Maas HJL, Grasman RPPP (2007) An EZ–diffusion model for response time and accuracy. Psychon Bull Rev 14:3–22Google Scholar
  12. 12.
    Donkin C, Brown S, Heathcote A, Wagenmakers EJ (2011) Different models but the same conclusions about psychological processes? Psychon Bull Rev 18:61–69Google Scholar
  13. 13.
    Lejuez CW, Read JP, Kahler CW, Richards JB, Ramsey SE, Stuart GL, Strong DR, Brown RA (2002) Evaluation of a behavioral measure of risk taking: the balloon analogue risk task (BART). J Exp Psychol Appl 8:75–84Google Scholar
  14. 14.
    Wallsten TS, Pleskac TJ, Lejuez CW (2005) Modeling behavior in a clinically diagnostic sequential risk–taking task. Psychol Rev 112:862–880Google Scholar
  15. 15.
    van Ravenzwaaij D, Dutilh G, Wagenmakers EJ (2011) Cognitive model decomposition of the BART: assessment and application. J Math Psychol 55:94–105Google Scholar
  16. 16.
    Hintze JL, Nelson RD (1998) Violin plots: a box plot–density trace synergism. Am Stat 52:181–184Google Scholar
  17. 17.
    Rolison JJ, Hanoch Y, Wood S (2012) Risky decision making in younger and older adults: the role of learning.Psychol Aging 27:129–140Google Scholar
  18. 18.
    Parks TE (1966) Signal-detectability theory of recognition-memory performance. Psychol Rev 73(1):44–58Google Scholar
  19. 19.
    Macmillan NA, Creelman CD (2005) Detection theory: a user’s guide, 2nd edn. Erlbaum, MahwahGoogle Scholar
  20. 20.
    Wixted JT, Stretch V (2000) The case against a criterion-shift account of false memory. Psychol Rev 107:368–376Google Scholar
  21. 21.
    Ratcliff R,Rouder JN (1998) Modeling response times for two-choice decisions. Psychol Sci 9:347–356Google Scholar
  22. 22.
    Voss A, Rothermund K, Voss J (2004) Interpreting the parameters of the diffusion model: an empirical validation. Mem Cognit 32:1206–1220Google Scholar
  23. 23.
    Ho TC, Brown S, Serences JT (2009) Domain general mechanisms of perceptual decision making in human cortex. J Neurosci 29:8675–8687Google Scholar
  24. 24.
    Schwarz G (1978) Estimating the dimension of a model. Annals Stat 6:461–464Google Scholar
  25. 25.
    Lee M, Vandekerckhove J, Navarro DJ, Tuerlinckx F (2007) Presentation at the 40th Annual Meeting of the Society for Mathematical Psychology, Irvine, USA, July 2007Google Scholar
  26. 26.
    Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variance and dualprocess explanations of zROC slopes with response time data and the diffusion model. Cogniti Psychol 64:1–34Google Scholar
  27. 27.
    Rae B, Heathcote C, Donkin A, Averell L, Brown SD (in press) The hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem CognGoogle Scholar
  28. 28.
    Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New YorkGoogle Scholar
  29. 29.
    Wagenaar WA, Boer JPA (1987) Misleading postevent information: testing parameterized models of integration in memory. Acta Psychol 66:291–306Google Scholar
  30. 30.
    Vandekerckhove J, Matzke D, Wagenmakers EJ 2013) In Busemeyer J, Townsend J, Wang ZJ, Eidels A (eds) Oxford handbook of computational and mathematical psychology. Oxford University PressGoogle Scholar
  31. 31.
    Lee MD, Wagenmakers EJ (in press) Bayesian modeling for cognitive science: a practical course. Cambridge University PressGoogle Scholar
  32. 32.
    Jeffreys H (1961) Theory of probability, 3rd edn. Oxford University Press, OxfordGoogle Scholar
  33. 33.
    Anscombe FJ (1973) Graphs in statistical analysis. Am Stat 27:17–21Google Scholar
  34. 34.
    McCullagh PN, Nelder JA (1983) Generalized linear models. Chapman & Hall, LondonGoogle Scholar
  35. 35.
    Raftery AE (1995) Bayesian model selection in social research. Sociol Methodol 25:111–164Google Scholar
  36. 36.
    Averell L, Heathcote A (2011) The form of the forgetting curve and the fate of memories. J Math Psychol 55:25–35Google Scholar
  37. 37.
    Brown S, Heathcote A (2003) Averaging learning curves across and within participants. Behav Res Methods Instrum Comput 35:11–21Google Scholar
  38. 38.
    Heathcote A, Brown S, Mewhort DJK (2000) The power law repealed: the case for an exponential law of practice. Psychon Bull Rev 7:185–207Google Scholar
  39. 39.
    Pachella RG (1974) The interpretation of reaction time in information–processing research. In: Kantowitz BH (ed) Human information processing: tutorials in performance and cognition. Lawrence Erlbaum Associates, Hillsdale, pp 41–82Google Scholar
  40. 40.
    Audley RJ, Pike AR (1965) Some alternative models of stochastic choice. Br J Math Stat Psychol 207–225Google Scholar
  41. 41.
    Vickers D, Caudrey D, Willson R (1971) Discriminating between the frequency of occurrence of two alternative events. ACTPSY 35:151–172Google Scholar
  42. 42.
    Ratcliff R, Thapar A, McKoon G (2001) The effects of aging on reaction time in a signal detection task. Psychol Aging 16:323Google Scholar
  43. 43.
    Wagenmakers EJ, Ratcliff R, Gómez P, McKoon G (2008) A diffusion model account of criterion shifts in the lexical decision task. J Memory Lang 58:140–159Google Scholar
  44. 44.
    Rae B, Heathcote A, Donkin C, Averell L, Brown SD (2013) The Hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem Cogn 1–45Google Scholar
  45. 45.
    Wagenmakers EJ, Ratcliff R, Gómez P, Iverson GJ (2004) Assessing model mimicry using the parametric bootstrap. J Math Psychol 48:28–50Google Scholar
  46. 46.
    Ratcliff R, Rouder JN (1998) Modeling response times for two–choice decisions. Psychol Sci 9:347–356Google Scholar
  47. 47.
    Morey RD (2008) Confidence intervals from normalized data: a correction to Cousineau. Tutor Quant Methods Psychol 4:61–64Google Scholar
  48. 48.
    Heathcote A, Love J (2012) Linear deterministic accumulator models of simple choice. Front Psychol 3Google Scholar
  49. 49.
    Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variability and dual-process explanations of zroc slopes with response time data and the diffusion model. Cognit Psychol 64:1–34Google Scholar
  50. 50.
    Cleveland WS (1993) Visualizing data. Hobart Press, New JerseyGoogle Scholar
  51. 51.
    Heathcote A, Brown S, Mewhort DJK (2002) Quantile maximum likelihood estimation of response time distributions. Psychon Bull Rev 9(2):394–401Google Scholar
  52. 52.
    Heathcote A, Brown S (2004) A theoretical basis for QML. Psychon Bull Rev 11:577–578Google Scholar
  53. 53.
    Speckman PL, Rouder JN (2004) A comment on Heathcote, Brown, and Mewhort’s QMLE method for response time distributions. Psychon Bull Rev 11:574–576Google Scholar
  54. 54.
    Kass RE, Raftery AE (1995) Bayes factors. J Am Stat Assoc 90:773–795Google Scholar
  55. 55.
    Lodewyckx T, Kim W, Lee MD, Tuerlinckx F, Kuppens P, Wagenmakers EJ (2011) A tutorial on Bayes factor estimation with the product space method. J Math Psychol 55:331–347Google Scholar
  56. 56.
    Shiffrin R, Lee M, Kim W, Wagenmakers EJ (2008) A survey of model evaluation approaches with a tutorial on Hierarchical Bayesian Methods. Cognit Sci A Multidiscip J 32:1248–1284Google Scholar
  57. 57.
    Spiegelhalter DJ, Best NG, Carlin BP, Van Der Linde A (2002) Bayesian measures of model complexity and fit. J Royal Stat Soc Series B (Stat Methodol) 64:583–639Google Scholar
  58. 58.
    Ando T (2007) Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models. Biometrika 94:443–458Google Scholar
  59. 59.
    Myung IJ, Pitt MA (1997) Applying Occam’s razor in modeling cognition: a Bayesian approach. Psych Bull Rev 4:79–95Google Scholar
  60. 60.
    Burnham KP, Anderson DR (2004) Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res 33:261–304Google Scholar
  61. 61.
    Donkin C, Brown S, Heathcote A (2011) Drawing conclusions from choice response time models: a tutorial using the linear ballistic accumulator. J Math Psychol 55:140–151Google Scholar
  62. 62.
    Turner BM, Sederberg PB, Brown SD, Steyvers M (2013) A method for efficiently sampling from distributions with correlated dimensions. Psychol Methods 18:368–384Google Scholar
  63. 63.
    Farrell S, Wagenmakers EJ, Ratcliff R (2006) 1/f noise in human cognition: is it ubiquitous, and what does it mean? Psych Bull Rev 13:737–741Google Scholar
  64. 64.
    Gilden DL (1997) Fluctuations in the time required for elementary decisions. Psychol Sci 8:296–301Google Scholar
  65. 65.
    Box GEP (1979) Robustness in the strategy of scientific model building. In: Launer RL, Wilkinson GN (ed) Robustness in statistics: proceedings of a workshop. Academic, New York, pp 201–236Google Scholar
  66. 66.
    Lewandowsky S, Farrell S (2010) Computational modeling in cognition: principles and practice. Sage, Thousand OaksGoogle Scholar
  67. 67.
    Busemeyer JR, Diederich A (2010) Cognitive modeling. Sage, Thousand OaksGoogle Scholar
  68. 68.
    Hunt E (2006) The mathematics of behavior. Cambridge University Press, CambridgeGoogle Scholar
  69. 69.
    Polk TA, Seifert CM (eds) (2002) Cognitive modeling. MIT Press, CambridgeGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2015

Authors and Affiliations

  • Andrew Heathcote
    • 1
    Email author
  • Scott D. Brown
    • 1
  • Eric-Jan Wagenmakers
    • 2
  1. 1.School of PsychologyUniversity of Newcastle, University AvenueCallaghanAustralia
  2. 2.Department of Psychological MethodsUniversity of AmsterdamWeesperplein 4The Netherlands

Personalised recommendations