Abstract
Cognitive modeling can provide important insights into the underlying causes of behavior, but the validity of those insights rests on careful model development and checking. We provide guidelines on five important aspects of the practice of cognitive modeling: parameter recovery, testing selective influence of experimental manipulations on model parameters, quantifying uncertainty in parameter estimates, testing and displaying model fit, and selecting among different model parameterizations and types of models. Each aspect is illustrated with examples.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Extensive details are reported here: http://www.donvanravenzwaaij.com/Papers_files/BART_Appendix.pdf.
- 2.
With only 90 trials—the standard number—parameter recovery was very poor.
- 3.
The memory for the traffic light was later assessed by reminding participants that there was a traffic light at the intersection, and asking them to indicate its color.
- 4.
Wagenaar and Boer put forward a similar conclusion, albeit not formalized within a Bayesian framework.
- 5.
The absolute value of the deviance depends on the measurement units for time and so only relative values of deviance are meaningful. Deviance is on an exponential scale, and as a rule of thumb a difference less than 3 is negligible and and difference greater than 10 indicates a strong difference.
- 6.
Note that deviance can be summed over participants, as can AIC, but BIC cannot, due to the nonlinear \(\log(n)\) term in its complexity penalty. Instead the aggregate BIC is calculated from the deviance, number of parameters and sample size summed over participants
References
Brown SD, Heathcote AJ (2008) The simplest complete model of choice reaction time: linear ballistic accumulation. Cognit Psychol 57:153–178
Ratcliff R, Thapar A, McKoon G (2006) Aging, practice, and perceptual tasks: a diffusion model analysis. Psychol Aging 21:353–371
Riefer DM, Knapp BR, Batchelder WH, Bamber D, Manifold V. (2002) Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychol Assess 14:184–201
Mulder M., Wagenmakers EJ, Ratcliff R, Boekel W, Forstmann BU (2012) Bias in the brain: a diffusion model analysis of prior probability and potential payoff. J Neurosci 32:2335–2343
Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85:59–108
Ratcliff R, Gomez P, McKoon G (2004) Diffusion model account of lexical decision. Psychol Rev 111:159–182
Ratcliff R, Starns JJ (2009) Modeling confidence and response time in recognition memory. Psychol Rev 116:59–83
Ratcliff R, van Dongen HPA (2009) Sleep deprivation affects multiple distinct cognitive processes. Psychon Bull Rev 16:742–751
Smith PL, Ratcliff R (2004) The psychology and neurobiology of simple decisions. Trends Neurosci 27:161–168
Ratcliff R, Tuerlinckx F (2002) Estimating parameters of the diffusion model: approaches to dealing with contaminant reaction times and parameter variability. Psychon Bull Rev 9:438–481
Wagenmakers EJ, van der Maas HJL, Grasman RPPP (2007) An EZ–diffusion model for response time and accuracy. Psychon Bull Rev 14:3–22
Donkin C, Brown S, Heathcote A, Wagenmakers EJ (2011) Different models but the same conclusions about psychological processes? Psychon Bull Rev 18:61–69
Lejuez CW, Read JP, Kahler CW, Richards JB, Ramsey SE, Stuart GL, Strong DR, Brown RA (2002) Evaluation of a behavioral measure of risk taking: the balloon analogue risk task (BART). J Exp Psychol Appl 8:75–84
Wallsten TS, Pleskac TJ, Lejuez CW (2005) Modeling behavior in a clinically diagnostic sequential risk–taking task. Psychol Rev 112:862–880
van Ravenzwaaij D, Dutilh G, Wagenmakers EJ (2011) Cognitive model decomposition of the BART: assessment and application. J Math Psychol 55:94–105
Hintze JL, Nelson RD (1998) Violin plots: a box plot–density trace synergism. Am Stat 52:181–184
Rolison JJ, Hanoch Y, Wood S (2012) Risky decision making in younger and older adults: the role of learning.Psychol Aging 27:129–140
Parks TE (1966) Signal-detectability theory of recognition-memory performance. Psychol Rev 73(1):44–58
Macmillan NA, Creelman CD (2005) Detection theory: a user’s guide, 2nd edn. Erlbaum, Mahwah
Wixted JT, Stretch V (2000) The case against a criterion-shift account of false memory. Psychol Rev 107:368–376
Ratcliff R,Rouder JN (1998) Modeling response times for two-choice decisions. Psychol Sci 9:347–356
Voss A, Rothermund K, Voss J (2004) Interpreting the parameters of the diffusion model: an empirical validation. Mem Cognit 32:1206–1220
Ho TC, Brown S, Serences JT (2009) Domain general mechanisms of perceptual decision making in human cortex. J Neurosci 29:8675–8687
Schwarz G (1978) Estimating the dimension of a model. Annals Stat 6:461–464
Lee M, Vandekerckhove J, Navarro DJ, Tuerlinckx F (2007) Presentation at the 40th Annual Meeting of the Society for Mathematical Psychology, Irvine, USA, July 2007
Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variance and dualprocess explanations of zROC slopes with response time data and the diffusion model. Cogniti Psychol 64:1–34
Rae B, Heathcote C, Donkin A, Averell L, Brown SD (in press) The hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem Cogn
Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York
Wagenaar WA, Boer JPA (1987) Misleading postevent information: testing parameterized models of integration in memory. Acta Psychol 66:291–306
Vandekerckhove J, Matzke D, Wagenmakers EJ 2013) In Busemeyer J, Townsend J, Wang ZJ, Eidels A (eds) Oxford handbook of computational and mathematical psychology. Oxford University Press
Lee MD, Wagenmakers EJ (in press) Bayesian modeling for cognitive science: a practical course. Cambridge University Press
Jeffreys H (1961) Theory of probability, 3rd edn. Oxford University Press, Oxford
Anscombe FJ (1973) Graphs in statistical analysis. Am Stat 27:17–21
McCullagh PN, Nelder JA (1983) Generalized linear models. Chapman & Hall, London
Raftery AE (1995) Bayesian model selection in social research. Sociol Methodol 25:111–164
Averell L, Heathcote A (2011) The form of the forgetting curve and the fate of memories. J Math Psychol 55:25–35
Brown S, Heathcote A (2003) Averaging learning curves across and within participants. Behav Res Methods Instrum Comput 35:11–21
Heathcote A, Brown S, Mewhort DJK (2000) The power law repealed: the case for an exponential law of practice. Psychon Bull Rev 7:185–207
Pachella RG (1974) The interpretation of reaction time in information–processing research. In: Kantowitz BH (ed) Human information processing: tutorials in performance and cognition. Lawrence Erlbaum Associates, Hillsdale, pp 41–82
Audley RJ, Pike AR (1965) Some alternative models of stochastic choice. Br J Math Stat Psychol 207–225
Vickers D, Caudrey D, Willson R (1971) Discriminating between the frequency of occurrence of two alternative events. ACTPSY 35:151–172
Ratcliff R, Thapar A, McKoon G (2001) The effects of aging on reaction time in a signal detection task. Psychol Aging 16:323
Wagenmakers EJ, Ratcliff R, Gómez P, McKoon G (2008) A diffusion model account of criterion shifts in the lexical decision task. J Memory Lang 58:140–159
Rae B, Heathcote A, Donkin C, Averell L, Brown SD (2013) The Hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem Cogn 1–45
Wagenmakers EJ, Ratcliff R, Gómez P, Iverson GJ (2004) Assessing model mimicry using the parametric bootstrap. J Math Psychol 48:28–50
Ratcliff R, Rouder JN (1998) Modeling response times for two–choice decisions. Psychol Sci 9:347–356
Morey RD (2008) Confidence intervals from normalized data: a correction to Cousineau. Tutor Quant Methods Psychol 4:61–64
Heathcote A, Love J (2012) Linear deterministic accumulator models of simple choice. Front Psychol 3
Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variability and dual-process explanations of zroc slopes with response time data and the diffusion model. Cognit Psychol 64:1–34
Cleveland WS (1993) Visualizing data. Hobart Press, New Jersey
Heathcote A, Brown S, Mewhort DJK (2002) Quantile maximum likelihood estimation of response time distributions. Psychon Bull Rev 9(2):394–401
Heathcote A, Brown S (2004) A theoretical basis for QML. Psychon Bull Rev 11:577–578
Speckman PL, Rouder JN (2004) A comment on Heathcote, Brown, and Mewhort’s QMLE method for response time distributions. Psychon Bull Rev 11:574–576
Kass RE, Raftery AE (1995) Bayes factors. J Am Stat Assoc 90:773–795
Lodewyckx T, Kim W, Lee MD, Tuerlinckx F, Kuppens P, Wagenmakers EJ (2011) A tutorial on Bayes factor estimation with the product space method. J Math Psychol 55:331–347
Shiffrin R, Lee M, Kim W, Wagenmakers EJ (2008) A survey of model evaluation approaches with a tutorial on Hierarchical Bayesian Methods. Cognit Sci A Multidiscip J 32:1248–1284
Spiegelhalter DJ, Best NG, Carlin BP, Van Der Linde A (2002) Bayesian measures of model complexity and fit. J Royal Stat Soc Series B (Stat Methodol) 64:583–639
Ando T (2007) Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models. Biometrika 94:443–458
Myung IJ, Pitt MA (1997) Applying Occam’s razor in modeling cognition: a Bayesian approach. Psych Bull Rev 4:79–95
Burnham KP, Anderson DR (2004) Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res 33:261–304
Donkin C, Brown S, Heathcote A (2011) Drawing conclusions from choice response time models: a tutorial using the linear ballistic accumulator. J Math Psychol 55:140–151
Turner BM, Sederberg PB, Brown SD, Steyvers M (2013) A method for efficiently sampling from distributions with correlated dimensions. Psychol Methods 18:368–384
Farrell S, Wagenmakers EJ, Ratcliff R (2006) 1/f noise in human cognition: is it ubiquitous, and what does it mean? Psych Bull Rev 13:737–741
Gilden DL (1997) Fluctuations in the time required for elementary decisions. Psychol Sci 8:296–301
Box GEP (1979) Robustness in the strategy of scientific model building. In: Launer RL, Wilkinson GN (ed) Robustness in statistics: proceedings of a workshop. Academic, New York, pp 201–236
Lewandowsky S, Farrell S (2010) Computational modeling in cognition: principles and practice. Sage, Thousand Oaks
Busemeyer JR, Diederich A (2010) Cognitive modeling. Sage, Thousand Oaks
Hunt E (2006) The mathematics of behavior. Cambridge University Press, Cambridge
Polk TA, Seifert CM (eds) (2002) Cognitive modeling. MIT Press, Cambridge
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Heathcote, A., Brown, S., Wagenmakers, EJ. (2015). An Introduction to Good Practices in Cognitive Modeling. In: Forstmann, B., Wagenmakers, EJ. (eds) An Introduction to Model-Based Cognitive Neuroscience. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-2236-9_2
Download citation
DOI: https://doi.org/10.1007/978-1-4939-2236-9_2
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4939-2235-2
Online ISBN: 978-1-4939-2236-9
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)