Psychonomic Bulletin & Review

, Volume 22, Issue 2, pp 391–407 | Cite as

Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice

  • Benjamin Scheibehenne
  • Thorsten Pachur
Theoretical Review


To be useful, cognitive models with fitted parameters should show generalizability across time and allow accurate predictions of future observations. It has been proposed that hierarchical procedures yield better estimates of model parameters than do nonhierarchical, independent approaches, because the formers’ estimates for individuals within a group can mutually inform each other. Here, we examine Bayesian hierarchical approaches to evaluating model generalizability in the context of two prominent models of risky choice—cumulative prospect theory (Tversky & Kahneman, 1992) and the transfer-of-attention-exchange model (Birnbaum & Chavez, 1997). Using empirical data of risky choices collected for each individual at two time points, we compared the use of hierarchical versus independent, nonhierarchical Bayesian estimation techniques to assess two aspects of model generalizability: parameter stability (across time) and predictive accuracy. The relative performance of hierarchical versus independent estimation varied across the different measures of generalizability. The hierarchical approach improved parameter stability (in terms of a lower absolute discrepancy of parameter values across time) and predictive accuracy (in terms of deviance; i.e., likelihood). With respect to test–retest correlations and posterior predictive accuracy, however, the hierarchical approach did not outperform the independent approach. Further analyses suggested that this was due to strong correlations between some parameters within both models. Such intercorrelations make it difficult to identify and interpret single parameters and can induce high degrees of shrinkage in hierarchical models. Similar findings may also occur in the context of other cognitive models of choice.


Bayesian inference Parameter estimation Bayesian modeling Decision making Math modeling Model evaluation 

Supplementary material

13423_2014_684_MOESM1_ESM.docx (230 kb)
ESM 1 (DOCX 230 kb)


  1. Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26, 217–238.CrossRefPubMedGoogle Scholar
  2. Berkowitsch, N. A. J., Scheibehenne, B., & Rieskamp, J. (2014). Testing multialternative decision field theory rigorously against random utility models. Journal of Experimental Psychology: General, 143, 1331–1348. doi: 10.1037/a0035159 CrossRefGoogle Scholar
  3. Birnbaum, M. H. (2008). New paradoxes of risky decision making. Psychological Review, 115, 463–501. doi: 10.1037/0033-295X.115.2.463 CrossRefPubMedGoogle Scholar
  4. Birnbaum, M. H., & Chavez, A. (1997). Tests of theories of decision making: Violations of branch independence and distribution independence. Organizational Behavior and Human Decision Processes, 71, 161–194.CrossRefGoogle Scholar
  5. Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 327, 307–310.CrossRefGoogle Scholar
  6. Brown, G. D. A., Neath, I., & Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114, 539–576. doi: 10.1037/0033-295X.114.3.539 CrossRefPubMedGoogle Scholar
  7. Busemeyer, J. R., & Diederich, A. (2010). Cognitive modeling. New York, NY: Sage.Google Scholar
  8. Dutilh, G., Forstmann, B. U., Vandekerckhove, J., & Wagenmakers, E.-J. (2013). A diffusion model account of age differences in posterror slowing. Psychology and Aging, 28, 64–76. doi: 10.1037/a0029875 CrossRefPubMedGoogle Scholar
  9. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242.CrossRefGoogle Scholar
  10. Efron, B., & Morris, C. N. (1977). Stein’s paradox in statistics. Scientific American, 236, 119–127.CrossRefGoogle Scholar
  11. Fehr-Duda, H., De Gennaro, M., & Schubert, R. (2006). Gender, financial risk, and probability weights. Theory and Decision, 60, 283–313.CrossRefGoogle Scholar
  12. Fox, C. R., & Poldrack, R. A. (2008). Prospect theory and the brain. In P. W. Glimcher, E. Fehr, C. Camerer, & R. A. Poldrack (Eds.), Neuroeconomics: Decision making and the brain (pp. 145–174). San Diego, CA: Academic Press.Google Scholar
  13. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
  14. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, UK: Cambridge University Press.Google Scholar
  15. Glöckner, A., & Pachur, T. (2012). Cognitive models of risky choice: Parameter stability and predictive accuracy of prospect theory. Cognition, 123, 21–32. doi: 10.1016/j.cognition.2011.12.002 CrossRefPubMedGoogle Scholar
  16. Goldstein, W. M., & Einhorn, H. J. (1987). Expression theory and the preference reversal phenomena. Psychological Review, 94, 236–254. doi: 10.1037/0033-295X.94.2.236 CrossRefGoogle Scholar
  17. Gonzalez, R., & Wu, G. (1999). On the shape of the probability weighting function. Cognitive Psychology, 38, 129–166.CrossRefPubMedGoogle Scholar
  18. Harbaugh, W. T., Krause, K., & Vesterlund, L. (2002). Risk attitudes of children and adults: Choices over small and large probability gains and losses. Experimental Economics, 5, 53–84.CrossRefGoogle Scholar
  19. Hendricks, W. A., & Robey, K. W. (1936). The sampling distribution of the coefficient of variation. The Annals of Mathematical Statistics, 7, 129–132.CrossRefGoogle Scholar
  20. Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30, 1–15.CrossRefPubMedGoogle Scholar
  21. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–291.CrossRefGoogle Scholar
  22. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299–312. doi: 10.1177/1745691611406925 CrossRefGoogle Scholar
  23. Lee, M. D., & Newell, B. R. (2011). Using hierarchical Bayesian methods to examine the tools of decision-making. Judgment and Decision Making, 6, 832–842.Google Scholar
  24. Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge, UK: Cambridge University Press.Google Scholar
  25. Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605–621. doi: 10.3758/BF03196751 CrossRefGoogle Scholar
  26. Lewandowsky, S. (2011). Working memory capacity and categorization: Individual differences and modeling. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 720–738. doi: 10.1037/a0022639 PubMedGoogle Scholar
  27. Lewandowsky, S., & Farrell, S. (2010). Computational modeling in cognition: Principles and practice. Thousand Oaks, CA: Sage.Google Scholar
  28. Li, S.-C., Lewandowsky, S., & DeBrunner, V. E. (1996). Using parameter sensitivity and interdependence to predict model scope and falsifiability. Journal of Experimental Psychology: General, 125, 360–369. doi: 10.1037/0096-3445.125.4.360 CrossRefGoogle Scholar
  29. Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28, 3049–3067.CrossRefPubMedGoogle Scholar
  30. Nilsson, H., Rieskamp, J., & Wagenmakers, E.-J. (2011). Hierarchical Bayesian parameter estimation for cumulative prospect theory. Journal of Mathematical Psychology, 55, 84–93. doi: 10.1016/ CrossRefGoogle Scholar
  31. Nosofsky, R. M. (1986). Attention, similarity, and the identification–categorization relationship. Journal of Experimental Psychology: General, 115, 39–57. doi: 10.1037/0096-3445.115.1.39 CrossRefGoogle Scholar
  32. Nosofsky, R. M., & Zaki, S. R. (2002). Exemplar and prototype models revisited: Response strategies, selective attention, and stimulus generalization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 924–940. doi: 10.1037/0278-7393.28.5.924 PubMedGoogle Scholar
  33. Pachur, T., Hanoch, Y., & Gummerum, M. (2010). Prospects behind bars: Analyzing decisions under risk in a prison population. Psychonomic Bulletin & Review, 17, 630–636. doi: 10.3758/PBR.17.5.630 CrossRefGoogle Scholar
  34. Pachur, T., Hertwig, R., Gigerenzer, G., & Brandstätter, E. (2013). Testing process predictions of models of risky choice: A quantitative model comparison approach. Frontiers in Psychology, 4, 646. doi: 10.3389/fpsyg.2013.00646 CrossRefPubMedCentralPubMedGoogle Scholar
  35. Pachur, T., Hertwig, R., & Wolkewitz, R. (2014). The affect gap in risky choice: Affect-rich outcomes attenuate attention to probability information. Decision, 1, 64–78.CrossRefGoogle Scholar
  36. Pachur, T., & Olsson, H. (2012). Type of learning task impacts performance and strategy selection in decision making. Cognitive Psychology, 65, 207–240. doi: 10.1016/j.cogpsych.2012.03.003 CrossRefPubMedGoogle Scholar
  37. Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing, 1–10. Retrieved from
  38. Pratte, M. S., & Rouder, J. N. (2011). Hierarchical single- and dual-process models of recognition memory. Journal of Mathematical Psychology, 55, 36–46. doi: 10.1016/ CrossRefGoogle Scholar
  39. Qiu, J., & Steiger, E.-M. (2011). Understanding the two components of risk attitudes: An experimental analysis. Management Science, 57, 193–199.CrossRefGoogle Scholar
  40. R Development Core Team. (2012). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from Google Scholar
  41. Rieskamp, J. (2008). The probabilistic nature of preferential choice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1446–1465. doi: 10.1037/a0013646 PubMedGoogle Scholar
  42. Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604. doi: 10.3758/BF03196750 CrossRefGoogle Scholar
  43. Rouder, J. N., Lu, J., Morey, R. D., Sun, D., & Speckman, P. L. (2008). A hierarchical process-dissociation model. Journal of Experimental Psychology: General, 137, 370–389. doi: 10.1037/0096-3445.137.2.370 CrossRefGoogle Scholar
  44. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi: 10.3758/PBR.16.2.225 CrossRefGoogle Scholar
  45. Scheibehenne, B., Rieskamp, J., & Wagenmakers, E.-J. (2013). Testing adaptive toolbox models: A Bayesian hierarchical approach. Psychological Review, 120, 39–64. doi: 10.1037/a0030777 CrossRefPubMedGoogle Scholar
  46. Scheibehenne, B., & Studer, B. (2014). A hierarchical Bayesian model of the influence of run length on sequential predictions. Psychonomic Bulletin & Review, 20, 211–217. doi: 10.3758/s13423-013-0469-1 CrossRefGoogle Scholar
  47. Schmiedek, F., Oberauer, K., Wilhelm, O., Süß, H.-M., & Wittmann, W. W. (2007). Individual differences in components of reaction time distributions and their relations to working memory and intelligence. Journal of Experimental Psychology: General, 136, 414–429. doi: 10.1037/0096-3445.136.3.414 CrossRefGoogle Scholar
  48. Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1, 43–62.Google Scholar
  49. Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.CrossRefPubMedGoogle Scholar
  50. Stewart, N. (2011). Information integration in risky choice: Identification and stability. Frontiers in Psychology, 2, 301. doi: 10.3389/fpsyg.2011.00301 CrossRefPubMedCentralPubMedGoogle Scholar
  51. Stott, H. P. (2006). Cumulative prospect theory’s functional menagerie. Journal of Risk and Uncertainty, 32, 101–130.CrossRefGoogle Scholar
  52. Su, Y., Rao, L.-L., Sun, H.-Y., Du, X.-L., Li, X., & Li, S. (2013). Is making a risky choice based on a weighting and adding process? An eye-tracking investigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1765–1780. doi: 10.1037/a0032861 PubMedGoogle Scholar
  53. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.Google Scholar
  54. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5, 297–323.CrossRefGoogle Scholar
  55. van Ravenzwaaij, D., Dutilh, G., & Wagenmakers, E.-J. (2011). Cognitive model decomposition of the BART: Assessment and application. Journal of Mathematical Psychology, 55, 94–105. doi: 10.1016/ CrossRefGoogle Scholar
  56. Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (2010). Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task. Journal of Mathematical Psychology, 54, 14–27. doi: 10.1016/ CrossRefGoogle Scholar
  57. Yechiam, E., & Busemeyer, J. R. (2008). Evaluating generalizability and parameter consistency in learning models. Games and Economic Behavior, 63, 370–394.CrossRefGoogle Scholar
  58. Yechiam, E., & Ert, E. (2011). Risk attitude in decision making: In search of trait‐like constructs. Topics in Cognitive Science, 3, 166–186.CrossRefPubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  1. 1.University of BaselBaselSwitzerland
  2. 2.Max Planck Institute for Human DevelopmentBerlinGermany

Personalised recommendations