Skip to main content
SpringerLink
Log in

Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice

  • Theoretical Review
  • Published:
Psychonomic Bulletin & Review Aims and scope Submit manuscript

Abstract

To be useful, cognitive models with fitted parameters should show generalizability across time and allow accurate predictions of future observations. It has been proposed that hierarchical procedures yield better estimates of model parameters than do nonhierarchical, independent approaches, because the formers’ estimates for individuals within a group can mutually inform each other. Here, we examine Bayesian hierarchical approaches to evaluating model generalizability in the context of two prominent models of risky choice—cumulative prospect theory (Tversky & Kahneman, 1992) and the transfer-of-attention-exchange model (Birnbaum & Chavez, 1997). Using empirical data of risky choices collected for each individual at two time points, we compared the use of hierarchical versus independent, nonhierarchical Bayesian estimation techniques to assess two aspects of model generalizability: parameter stability (across time) and predictive accuracy. The relative performance of hierarchical versus independent estimation varied across the different measures of generalizability. The hierarchical approach improved parameter stability (in terms of a lower absolute discrepancy of parameter values across time) and predictive accuracy (in terms of deviance; i.e., likelihood). With respect to test–retest correlations and posterior predictive accuracy, however, the hierarchical approach did not outperform the independent approach. Further analyses suggested that this was due to strong correlations between some parameters within both models. Such intercorrelations make it difficult to identify and interpret single parameters and can induce high degrees of shrinkage in hierarchical models. Similar findings may also occur in the context of other cognitive models of choice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Note that Nilsson et al. (2011) used Tversky and Kahneman’s (1992) one-parameter probability-weighting function, which does not disentangle elevation and curvature.

  2. The BUGS programming code for each model implementation is available in the online supplementary materials.

  3. Because Nilsson et al. (2011) employed a different weighting function in their comparison of independent and hierarchical parameter estimations for CPT, our results are not directly comparable with theirs. Nevertheless, note that Nilsson et al. also found the hierarchical approach to yield a lower choice sensitivity. Interestingly, they obtained a pattern of results opposite to ours with regard to loss aversion, with the hierarchical approach yielding a higher λ.

  4. See the online supplementary materials for the BUGS programming code.

  5. See the online supplemental materials for similar plots of the other parameters.

  6. Note, however, that this would not necessarily be the case. It is possible to conceive of situations in which shrinkage reduces the variance but retains the (linear) relationship between the individual parameters; in such cases, the test–retest correlations would not be lower for hierarchically estimated parameters, as they indeed are not for most of the parameters in Fig. 2.

  7. For pragmatic reasons, in the hierarchical case all participants, including those predicted at t2 at any one time, were included in the parameter estimation. This may have yielded a small advantage for the hierarchical approach over the independent approach.

  8. Bayes factor estimates were calculated from conventional t-test outputs on the basis of the template by Rouder, Speckman, Sun, Morey, and Iverson (2009), assuming the Jeffrey–Zellner–Siow prior and r = 1.

  9. The range of the prior distribution has very little impact on the results when taking all 138 choices into account. Presumably, with this amount of data on the individual level, the influence of the prior on the posterior estimates is negligible.

  10. If p is a vector of probabilities for making a correct prediction, the deviance is defined as –2*sum[log(p)], whereas the squared error is defined as sum[(1 – p)2].

References

  • Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26, 217–238.

    Article  PubMed  Google Scholar 

  • Berkowitsch, N. A. J., Scheibehenne, B., & Rieskamp, J. (2014). Testing multialternative decision field theory rigorously against random utility models. Journal of Experimental Psychology: General, 143, 1331–1348. doi:10.1037/a0035159

    Article  Google Scholar 

  • Birnbaum, M. H. (2008). New paradoxes of risky decision making. Psychological Review, 115, 463–501. doi:10.1037/0033-295X.115.2.463

    Article  PubMed  Google Scholar 

  • Birnbaum, M. H., & Chavez, A. (1997). Tests of theories of decision making: Violations of branch independence and distribution independence. Organizational Behavior and Human Decision Processes, 71, 161–194.

    Article  Google Scholar 

  • Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 327, 307–310.

    Article  Google Scholar 

  • Brown, G. D. A., Neath, I., & Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114, 539–576. doi:10.1037/0033-295X.114.3.539

    Article  PubMed  Google Scholar 

  • Busemeyer, J. R., & Diederich, A. (2010). Cognitive modeling. New York, NY: Sage.

    Google Scholar 

  • Dutilh, G., Forstmann, B. U., Vandekerckhove, J., & Wagenmakers, E.-J. (2013). A diffusion model account of age differences in posterror slowing. Psychology and Aging, 28, 64–76. doi:10.1037/a0029875

    Article  PubMed  Google Scholar 

  • Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242.

    Article  Google Scholar 

  • Efron, B., & Morris, C. N. (1977). Stein’s paradox in statistics. Scientific American, 236, 119–127.

    Article  Google Scholar 

  • Fehr-Duda, H., De Gennaro, M., & Schubert, R. (2006). Gender, financial risk, and probability weights. Theory and Decision, 60, 283–313.

    Article  Google Scholar 

  • Fox, C. R., & Poldrack, R. A. (2008). Prospect theory and the brain. In P. W. Glimcher, E. Fehr, C. Camerer, & R. A. Poldrack (Eds.), Neuroeconomics: Decision making and the brain (pp. 145–174). San Diego, CA: Academic Press.

  • Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.

    Google Scholar 

  • Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Glöckner, A., & Pachur, T. (2012). Cognitive models of risky choice: Parameter stability and predictive accuracy of prospect theory. Cognition, 123, 21–32. doi:10.1016/j.cognition.2011.12.002

    Article  PubMed  Google Scholar 

  • Goldstein, W. M., & Einhorn, H. J. (1987). Expression theory and the preference reversal phenomena. Psychological Review, 94, 236–254. doi:10.1037/0033-295X.94.2.236

    Article  Google Scholar 

  • Gonzalez, R., & Wu, G. (1999). On the shape of the probability weighting function. Cognitive Psychology, 38, 129–166.

    Article  PubMed  Google Scholar 

  • Harbaugh, W. T., Krause, K., & Vesterlund, L. (2002). Risk attitudes of children and adults: Choices over small and large probability gains and losses. Experimental Economics, 5, 53–84.

    Article  Google Scholar 

  • Hendricks, W. A., & Robey, K. W. (1936). The sampling distribution of the coefficient of variation. The Annals of Mathematical Statistics, 7, 129–132.

    Article  Google Scholar 

  • Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30, 1–15.

    Article  PubMed  Google Scholar 

  • Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–291.

    Article  Google Scholar 

  • Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299–312. doi:10.1177/1745691611406925

    Article  Google Scholar 

  • Lee, M. D., & Newell, B. R. (2011). Using hierarchical Bayesian methods to examine the tools of decision-making. Judgment and Decision Making, 6, 832–842.

    Google Scholar 

  • Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge, UK: Cambridge University Press.

  • Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605–621. doi:10.3758/BF03196751

    Article  Google Scholar 

  • Lewandowsky, S. (2011). Working memory capacity and categorization: Individual differences and modeling. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 720–738. doi:10.1037/a0022639

    PubMed  Google Scholar 

  • Lewandowsky, S., & Farrell, S. (2010). Computational modeling in cognition: Principles and practice. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Li, S.-C., Lewandowsky, S., & DeBrunner, V. E. (1996). Using parameter sensitivity and interdependence to predict model scope and falsifiability. Journal of Experimental Psychology: General, 125, 360–369. doi:10.1037/0096-3445.125.4.360

    Article  Google Scholar 

  • Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28, 3049–3067.

    Article  PubMed  Google Scholar 

  • Nilsson, H., Rieskamp, J., & Wagenmakers, E.-J. (2011). Hierarchical Bayesian parameter estimation for cumulative prospect theory. Journal of Mathematical Psychology, 55, 84–93. doi:10.1016/j.jmp.2010.08.006

    Article  Google Scholar 

  • Nosofsky, R. M. (1986). Attention, similarity, and the identification–categorization relationship. Journal of Experimental Psychology: General, 115, 39–57. doi:10.1037/0096-3445.115.1.39

    Article  Google Scholar 

  • Nosofsky, R. M., & Zaki, S. R. (2002). Exemplar and prototype models revisited: Response strategies, selective attention, and stimulus generalization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 924–940. doi:10.1037/0278-7393.28.5.924

    PubMed  Google Scholar 

  • Pachur, T., Hanoch, Y., & Gummerum, M. (2010). Prospects behind bars: Analyzing decisions under risk in a prison population. Psychonomic Bulletin & Review, 17, 630–636. doi:10.3758/PBR.17.5.630

    Article  Google Scholar 

  • Pachur, T., Hertwig, R., Gigerenzer, G., & Brandstätter, E. (2013). Testing process predictions of models of risky choice: A quantitative model comparison approach. Frontiers in Psychology, 4, 646. doi:10.3389/fpsyg.2013.00646

    Article  PubMed Central  PubMed  Google Scholar 

  • Pachur, T., Hertwig, R., & Wolkewitz, R. (2014). The affect gap in risky choice: Affect-rich outcomes attenuate attention to probability information. Decision, 1, 64–78.

    Article  Google Scholar 

  • Pachur, T., & Olsson, H. (2012). Type of learning task impacts performance and strategy selection in decision making. Cognitive Psychology, 65, 207–240. doi:10.1016/j.cogpsych.2012.03.003

    Article  PubMed  Google Scholar 

  • Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing, 1–10. Retrieved from www.r-project.org/conferences/DSC-2003/Proceedings/Plummer.pdf

  • Pratte, M. S., & Rouder, J. N. (2011). Hierarchical single- and dual-process models of recognition memory. Journal of Mathematical Psychology, 55, 36–46. doi:10.1016/j.jmp.2010.08.007

    Article  Google Scholar 

  • Qiu, J., & Steiger, E.-M. (2011). Understanding the two components of risk attitudes: An experimental analysis. Management Science, 57, 193–199.

    Article  Google Scholar 

  • R Development Core Team. (2012). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org

    Google Scholar 

  • Rieskamp, J. (2008). The probabilistic nature of preferential choice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1446–1465. doi:10.1037/a0013646

    PubMed  Google Scholar 

  • Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604. doi:10.3758/BF03196750

    Article  Google Scholar 

  • Rouder, J. N., Lu, J., Morey, R. D., Sun, D., & Speckman, P. L. (2008). A hierarchical process-dissociation model. Journal of Experimental Psychology: General, 137, 370–389. doi:10.1037/0096-3445.137.2.370

    Article  Google Scholar 

  • Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225

    Article  Google Scholar 

  • Scheibehenne, B., Rieskamp, J., & Wagenmakers, E.-J. (2013). Testing adaptive toolbox models: A Bayesian hierarchical approach. Psychological Review, 120, 39–64. doi:10.1037/a0030777

    Article  PubMed  Google Scholar 

  • Scheibehenne, B., & Studer, B. (2014). A hierarchical Bayesian model of the influence of run length on sequential predictions. Psychonomic Bulletin & Review, 20, 211–217. doi:10.3758/s13423-013-0469-1

    Article  Google Scholar 

  • Schmiedek, F., Oberauer, K., Wilhelm, O., Süß, H.-M., & Wittmann, W. W. (2007). Individual differences in components of reaction time distributions and their relations to working memory and intelligence. Journal of Experimental Psychology: General, 136, 414–429. doi:10.1037/0096-3445.136.3.414

    Article  Google Scholar 

  • Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1, 43–62.

    Google Scholar 

  • Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.

    Article  PubMed  Google Scholar 

  • Stewart, N. (2011). Information integration in risky choice: Identification and stability. Frontiers in Psychology, 2, 301. doi:10.3389/fpsyg.2011.00301

    Article  PubMed Central  PubMed  Google Scholar 

  • Stott, H. P. (2006). Cumulative prospect theory’s functional menagerie. Journal of Risk and Uncertainty, 32, 101–130.

    Article  Google Scholar 

  • Su, Y., Rao, L.-L., Sun, H.-Y., Du, X.-L., Li, X., & Li, S. (2013). Is making a risky choice based on a weighting and adding process? An eye-tracking investigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1765–1780. doi:10.1037/a0032861

    PubMed  Google Scholar 

  • Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.

    Google Scholar 

  • Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5, 297–323.

    Article  Google Scholar 

  • van Ravenzwaaij, D., Dutilh, G., & Wagenmakers, E.-J. (2011). Cognitive model decomposition of the BART: Assessment and application. Journal of Mathematical Psychology, 55, 94–105. doi:10.1016/j.jmp.2010.08.010

    Article  Google Scholar 

  • Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (2010). Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task. Journal of Mathematical Psychology, 54, 14–27. doi:10.1016/j.jmp.2008.12.001

    Article  Google Scholar 

  • Yechiam, E., & Busemeyer, J. R. (2008). Evaluating generalizability and parameter consistency in learning models. Games and Economic Behavior, 63, 370–394.

    Article  Google Scholar 

  • Yechiam, E., & Ert, E. (2011). Risk attitude in decision making: In search of trait‐like constructs. Topics in Cognitive Science, 3, 166–186.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Basel, Basel, Switzerland

    Benjamin Scheibehenne

  2. Max Planck Institute for Human Development, Berlin, Germany

    Thorsten Pachur

Authors
  1. Benjamin Scheibehenne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thorsten Pachur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin Scheibehenne.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(DOCX 230 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Scheibehenne, B., Pachur, T. Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice. Psychon Bull Rev 22, 391–407 (2015). https://doi.org/10.3758/s13423-014-0684-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3758/s13423-014-0684-4

Keywords

Search

Navigation