Abstract
Lee et al. (Computational Brain & Behavior, 2019) discuss ways to improve research practices for evaluating quantitative cognitive models. We propose the additional research practices of careful consideration, documentation, and analysis of the stimuli used to generate responses. Current modeling practice too often fails to acknowledge how the stimuli used to generate responses from research participants can influence the results of model comparisons. We recommend researchers (a) disclose how the research stimuli were selected and (b) uncover and report the diagnosticity of the stimuli for the models being tested. To demonstrate the importance of this recommendation, we present lessons learned from model testing in judgment and decision-making research. We focus on the documentation and reporting of model-stimulus relationships, specifically diagnosticity, and demonstrate how transparent documentation of diagnosticity facilitates interpretation of the evidence it generates. We conclude with recommendations regarding research tools available to achieve these goals.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that this differs from ecological representativeness, mentioned above.
References
Birnbaum, M. H. (2008). Evaluation of the priority heuristic as a descriptive model of risky decision making: comment on Brandstätter, Gigerenzer, and Hertwig (2006). Psychological Review, 115(1), 253–260.
Birnbaum, M. (2011). Testing theories of risky decision making via critical tests. Frontiers in Psychology, 2, 315.
Brandstätter, E., Gigerenzer, G., & Hertwig, R. (2006). The priority heuristic: making choices without trade-offs. Psychological Review, 113(2), 409–432.
Broomell, S. B., Budescu, D. V., & Por, H. H. (2011). Pair-wise comparisons of multiple models. Judgment and Decision making, 6(8), 821–831.
Broomell, S. B., & Bhatia, S. (2014). Parameter recovery for decision modeling using choice data. Decision, 1(4), 252–274.
Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62(3), 193–217.
Cavagnaro, D. R., Myung, J. I., Pitt, M. A., & Kujala, J. V. (2010). Adaptive design optimization: a mutual information-based approach to model discrimination in cognitive science. Neural Computation, 22(4), 887–905.
Dzhafarov, E. N. (2003). Selective influence through conditional independence. Psychometrika, 68(1), 7–25.
Fischhoff, B. (1991). Value elicitation: is there anything in there? American Psychologist, 46, 835–847.
Fox, C. R., & Hadar, L. (2006). “Decisions from experience”= sampling error+ prospect theory: reconsidering Hertwig, Barron, Weber & Erev (2004). Judgment and Decision making, 1(2), 159–161.
Glöckner, A., & Betsch, T. (2008). Do people make decisions under risk based on ignorance? An empirical test of the priority heuristic against cumulative prospect theory. Organizational Behavior and Human Decision Processes, 107(1), 75–95.
Glöckner, A., Hilbig, B. E., Henninger, F., & Fiedler, S. (2016). The reversed description-experience gap: disentangling sources of presentation format effects in risky choice. Journal of Experimental Psychology: General, 145(4), 486–508.
Hau, R., Pleskac, T. J., Kiefer, J., & Hertwig, R. (2008). The description–experience gap in risky choice: the role of sample size and experienced probabilities. Journal of Behavioral Decision Making, 21(5), 493–518.
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004). Decisions from experience and the effect of rare events in risky choice. Psychological Science, 15(8), 534–539.
Jekel, M., Fiedler, S., & Glöckner, A. (2011). Diagnostic task selection for strategy classification in judgment and decision making: theory, validation, and implementation in R. Judgment and Decision making, 6(8), 782–799.
Kahneman, D., & Tversky, A. (1979). Prospect theory: an analysis of decision under risk. Econometrica, 47, 263–292.
Kellen, D., Pachur, T., & Hertwig, R. (2016). How (in) variant are subjective representations of described and experienced risk and rewards? Cognition, 157, 126–138.
Kim, W., Pitt, M. A., Lu, Z. L., Steyvers, M., & Myung, J. I. (2014). A hierarchical adaptive approach to optimal experimental design. Neural Computation, 26(11), 2465–2492.
Lee, M. D., Criss, A. H., Devezer, B., Donkin, C., Etz, A., Leite, F. P., Matzke, D., Rouder, J. N., Trueblood, J. S., White, C. N., & Vandekerckhove, J. (2019). Robust modeling in cognitive science. Computational Brain & Behavior. https://doi.org/10.1007/s42113-019-00029-y.
Myung, J. I., & Pitt, M. A. (2009). Optimal experimental design for model discrimination. Psychological Review, 116(3), 499–518.
Navarro, D. J. (2019). Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection. Computational Brain & Behavior, 2(1), 28–34.
Navarro, D. J., Myung, I. J., Pitt, M. A., & Kim, W. (2002). Global model analysis by landscaping. Proceedings of the 25th annual conference of the cognitive science society.
Navarro, D. J., Pitt, M. A., & Myung, I. J. (2004). Assessing the distinguishability of models and the informativeness of data. Cognitive Psychology, 49(1), 47–84.
von Neumann, J., & Morgenstern, O. (1947). Theory of Games and Economic Behavior, 2nd rev. Princeton University Press. Princeton, NJ.
Pfeiffer, J., Duzevik, D., Rothlauf, F., Bonabeau, E., & Yamamoto, K. (2015). An optimized design of choice experiments: a new approach for studying decision behavior in choice task experiments. Journal of Behavioral Decision Making, 28(3), 262–280.
Pitt, M. A., Kim, W., Navarro, D. J., & Myung, J. I. (2006). Global model analysis by parameter space partitioning. Psychological Review, 113(1), 57–83.
Rieskamp, J. (2008). The probabilistic nature of preferential choice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1146–1465.
Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358–367.
Roberts, S., & Pashler, H. (2002). Reply to Rodgers and Rowe (2002). Psychological Review, 109(3), 605–607.
Steegen, S., Tuerlinckx, F., & Vanpaemel, W. (2017). Using parameter space partitioning to evaluate a model’s qualitative fit. Psychonomic Bulletin & Review, 24(2), 617–631.
Townsend, J. T., & Nozawa, G. (1995). Spatio-temporal properties of elementary perception: an investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359.
Ungemach, C., Chater, N., & Stewart, N. (2009). Are probabilities overweighted or underweighted when rare outcomes are experienced (rarely)? Psychological Science, 20(4), 473–479.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Broomell, S.B., Sloman, S.J., Blaha, L.M. et al. Interpreting Model Comparison Requires Understanding Model-Stimulus Relationships. Comput Brain Behav 2, 233–238 (2019). https://doi.org/10.1007/s42113-019-00052-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42113-019-00052-z