Skip to main content
SpringerLink
Log in

The effect of incentivization on the conjunction fallacy in judgments: a meta-analysis

  • Review
  • Published:
Psychological Research Aims and scope Submit manuscript

Abstract

The conjunction fallacy is a classical judgment bias that was argued to be a robust cognitive illusion insensitive to the positive effect of incentivization. We conducted a meta-analysis of the literature (n = 3276) and found that although most studies did not report a significant effect of incentivization, the results across studies show a significant positive effect for incentivization, d = 0.19, with an odds ratio of 1.40 for answering correctly when incentivized. There was no moderating effect of payoff size despite the differences in incentive value between studies. Additionally, the effect was relatively smaller when examining absolute differences in the probability of correct judgment instead of odds ratios, suggesting that it may be partly driven by studies with low baseline performance. These findings join those of other judgment-bias studies to suggest a small but nevertheless robust debiasing effect of incentivization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The data for this meta-analysis is available on https://osf.io/dh7sb/?view_only=262aeec537094702bcb508be3a50ce07

Notes

  1. Likewise, the liberal paternalism (or “nudge”) approach was originally based on the assumption that people cannot in practice increase their cognitive effort (e.g., through incentives) so as to overcome judgment and decision biases (Thaler & Sunstein, 2008).

  2. Reviews of this literature concluded either that incentives had a positive effect on judgment performance (Hertwig & Ortmann, 2001) or that it had a very limited effect (Camerer & Hogarth, 1999).

  3. On average the rate of correct choices in the non-incentivized condition was 45% compared to 55% in the incentivized condition.

  4. Rather, the two metrics were used as robustness tests. Formally comparing the two statistics likely requires a larger number of studies.

References

  • Arkes, H. R. (1991). Costs and benefits of judgment errors: Implications for debiasing. Psychological Bulletin, 110, 486–498.

    Article  Google Scholar 

  • Awasthi, V., & Pratt, J. (1990). The effects of monetary incentives on effort and decision performance: The role of cognitive characteristics. Accounting Review, 65, 797–811.

    Google Scholar 

  • Baillon, A., Bleichrodt, H., & Granic, G.D. (2022). Incentives in surveys. Working paper, available at http://docplayer.net/164787861-Incentives-in-surveys.html

  • Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.

    Article  PubMed  Google Scholar 

  • Brañas-Garza, P., Kujal, P., & Lenkei, B. (2019). Cognitive reflection test: Whom, how, when. Journal of Behavioral and Experimental Economics, 82, 101455.

    Article  Google Scholar 

  • Camerer, C. F., & Hogarth, R. M. (1999). The effects of financial incentives in experiments: A review and capital-labor-production framework. Journal of Risk and Uncertainty, 19, 7–42.

    Article  Google Scholar 

  • Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis. Psychological Bulletin, 140, 980–1008.

    Article  PubMed  Google Scholar 

  • Chandrashekar, S.P., Cheng, Y.H., Fong, C.L., Leung, Y.C., Wong, Y.T., Cheng, B.L., & Feldman, G. (2021). Frequency estimation and semantic ambiguity do not eliminate conjunction bias, when it occurs: Replication and extension of Mellers, Hertwig, and Kahneman (2001). Meta-Psychology, 5, MP.2020.2474.

  • Charness, G., Karni, E., & Levin, D. (2008). On the conjunction fallacy in probability judgment: New experimental evidence. Johns Hopkins University working paper. See: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1155219

  • Charness, G., Karni, E., & Levin, D. (2010). On the conjunction fallacy in probability judgment: New experimental evidence regarding Linda. Games and Economic Behavior, 68, 551–556.

    Article  Google Scholar 

  • Cortina, J. M., & Landis, R. S. (2009). When small effect sizes tell a big story, and when large effect sizes don’t. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 287–308). Routledge.

    Google Scholar 

  • Dale, D., Rudski, J., Schwarz, A., & Smith, E. (2007). Innumeracy and incentives: A ratio bias experiment. Judgment and Decision Making, 2, 243–250.

    Article  Google Scholar 

  • Devetag, G., Ceccacci, F., & De Salvo, P. (2013). Do reputation concerns make behavioral biases disappear? The conjunction fallacy on Facebook and Mechanical Turk. Working paper available in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2263179

  • Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.

    Article  PubMed  Google Scholar 

  • Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.

    Article  PubMed  PubMed Central  Google Scholar 

  • Enke, B., Gneezy, U., Hall, B., Martin, D., Nelidov, V., Offerman, T., & van de Ven, J. (2022). Cognitive biases: Mistakes or missing stakes? In press.

    Google Scholar 

  • Epley, N., & Gilovich, T. (2005). When effortful thinking influences judgmental anchoring: Differential effects of forewarning and incentives on self-generated and externally provided anchors. Journal of Behavioral Decision Making, 18, 199–212.

    Article  Google Scholar 

  • Erev, I., & Cohen, B. L. (1990). Verbal versus numerical probabilities: Efficiency, biases, and the preference paradox. Organizational Behavior and Human Decision Processes, 45, 1–18.

    Article  Google Scholar 

  • Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.

    Article  Google Scholar 

  • Gao, G. (2009). Probability error in diagnosis: The conjunction fallacy among beginning medical students. Family Medicine, 41, 262–265.

    Google Scholar 

  • Harrison, G. W. (1994). Expected utility theory and the experimentalists. In J. D. Hey (Ed.), Experimental economics (pp. 43–76). Springer-Verlag.

    Chapter  Google Scholar 

  • Hedges, L. V., & Olkin, I. (2014). Statistical methods for meta-analysis. Academic press.

    Google Scholar 

  • Hertwig, R., & Ortmann, A. (2001). Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24, 383–403.

    Article  PubMed  Google Scholar 

  • Isler, O., & Yilmaz, O. (2022). How to activate intuitive and reflective thinking in behavior research? A comprehensive examination of experimental technique. Working paper. Retrieved from https://psyarxiv.com/jcyt2/download?format=pdf

  • Jackson, D., & Turner, R. (2017). Power analysis for random-effects meta-analysis. Research Synthesis Methods, 8, 290–302.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kahneman, D. (1973). Attention and effort. Prentice-Hall.

    Google Scholar 

  • Kahneman, D. (1991). Judgment and decision making: A personal view. Psychological Science, 2, 142–145.

    Article  Google Scholar 

  • Kim, A. J., Lee, D. S., & Anderson, B. A. (2021). The influence of threat on the efficiency of goal-directed attentional control. Psychological Research Psychologische Forschung, 85, 980–986.

    Article  PubMed  Google Scholar 

  • Kontopantelis, E., & Reeves, D. (2012). Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A comparison between DerSimonian–Laird and restricted maximum likelihood. Statistical Methods in Medical Research, 21, 657–659.

    Article  PubMed  Google Scholar 

  • Larrick, R. P. (2004). Debiasing. In D. Koehler & N. Harvey (Eds.), Blackwell handbook of judgment and decision making (pp. 316–337). Blackwell Publishing.

    Chapter  Google Scholar 

  • Lawson, M. A., Larrick, R. P., & Soll, J. B. (2020). Comparing fast thinking and slow thinking: The relative benefits of interventions, individual differences, and inferential rules. Judgment and Decision Making, 15, 660–684.

    Article  Google Scholar 

  • Lefebvre, M., Vieider, F. M., & Villeval, M. C. (2011). The ratio bias phenomenon: Fact or artifact? Theory and Decision, 71, 615–641.

    Article  Google Scholar 

  • Levin, I. P., Chapman, D. P., & Johnson, R. D. (1988). Confidence in judgments based on incomplete information: An investigation using both hypothetical and real gambles. Journal of Behavioral Decision Making, 1, 29–41.

    Article  Google Scholar 

  • McInnes, M. D. F., Moher, D., Thombs, B. D., McGrath, T. A., Bossuyt, P. M., the PRISMA-DTA Group, & Willis, B. H. (2018). Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: The PRISMA-DTA statement. Journal of the American Medical Association, 319, 388–396.

    Article  PubMed  Google Scholar 

  • Mellers, B., Hertwig, R., & Kahneman, D. (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science, 12, 269–275.

    Article  PubMed  Google Scholar 

  • Navon, D., & Gopher, D. (1979). On the economy of the human processing system. Psychological Review, 86, 214–253.

    Article  Google Scholar 

  • Richter, M., & Gendolla, G. H. E. (2009). The heart contracts to reward: Monetary incentives and preejection period. Psychophysiology, 46, 451–457.

    Article  PubMed  Google Scholar 

  • Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8, 448–467.

    Article  PubMed  Google Scholar 

  • Shafir, E., & LeBoeuf, R. A. (2002). Rationality. Annual Review of Psychology, 53, 419–517.

    Article  Google Scholar 

  • Shaw, J. D., & Gupta, N. (2015). Let the evidence speak again! Financial incentives are more effective than we thought. Human Resource Management Journal, 25, 281–293.

    Article  Google Scholar 

  • Sjastad, H., & Baumeister, R.F. (2020). Fast optimism, slow realism? Causal evidence for a two-step model of future thinking. Working paper, available at https://psyarxiv.com/v6ksu/

  • Stolarz-Fantino, S., Fantino, E., Zizzo, D. J., & Wen, J. (2003). The conjunction effect: New evidence for robustness. American Journal of Psychology, 116, 15–34.

    Article  PubMed  Google Scholar 

  • Szollosi, A., Bago, B., Szaszi, B., & Aczel, B. (2017). Exploring the determinants of confidence in the bat-and-ball problem. Acta Psychologica, 180, 1–7.

    Article  PubMed  Google Scholar 

  • Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Yale University Press.

    Google Scholar 

  • Travers, E., Rolison, J. J., & Feeney, A. (2016). The time course of conflict on the cognitive reflection test. Cognition, 150, 109–118.

    Article  PubMed  Google Scholar 

  • Tversky, A., & Kahneman, D. (1982). Judgments of and by representativeness. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 84–100). Cambridge University Press.

    Chapter  Google Scholar 

  • Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90, 293–315.

    Article  Google Scholar 

  • Viechtbauer, W. (2010). Conducting meta-analyses in R with the metaphor package. Journal of Statistical Software, 36, 1–48.

    Article  Google Scholar 

  • von Hippel, P. T. (2015). The heterogeneity statistic I2 can be biased in small meta-analyses. BMC Medical Research Methodology, 15, 35.

    Article  Google Scholar 

  • Wright, R. A. (1998). Ability perception and cardiovascular response to behavioral challenge. In M. Kofta, G. Weary, & G. Sedek (Eds.), Control in action: Cognitive and motivational mechanisms (pp. 197–232). Plenum Press.

    Google Scholar 

  • Wright, W. F., & Anderson, U. (1989). Effects of situation familiarity and financial incentives on use of the anchoring and adjustment heuristic for probability assessment. Organizational Behavior and Human Decision Processes, 44, 68–82.

    Article  Google Scholar 

  • Xue, G., Lu, Z., Levin, I. P., Weller, J. A., Li, X., & Bechara, A. (2009). Functional dissociations of risk and reward processing in the medial prefrontal cortex. Cerebral Cortex, 19, 1019–1027.

    Article  PubMed  Google Scholar 

  • Yechiam, E., & Hochman, G. (2013). Losses as modulators of attention: Review and analysis of the unique effects of losses over gains. Psychological Bulletin, 139, 497–518.

    Article  PubMed  Google Scholar 

  • Yechiam, E., & Hochman, G. (2014). Loss attention in a dual task setting. Psychological Science, 25, 494–502.

    Article  PubMed  Google Scholar 

  • Yechiam, E., & Zeif, D. (2022). Revisiting the effect of incentivization on cognitive reflection: A meta-analysis. Journal of Behavioral Decision Making., 36, e2286.

    Article  Google Scholar 

  • Young, M. S., & Stanton, N. A. (2002). Malleable attentional resources theory: A new explanation for the effects of mental underload on performance. Human Factors, 44, 365–375.

    Article  PubMed  Google Scholar 

  • Zeif, D. (2022). The effect of strategies of attention on judgment and decision-making performance. Unpublished doctoral dissertation. Technion – Israel Institute of Technology.

  • Zizzo, J. Z., Stolarz-Fantino, S., Wen, J., & Fantino, E. (2000). A violation of the monotonicity axiom: Experimental evidence on the conjunction fallacy. Journal of Economic Behavior & Organization, 41, 263–276.

    Article  Google Scholar 

Download references

Funding

The study was supported by a grant from the Technion Research & Development Foundation to the second author and by the Max Wertheimer Minerva Center for Cognitive Studies.

Author information

Authors and Affiliations

  1. Faculty of Data and Decision Sciences, Technion - Israel Institute of Technology, 3200003, Haifa, Israel

    Eldad Yechiam & Dana Zeif

Authors
  1. Eldad Yechiam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dana Zeif
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

EY and DZ conceptualized the study and conducted the search for papers. EY wrote the first draft of the paper and both authors reviewed the paper.

Corresponding author

Correspondence to Eldad Yechiam.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 44 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yechiam, E., Zeif, D. The effect of incentivization on the conjunction fallacy in judgments: a meta-analysis. Psychological Research 87, 2336–2344 (2023). https://doi.org/10.1007/s00426-023-01837-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00426-023-01837-5

Search

Navigation