Experimental Economics

, Volume 18, Issue 3, pp 457–490 | Cite as

A penny for your thoughts: a survey of methods for eliciting beliefs

  • Karl H. Schlag
  • James Tremewan
  • Joël J. van der Weele
Original Paper


Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.


Belief elicitation Subjective beliefs Scoring rules Experimental design 

JEL Classification

C83 C91 D83 



We would like to thank Peter Wakker, Theo Offerman, Glenn Harrison, Gerhard Sorger and two anonymous referees for useful comments.


  1. Allen, F. (1987). Discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544.CrossRefGoogle Scholar
  2. Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities. Journal of Risk and Uncertainty, 48(3), 207–220.Google Scholar
  3. Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle. International Economic Review, 50(4), 1079–1102.CrossRefGoogle Scholar
  4. Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. European Economic Review, 62, 17–40.CrossRefGoogle Scholar
  5. Artinger, F., Exadaktylos, F., Koppel, H., & Sääksvuori, L. (2010). Applying Quadratic Scoring Rule transparently in multiple choice settings: a note. Technical report, Jena Economic Research Paper.Google Scholar
  6. Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75(3), 354–359.CrossRefGoogle Scholar
  7. Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions. Journal of Experimental Psychology, 81(3), 561–565.CrossRefGoogle Scholar
  8. Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226.CrossRefGoogle Scholar
  9. Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences. Handbook of Experimental Economics Results, 1, 1087–1097.CrossRefGoogle Scholar
  10. Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem? Experimental Economics, 13(4), 412–438.CrossRefGoogle Scholar
  11. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.CrossRefGoogle Scholar
  12. Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 513–519). Oxford: Oxford University Press.Google Scholar
  13. Costa-Gomes, M. A., Huck, S. and Weizsacker, G. (2012). Beliefs and actions in the trust game: creating instrumental variables to estimate the causal effect. WZB Discussion Paper, 2012–302.Google Scholar
  14. Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play. Journal of Economic Behavior & Organization, 41(3), 299–314.CrossRefGoogle Scholar
  15. Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation. Journal of Personality and Social Psychology, 35(1), 1.CrossRefGoogle Scholar
  16. De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 18(1), 87–123.CrossRefGoogle Scholar
  17. De Finetti, B. (1970). Logical foundations and measurement of subjective probability. Acta Psychologica, 34, 129–145.CrossRefGoogle Scholar
  18. De Finetti, B. (1974). Theory of probability (Vol. 1). New York: Wiley.Google Scholar
  19. Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence. Journal of Development Economics, 94(2), 151–163.CrossRefGoogle Scholar
  20. Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment. Organizational Behavior and Human Performance, 10, 108–117.CrossRefGoogle Scholar
  21. Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given. Experimental Economics, 260(2000), 241–260.CrossRefGoogle Scholar
  22. Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality. Organizational Behavior and Human Decision Processes, 55, 78–94.CrossRefGoogle Scholar
  23. Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance, 29(3), 352–369.CrossRefGoogle Scholar
  24. Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict? Applied Economics Letters, 18(3), 267–272.CrossRefGoogle Scholar
  25. Friedman, D. (1983). Effective scoring rules for probabilistic forecasts. Management Science, 29(4), 447–454.CrossRefGoogle Scholar
  26. Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377.CrossRefGoogle Scholar
  27. Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701.CrossRefGoogle Scholar
  28. Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704.CrossRefGoogle Scholar
  29. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.CrossRefGoogle Scholar
  30. Good, I. (1952). Rational decisions. Journal of the Royal Statistical Society. Series B, 14(1), 107–114.Google Scholar
  31. Grether, D. (1981). Financial incentive effects and individual decision-making. California Institute of Technology, Working Paper 401.Google Scholar
  32. Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs. Journal of Economic Behavior & Organization, 55(1), 25–30.CrossRefGoogle Scholar
  33. Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study. Journal of Risk and Uncertainty, 2, 161–180.CrossRefGoogle Scholar
  34. Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior and Organization, 101, 128–140.CrossRefGoogle Scholar
  35. Harrison, G. W., Martínez-correa, J., Swarthout, J. T., & Ulm, E. R. (2013a). Scoring rules for subjective probability distributions. Manuscript, Georgia State University.Google Scholar
  36. Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior and Organization, 94, 145–159.CrossRefGoogle Scholar
  37. Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral. Experimental Economics, 12(2), 133–158.CrossRefGoogle Scholar
  38. Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games. Review of Economic Studies, 76, 181–221.CrossRefGoogle Scholar
  39. Hoffmann, T. (2013). The Effect of Belief Elicitation on Game Play. Manuscript, Mannheim University.Google Scholar
  40. Hollard, G., Massoni, S. and Vergnaud, J. (2010). Subjective beliefs formation and elicitation rules : Experimental evidence. Centre d’Economie de la Sorbonne Working Paper, 2010.88.Google Scholar
  41. Holt, C. (2006). Markets, games and strategic behavior. Boston: Pearson/Addison-Wesley.Google Scholar
  42. Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects. The American Economic Review, 92(5), 1644.CrossRefGoogle Scholar
  43. Hossain, T., & Okui, R. (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001.CrossRefGoogle Scholar
  44. Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs. Journal of Economic Behavior & Organization, 47(1), 71–85.CrossRefGoogle Scholar
  45. Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs. Journal of Risk and Uncertainty, 30(2), 169–188.CrossRefGoogle Scholar
  46. Hurley, T. T. M., Peterson, N., & Shogren, J. J. F. (2007). Belief elicitation: An experimental comparison of scoring rule and prediction methods. Manuscript, University of Minnesota.Google Scholar
  47. Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable. Journal of Risk and Uncertainty, 8, 5–20.CrossRefGoogle Scholar
  48. Jenkinson, D. (2005). The elicitation of probabilities—A review of the statistical literature. Manuscript, University of Sheffield.Google Scholar
  49. Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules. Organizational Behavior and Human Performance, 9(2), 307–317.CrossRefGoogle Scholar
  50. Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments. Operations Research, 57(5), 1287–1297.CrossRefGoogle Scholar
  51. Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities. Journal of the American Statistical Association, 83(402), 357–363.CrossRefGoogle Scholar
  52. Karni, E. (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606.CrossRefGoogle Scholar
  53. Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities. Theory and Decision, 38, 313–320.CrossRefGoogle Scholar
  54. Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets. Journal of Economic Behavior & Organization, 83(2), 195–208.CrossRefGoogle Scholar
  55. Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. Journal of Multi-Criteria Decision Analysis, 101, 101–113.Google Scholar
  56. Krawczyk, M. (2011). Overconfident for real? Proper scoring for confidence intervals. Manuscript, University of Warsaw.Google Scholar
  57. Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights. ACM SIGecom Exchanges, 7(3), 1–5.CrossRefGoogle Scholar
  58. Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments. Journal of the Royal Statistical Society. Series A (General), 142(2), 146.CrossRefGoogle Scholar
  59. Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44.CrossRefGoogle Scholar
  60. Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780.CrossRefGoogle Scholar
  61. Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response. European Economic Review, 46, 880–891.CrossRefGoogle Scholar
  62. Manski, C. (2004). Measuring expectations. Econometrica, 72(5), 1329–1376.CrossRefGoogle Scholar
  63. Manski, C. F. (2006). Interpreting the predictions of prediction markets. Economics Letters, 91(3), 425–429.CrossRefGoogle Scholar
  64. Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096.CrossRefGoogle Scholar
  65. McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences of the United States of America, 42(9), 654–655.CrossRefGoogle Scholar
  66. McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling. Econometrica, 58(6), 1321–1339.CrossRefGoogle Scholar
  67. Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science, 51(9), 1359–1373.CrossRefGoogle Scholar
  68. Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica, 70(3), 971–1005.CrossRefGoogle Scholar
  69. Offerman, T., & Palley, A. B. (2013). Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents. Manuscript, University of Amsterdam.Google Scholar
  70. Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods. The Economic Journal, 106(437), 817–845.CrossRefGoogle Scholar
  71. Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians. Review of Economic Studies, 76(4), 1461–1489.CrossRefGoogle Scholar
  72. Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games. Journal of Economic Behavior & Organization, 71(2), 98–109.CrossRefGoogle Scholar
  73. Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72(3), 346–354.CrossRefGoogle Scholar
  74. Prelec, D. (2004). A bayesian truth serum for subjective data. Science, 306, 462–466.CrossRefGoogle Scholar
  75. Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes, 76(3), 277–297.CrossRefGoogle Scholar
  76. Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.), The foundations of mathematics and other logical essays (pp. 156–198). New York (1931): Harcourt.Google Scholar
  77. Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory.Google Scholar
  78. Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games and Economic Behavior, 67(2), 616–632.CrossRefGoogle Scholar
  79. Savage, L. J. (1954). The foundation of statistics. New York: Wiley.Google Scholar
  80. Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801.CrossRefGoogle Scholar
  81. Schervish, M. (1989). A general method for comparing probability assessors. The Annals of Statistics, 17(4), 1856–1879.CrossRefGoogle Scholar
  82. Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224.Google Scholar
  83. Schlag, K. H., & van der Weele, J. J. (2012). Incentives for interval elicitation. Manuscript, Vienna University.Google Scholar
  84. Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Economics Letters, 03(1), 38–42.CrossRefGoogle Scholar
  85. Schmalensee, R. (1976). An experimental study of expectation formation. Econometrica, 44(1), 17–41.CrossRefGoogle Scholar
  86. Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128.CrossRefGoogle Scholar
  87. Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements. Organizational Behavior and Human Performance, 2, 84–104.CrossRefGoogle Scholar
  88. Seghers, R. C., Fryback, D. G., & Goodman, B. C. (1973). Relative variance preferences in a choice-among-bets paradigm. Technical report, DTIC Document.Google Scholar
  89. Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 62, 43–62.CrossRefGoogle Scholar
  90. Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory and Decision, 46, 211–249.CrossRefGoogle Scholar
  91. Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures. Psychometrika, 31(2), 125–145.CrossRefGoogle Scholar
  92. Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games. Experimental Economics, 16(3), 414–425.CrossRefGoogle Scholar
  93. Smith, C. (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B, 23(1), 1–37.Google Scholar
  94. Sonnemans, J., & Offerman, T. T. T. (2001). Is the quadratic scoring rule behaviorally incentive compatible?. Manuscript, University of Amsterdam.Google Scholar
  95. Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146–159.CrossRefGoogle Scholar
  96. Toda, M. (1951). Measurement of intuitive probability by a method of game. Japanese Journal of Psychology, 22, 29–40.CrossRefGoogle Scholar
  97. Toda, M. (1963). Measurement of subjective probability distribution. Report No. 3(3), State College, Pennsylvania, Institute for Research, Division of Mathematical Psychology.Google Scholar
  98. Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums. The Economic Journal. doi: 10.1111/ecoj.12160.
  99. Tversky, A., & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101(4), 547.CrossRefGoogle Scholar
  100. Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities. British Journal of Mathematical and Statistical Psychology, 26(1), 18–30.CrossRefGoogle Scholar
  101. Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment. Organizational Behavior and Human Performance, 9(460–481), 460–481.CrossRefGoogle Scholar
  102. Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 32(1), 38–43.CrossRefGoogle Scholar
  103. Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters, 111(1), 30–33.CrossRefGoogle Scholar
  104. Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time. Journal of Behavioral Decision Making, 6(4), 283–296.CrossRefGoogle Scholar
  105. Wilcox, N. T. and Feltovich, N. (2000). Thinking like a game theorist: Comment. University of Houston Department of Economics working paper.Google Scholar
  106. Winkler, R. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60.CrossRefGoogle Scholar
  107. Winkler, R., & Murphy, A. (1968). ”Good” probability assessors. Journal of Applied Meteorology, 7, 751.CrossRefGoogle Scholar
  108. Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score. Journal of Applied Meteorology, 9, 143–148.CrossRefGoogle Scholar
  109. Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures. The Meteorological Magazine, 108(1288), 317–329.Google Scholar
  110. Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200.Google Scholar
  111. Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General, 124(4), 424–432.CrossRefGoogle Scholar
  112. Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation. Journal of Behavioral Decision Making, 10, 21–32.CrossRefGoogle Scholar
  113. Zizzo, D. J. (2009). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75–98.CrossRefGoogle Scholar

Copyright information

© Economic Science Association 2014

Authors and Affiliations

  • Karl H. Schlag
    • 1
  • James Tremewan
    • 1
  • Joël J. van der Weele
    • 2
    • 3
    • 4
  1. 1.University of ViennaViennaAustria
  2. 2.Department of Economics, Center for Experimental Economics and political Decision making (CREED)University of AmsterdamAmsterdamThe Netherlands
  3. 3.Tinbergen InstituteAmsterdamThe Netherlands
  4. 4.Center for Financial StudiesFrankfurtGermany

Personalised recommendations