Skip to main content
Log in

A penny for your thoughts: a survey of methods for eliciting beliefs

  • Original Paper
  • Published:
Experimental Economics Aims and scope Submit manuscript

Abstract

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Here we assume that \(\theta \) is uniquely determined given \(X\), so \(\theta =\theta \left( X\right) \). Definitions become a bit more involved when the characteristic of interest is not always uniquely defined, see Sect. 2.7.

  2. Truth-telling applies the concept of incentive compatibility to belief elicitation. Kothiyal et al. (2011) point out that the earliest incentive compatible scoring rules were proposed at least a decade before the first work on mechanism design was published.

  3. \({\mathbf{1}}_{\left\{ r\le c\right\} }=1\) if \(r\le c\) and \(=0\) if \(r>c\). \({\mathbf{1}}_{\left\{ r>c\right\} }\) is defined similarly.

  4. This scheme provide the same incentives as the QSR, and provides the same payoffs if the prize is given with certainty if the event does not occur.

  5. McCarthy (1956) mentions this claim and attributes it to Gleason (unpublished). Since we were unable to locate the latter study, we prove this here for \(n=2\), using the framework of Savage (1971). We search for a rule such that \(Y\left( r\right) =Z\left( 1-r\right) \) for all \(r\), and hence \(J^{\prime }\left( p\right) =Y\left( p\right) -Z\left( p\right) =Y\left( p\right) -Y\left( 1-p\right) \). Since \(J\left( p\right) =Y\left( p\right) p+Y\left( 1-p\right) \left( 1-p\right) \) we obtain \(J^{\prime }\left( p\right) =Y\left( p\right) -Y\left( 1-p\right) +Y^{\prime }\left( p\right) p-Y^{\prime }\left( 1-p\right) \left( 1-p\right) \) and hence \(Y^{\prime }\left( p\right) p=Y^{\prime }\left( 1-p\right) \left( 1-p\right) \) for all \(p\). This implies that \(Y\left( p\right) =a\ln p+b\) for some \(a>0\) and \(b\) and hence \(S\) is an affine transformation of the the logarithmic payment scheme.

  6. Rather than asking for the lowest price that the subject is willing to pay for prospect \(y_E g\), this mechanism can also be implemented by letting subjects complete a menu list of choices between a sure amount \(q\) and the prospect \(y_Eg\), where \(q\) is increasing for each choice. At the end, one decision is randomly selected for payment. The certainty equivalent is the value of \(q\) where the subject switches from the lottery to the sure amount.

    The mechanism can also be presented as a scoring rule. Let \(u\left( z\right) \) be the utility of prize \(z\). Then \(S\left( r,1\right) =P\left( Z\le r\right) u\left( y\right) +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \) and \(S\left( r,0\right) =P\left( Z\le r\right) u\left( g\right) +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \) so that \(ES\left( r,X\right) =P\left( Z\le r\right) \left[ pu\left( y\right) +\left( 1-p\right) u\left( g\right) \right] +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \).

  7. If \(F(x)\) can have discontinuities the general definition is \(F(x_1)<\alpha \) for all \(x_1<x\) and \(F(x)\ge \alpha \).

  8. Harrison et al. (2013a), however, show that risk aversion poses less of a problem for the QSR when it is used to elicit the distribution of a continuous event rather than a binary probability.

  9. There seems to be some confusion about the origin of this idea. Smith (1961) says the idea is ‘adapted from Savage (1954)’, but Savage (1971) attributes the idea to Smith.

  10. All definitions above immediately extend to randomized payment schemes, where the payment to the subject is a realization of some random variable. Here \(S:\varTheta \times {\mathcal{X}}\rightarrow \Delta {\mathbb{R}}\) where \(\Delta {\mathbb{R}}\) denotes the set of distributions over \({\mathbb{R}}\).

  11. Again, this method can implemented with a menu list, see Footnote 5. A problem arises when subjects do not have an unique switching threshold. Heinemann et al. (2009) exclude such subjects.

  12. Vlek (1973b) points out that even with hypothetical payoffs these mechanisms may still matter because they encourage subjects to think in a particular way and may align the preferences of experimenter and subject. An appropriate feedback rule can clarify what the experimenter really wants to know and avoid wrong interpretations. One example of misinterpretations by the subject comes from the elicitation of confidence intervals. Yaniv and Foster (1995, 1997) show that subjects seem to think that 50 % confidence intervals strike the right balance between accuracy and preciseness even when the requested level of confidence is much larger. This interpretation casts doubts on the widespread interpretation that intervals that are too narrow are a sign of ‘overconfidence’. Krawczyk (2011) shows that using incentives for interval elicitation improves the level of calibration of subjects. Winkler and Murphy (1968) provide a discussion of scoring rules as learning devices.

  13. Hurley and Shogren (2005), for example, argue that their inability to recover induced beliefs with a belief elicitation procedure stems from a failure to induce correct beliefs rather than a failure of the elicitation process.

  14. For example, in the context of public goods games, a deeper understanding of the game may have very different implications for selfish individuals (who would reduce contributions) or altruistic individuals (who would increase contributions). Indeed, in the public good game of Gächter and Renner (2010), the interpretation that elicitation improves understanding rests on the assumption that people are conditional cooperators. Note that in this study, the statistical effect is weak and the results are also consistent with a consensus effect or the use of stated beliefs to justify (selfish) actions.

  15. Most of the literature discussed so far is based on the decision theoretic approach by Savage (1954), where subjective utilities are a primitive concept used in evaluating uncertain prospects. In contrast, psychologists have argued that choices may affect beliefs. A discussion of the merits of these approaches is beyond the scope of this paper and we limit ourselves discussing the empirical effect of elicitation on responses. Costa-Gomes et al. (2012) and Smith (2013) use an instrumental variable approach to identify a causal relationship between beliefs and actions.

  16. Jaffray and Karni (1999) present mechanisms that can overcome these problems, which require either additional elicitation tasks, or the payment of very large sums of money to exploit the domain where the utility function is relatively flat.

  17. In Offerman et al. (1996), 50 % of the subjects indicate that they would have reported different beliefs in the absence of incentives, often deviating to an ‘easier’ report.

  18. Note that these conditions apply to most studies testing incentivized elicitation schemes, where belief elicitation is typically the only experimental task and thus receives full attention of the subjects. Therefore, these studies may understate effects of incentives in other, more complex, experimental settings (see the comments in Sonnemans and Offerman 2001).

References

  • Allen, F. (1987). Discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544.

    Article  Google Scholar 

  • Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities. Journal of Risk and Uncertainty, 48(3), 207–220.

  • Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle. International Economic Review, 50(4), 1079–1102.

    Article  Google Scholar 

  • Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. European Economic Review, 62, 17–40.

    Article  Google Scholar 

  • Artinger, F., Exadaktylos, F., Koppel, H., & Sääksvuori, L. (2010). Applying Quadratic Scoring Rule transparently in multiple choice settings: a note. Technical report, Jena Economic Research Paper.

  • Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75(3), 354–359.

    Article  Google Scholar 

  • Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions. Journal of Experimental Psychology, 81(3), 561–565.

    Article  Google Scholar 

  • Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226.

    Article  Google Scholar 

  • Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences. Handbook of Experimental Economics Results, 1, 1087–1097.

    Article  Google Scholar 

  • Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem? Experimental Economics, 13(4), 412–438.

    Article  Google Scholar 

  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.

    Article  Google Scholar 

  • Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 513–519). Oxford: Oxford University Press.

    Google Scholar 

  • Costa-Gomes, M. A., Huck, S. and Weizsacker, G. (2012). Beliefs and actions in the trust game: creating instrumental variables to estimate the causal effect. WZB Discussion Paper, 2012–302.

  • Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play. Journal of Economic Behavior & Organization, 41(3), 299–314.

    Article  Google Scholar 

  • Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation. Journal of Personality and Social Psychology, 35(1), 1.

    Article  Google Scholar 

  • De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 18(1), 87–123.

    Article  Google Scholar 

  • De Finetti, B. (1970). Logical foundations and measurement of subjective probability. Acta Psychologica, 34, 129–145.

    Article  Google Scholar 

  • De Finetti, B. (1974). Theory of probability (Vol. 1). New York: Wiley.

    Google Scholar 

  • Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence. Journal of Development Economics, 94(2), 151–163.

    Article  Google Scholar 

  • Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment. Organizational Behavior and Human Performance, 10, 108–117.

    Article  Google Scholar 

  • Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given. Experimental Economics, 260(2000), 241–260.

    Article  Google Scholar 

  • Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality. Organizational Behavior and Human Decision Processes, 55, 78–94.

    Article  Google Scholar 

  • Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance, 29(3), 352–369.

    Article  Google Scholar 

  • Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict? Applied Economics Letters, 18(3), 267–272.

    Article  Google Scholar 

  • Friedman, D. (1983). Effective scoring rules for probabilistic forecasts. Management Science, 29(4), 447–454.

    Article  Google Scholar 

  • Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377.

    Article  Google Scholar 

  • Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701.

    Article  Google Scholar 

  • Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704.

    Article  Google Scholar 

  • Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

    Article  Google Scholar 

  • Good, I. (1952). Rational decisions. Journal of the Royal Statistical Society. Series B, 14(1), 107–114.

    Google Scholar 

  • Grether, D. (1981). Financial incentive effects and individual decision-making. California Institute of Technology, Working Paper 401.

  • Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs. Journal of Economic Behavior & Organization, 55(1), 25–30.

    Article  Google Scholar 

  • Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study. Journal of Risk and Uncertainty, 2, 161–180.

    Article  Google Scholar 

  • Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior and Organization, 101, 128–140.

    Article  Google Scholar 

  • Harrison, G. W., Martínez-correa, J., Swarthout, J. T., & Ulm, E. R. (2013a). Scoring rules for subjective probability distributions. Manuscript, Georgia State University.

  • Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior and Organization, 94, 145–159.

    Article  Google Scholar 

  • Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral. Experimental Economics, 12(2), 133–158.

    Article  Google Scholar 

  • Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games. Review of Economic Studies, 76, 181–221.

    Article  Google Scholar 

  • Hoffmann, T. (2013). The Effect of Belief Elicitation on Game Play. Manuscript, Mannheim University.

  • Hollard, G., Massoni, S. and Vergnaud, J. (2010). Subjective beliefs formation and elicitation rules : Experimental evidence. Centre d’Economie de la Sorbonne Working Paper, 2010.88.

  • Holt, C. (2006). Markets, games and strategic behavior. Boston: Pearson/Addison-Wesley.

    Google Scholar 

  • Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects. The American Economic Review, 92(5), 1644.

    Article  Google Scholar 

  • Hossain, T., & Okui, R. (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001.

    Article  Google Scholar 

  • Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs. Journal of Economic Behavior & Organization, 47(1), 71–85.

    Article  Google Scholar 

  • Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs. Journal of Risk and Uncertainty, 30(2), 169–188.

    Article  Google Scholar 

  • Hurley, T. T. M., Peterson, N., & Shogren, J. J. F. (2007). Belief elicitation: An experimental comparison of scoring rule and prediction methods. Manuscript, University of Minnesota.

  • Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable. Journal of Risk and Uncertainty, 8, 5–20.

    Article  Google Scholar 

  • Jenkinson, D. (2005). The elicitation of probabilities—A review of the statistical literature. Manuscript, University of Sheffield.

  • Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules. Organizational Behavior and Human Performance, 9(2), 307–317.

    Article  Google Scholar 

  • Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments. Operations Research, 57(5), 1287–1297.

    Article  Google Scholar 

  • Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities. Journal of the American Statistical Association, 83(402), 357–363.

    Article  Google Scholar 

  • Karni, E. (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606.

    Article  Google Scholar 

  • Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities. Theory and Decision, 38, 313–320.

    Article  Google Scholar 

  • Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets. Journal of Economic Behavior & Organization, 83(2), 195–208.

    Article  Google Scholar 

  • Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. Journal of Multi-Criteria Decision Analysis, 101, 101–113.

    Google Scholar 

  • Krawczyk, M. (2011). Overconfident for real? Proper scoring for confidence intervals. Manuscript, University of Warsaw.

  • Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights. ACM SIGecom Exchanges, 7(3), 1–5.

    Article  Google Scholar 

  • Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments. Journal of the Royal Statistical Society. Series A (General), 142(2), 146.

    Article  Google Scholar 

  • Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44.

    Article  Google Scholar 

  • Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780.

    Article  Google Scholar 

  • Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response. European Economic Review, 46, 880–891.

    Article  Google Scholar 

  • Manski, C. (2004). Measuring expectations. Econometrica, 72(5), 1329–1376.

    Article  Google Scholar 

  • Manski, C. F. (2006). Interpreting the predictions of prediction markets. Economics Letters, 91(3), 425–429.

    Article  Google Scholar 

  • Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096.

    Article  Google Scholar 

  • McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences of the United States of America, 42(9), 654–655.

    Article  Google Scholar 

  • McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling. Econometrica, 58(6), 1321–1339.

    Article  Google Scholar 

  • Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science, 51(9), 1359–1373.

    Article  Google Scholar 

  • Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica, 70(3), 971–1005.

    Article  Google Scholar 

  • Offerman, T., & Palley, A. B. (2013). Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents. Manuscript, University of Amsterdam.

  • Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods. The Economic Journal, 106(437), 817–845.

    Article  Google Scholar 

  • Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians. Review of Economic Studies, 76(4), 1461–1489.

    Article  Google Scholar 

  • Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games. Journal of Economic Behavior & Organization, 71(2), 98–109.

    Article  Google Scholar 

  • Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72(3), 346–354.

    Article  Google Scholar 

  • Prelec, D. (2004). A bayesian truth serum for subjective data. Science, 306, 462–466.

    Article  Google Scholar 

  • Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes, 76(3), 277–297.

    Article  Google Scholar 

  • Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.), The foundations of mathematics and other logical essays (pp. 156–198). New York (1931): Harcourt.

  • Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory.

  • Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games and Economic Behavior, 67(2), 616–632.

    Article  Google Scholar 

  • Savage, L. J. (1954). The foundation of statistics. New York: Wiley.

    Google Scholar 

  • Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801.

    Article  Google Scholar 

  • Schervish, M. (1989). A general method for comparing probability assessors. The Annals of Statistics, 17(4), 1856–1879.

    Article  Google Scholar 

  • Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224.

  • Schlag, K. H., & van der Weele, J. J. (2012). Incentives for interval elicitation. Manuscript, Vienna University.

  • Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Economics Letters, 03(1), 38–42.

    Article  Google Scholar 

  • Schmalensee, R. (1976). An experimental study of expectation formation. Econometrica, 44(1), 17–41.

    Article  Google Scholar 

  • Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128.

    Article  Google Scholar 

  • Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements. Organizational Behavior and Human Performance, 2, 84–104.

    Article  Google Scholar 

  • Seghers, R. C., Fryback, D. G., & Goodman, B. C. (1973). Relative variance preferences in a choice-among-bets paradigm. Technical report, DTIC Document.

  • Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 62, 43–62.

    Article  Google Scholar 

  • Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory and Decision, 46, 211–249.

    Article  Google Scholar 

  • Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures. Psychometrika, 31(2), 125–145.

    Article  Google Scholar 

  • Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games. Experimental Economics, 16(3), 414–425.

    Article  Google Scholar 

  • Smith, C. (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B, 23(1), 1–37.

    Google Scholar 

  • Sonnemans, J., & Offerman, T. T. T. (2001). Is the quadratic scoring rule behaviorally incentive compatible?. Manuscript, University of Amsterdam.

  • Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146–159.

    Article  Google Scholar 

  • Toda, M. (1951). Measurement of intuitive probability by a method of game. Japanese Journal of Psychology, 22, 29–40.

    Article  Google Scholar 

  • Toda, M. (1963). Measurement of subjective probability distribution. Report No. 3(3), State College, Pennsylvania, Institute for Research, Division of Mathematical Psychology.

  • Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums. The Economic Journal. doi:10.1111/ecoj.12160.

  • Tversky, A., & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101(4), 547.

    Article  Google Scholar 

  • Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities. British Journal of Mathematical and Statistical Psychology, 26(1), 18–30.

    Article  Google Scholar 

  • Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment. Organizational Behavior and Human Performance, 9(460–481), 460–481.

    Article  Google Scholar 

  • Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 32(1), 38–43.

    Article  Google Scholar 

  • Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters, 111(1), 30–33.

    Article  Google Scholar 

  • Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time. Journal of Behavioral Decision Making, 6(4), 283–296.

    Article  Google Scholar 

  • Wilcox, N. T. and Feltovich, N. (2000). Thinking like a game theorist: Comment. University of Houston Department of Economics working paper.

  • Winkler, R. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60.

    Article  Google Scholar 

  • Winkler, R., & Murphy, A. (1968). ”Good” probability assessors. Journal of Applied Meteorology, 7, 751.

    Article  Google Scholar 

  • Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score. Journal of Applied Meteorology, 9, 143–148.

    Article  Google Scholar 

  • Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures. The Meteorological Magazine, 108(1288), 317–329.

    Google Scholar 

  • Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200.

  • Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General, 124(4), 424–432.

    Article  Google Scholar 

  • Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation. Journal of Behavioral Decision Making, 10, 21–32.

    Article  Google Scholar 

  • Zizzo, D. J. (2009). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75–98.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Peter Wakker, Theo Offerman, Glenn Harrison, Gerhard Sorger and two anonymous referees for useful comments.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Joël J. van der Weele.

Appendix

Appendix

Proof

(Proof of Proposition 1) We use the characterization of Schervish (1989). To simplify exposition assume that \(\nu \) has no point masses and admits a piecewise continuous density \(f\), hence \(S\left( r,1\right) =S\left( 1,1\right) -\int _{r}^{1}\left( 1-c\right) f\left( c\right) dc\) and \(S\left( r,0\right) =S\left( 0,0\right) -\int _{0}^{r}cf\left( c\right) dc\). Consequently, if \(EX=p\) then

$$\begin{aligned} E\left( r,X\right) =pS\left( 1,1\right) +\left( 1-p\right) S\left( 0,0\right) -p\int _{r}^{1}\left( 1-c\right) \,f\,\left( c\right) \, dc-\left( 1-p\right) \int _{0}^{r}cf\,\left( c\right) \, dc \end{aligned}$$

and

$$\begin{aligned} \frac{d}{dr}ES\left( r,X\right) =\left( p-r\right) f\left( r\right) . \end{aligned}$$

So \(f\left( r\right) \) describes the strength of the local incentives to tell the truth for reports that are close to \(r\).

Now note that

$$\begin{aligned} ES\left( 1,X\right) -ES\left( 0,X\right)&= -\left( 1-p\right) \int _{0}^{1}cf\,\left( c\right) \, dc+p\int _{0}^{1}\left( 1-c\right) \, f\,\left( c\right) \, dc \nonumber \\&= \int _{0}^{1}\left( p-c\right) \, f\left( c\right) \, dc \nonumber \\&\le \int _{0}^{1}\left( 1-c\right) \, f\,\left( c\right) \, dc. \end{aligned}$$
(10)

Assume now w.l.o.g. that the scoring rule gives payoffs in \(\left[ 0,k\right] \) (i.e. \(\omega _1=0\) and \(\omega _2=k\)). For instance, the quadratic scoring rule would be represented as \(S^{QSR}\left( r,1\right) =k\left( 1-\left( 1-r\right) ^{2}\right) \) and \(S^{QSR}\left( r,0\right) =k\left( 1-r^{2}\right) \). It is easy to show that for the QSR, \(f(r)=2k\). Note that

$$\begin{aligned} ES\left( 1,X\right) -ES\left( 0,X\right) \le k=2k\int _{0}^{1}\left( 1-c\right) \, dc. \end{aligned}$$
(11)

Comparing (10) and (11) it follows that \(f\equiv 2k\) if \(f\left( c\right) \ge 2k\) for all \(c\).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schlag, K.H., Tremewan, J. & van der Weele, J.J. A penny for your thoughts: a survey of methods for eliciting beliefs. Exp Econ 18, 457–490 (2015). https://doi.org/10.1007/s10683-014-9416-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10683-014-9416-x

Keywords

JEL Classification

Navigation