## Abstract

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.

This is a preview of subscription content, access via your institution.

## Notes

Here we assume that \(\theta \) is uniquely determined given \(X\), so \(\theta =\theta \left( X\right) \). Definitions become a bit more involved when the characteristic of interest is not always uniquely defined, see Sect. 2.7.

Truth-telling applies the concept of incentive compatibility to belief elicitation. Kothiyal et al. (2011) point out that the earliest incentive compatible scoring rules were proposed at least a decade before the first work on mechanism design was published.

\({\mathbf{1}}_{\left\{ r\le c\right\} }=1\) if \(r\le c\) and \(=0\) if \(r>c\). \({\mathbf{1}}_{\left\{ r>c\right\} }\) is defined similarly.

This scheme provide the same incentives as the QSR, and provides the same payoffs if the prize is given with certainty if the event does not occur.

McCarthy (1956) mentions this claim and attributes it to Gleason (unpublished). Since we were unable to locate the latter study, we prove this here for \(n=2\), using the framework of Savage (1971). We search for a rule such that \(Y\left( r\right) =Z\left( 1-r\right) \) for all \(r\), and hence \(J^{\prime }\left( p\right) =Y\left( p\right) -Z\left( p\right) =Y\left( p\right) -Y\left( 1-p\right) \). Since \(J\left( p\right) =Y\left( p\right) p+Y\left( 1-p\right) \left( 1-p\right) \) we obtain \(J^{\prime }\left( p\right) =Y\left( p\right) -Y\left( 1-p\right) +Y^{\prime }\left( p\right) p-Y^{\prime }\left( 1-p\right) \left( 1-p\right) \) and hence \(Y^{\prime }\left( p\right) p=Y^{\prime }\left( 1-p\right) \left( 1-p\right) \) for all \(p\). This implies that \(Y\left( p\right) =a\ln p+b\) for some \(a>0\) and \(b\) and hence \(S\) is an affine transformation of the the logarithmic payment scheme.

Rather than asking for the lowest price that the subject is willing to pay for prospect \(y_E g\), this mechanism can also be implemented by letting subjects complete a menu list of choices between a sure amount \(q\) and the prospect \(y_Eg\), where \(q\) is increasing for each choice. At the end, one decision is randomly selected for payment. The certainty equivalent is the value of \(q\) where the subject switches from the lottery to the sure amount.

The mechanism can also be presented as a scoring rule. Let \(u\left( z\right) \) be the utility of prize \(z\). Then \(S\left( r,1\right) =P\left( Z\le r\right) u\left( y\right) +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \) and \(S\left( r,0\right) =P\left( Z\le r\right) u\left( g\right) +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \) so that \(ES\left( r,X\right) =P\left( Z\le r\right) \left[ pu\left( y\right) +\left( 1-p\right) u\left( g\right) \right] +\int _{r}^{\infty }u\left( z\right) dP_{Z}\left( z\right) \).

If \(F(x)\) can have discontinuities the general definition is \(F(x_1)<\alpha \) for all \(x_1<x\) and \(F(x)\ge \alpha \).

Harrison et al. (2013a), however, show that risk aversion poses less of a problem for the QSR when it is used to elicit the distribution of a continuous event rather than a binary probability.

All definitions above immediately extend to randomized payment schemes, where the payment to the subject is a realization of some random variable. Here \(S:\varTheta \times {\mathcal{X}}\rightarrow \Delta {\mathbb{R}}\) where \(\Delta {\mathbb{R}}\) denotes the set of distributions over \({\mathbb{R}}\).

Again, this method can implemented with a menu list, see Footnote 5. A problem arises when subjects do not have an unique switching threshold. Heinemann et al. (2009) exclude such subjects.

Vlek (1973b) points out that even with hypothetical payoffs these mechanisms may still matter because they encourage subjects to think in a particular way and may align the preferences of experimenter and subject. An appropriate feedback rule can clarify what the experimenter really wants to know and avoid wrong interpretations. One example of misinterpretations by the subject comes from the elicitation of confidence intervals. Yaniv and Foster (1995, 1997) show that subjects seem to think that 50 % confidence intervals strike the right balance between accuracy and preciseness even when the requested level of confidence is much larger. This interpretation casts doubts on the widespread interpretation that intervals that are too narrow are a sign of ‘overconfidence’. Krawczyk (2011) shows that using incentives for interval elicitation improves the level of calibration of subjects. Winkler and Murphy (1968) provide a discussion of scoring rules as learning devices.

Hurley and Shogren (2005), for example, argue that their inability to recover induced beliefs with a belief elicitation procedure stems from a failure to induce correct beliefs rather than a failure of the elicitation process.

For example, in the context of public goods games, a deeper understanding of the game may have very different implications for selfish individuals (who would reduce contributions) or altruistic individuals (who would increase contributions). Indeed, in the public good game of Gächter and Renner (2010), the interpretation that elicitation improves understanding rests on the assumption that people are conditional cooperators. Note that in this study, the statistical effect is weak and the results are also consistent with a consensus effect or the use of stated beliefs to justify (selfish) actions.

Most of the literature discussed so far is based on the decision theoretic approach by Savage (1954), where subjective utilities are a primitive concept used in evaluating uncertain prospects. In contrast, psychologists have argued that choices may affect beliefs. A discussion of the merits of these approaches is beyond the scope of this paper and we limit ourselves discussing the empirical effect of

*elicitation*on responses. Costa-Gomes et al. (2012) and Smith (2013) use an instrumental variable approach to identify a causal relationship between beliefs and actions.Jaffray and Karni (1999) present mechanisms that can overcome these problems, which require either additional elicitation tasks, or the payment of very large sums of money to exploit the domain where the utility function is relatively flat.

In Offerman et al. (1996), 50 % of the subjects indicate that they would have reported different beliefs in the absence of incentives, often deviating to an ‘easier’ report.

Note that these conditions apply to most studies testing incentivized elicitation schemes, where belief elicitation is typically the only experimental task and thus receives full attention of the subjects. Therefore, these studies may understate effects of incentives in other, more complex, experimental settings (see the comments in Sonnemans and Offerman 2001).

## References

Allen, F. (1987). Discovering personal probabilities when utility functions are unknown.

*Management Science*,*33*(4), 542–544.Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities.

*Journal of Risk and Uncertainty*,*48*(3), 207–220.Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle.

*International Economic Review*,*50*(4), 1079–1102.Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging.

*European Economic Review*,*62*, 17–40.Artinger, F., Exadaktylos, F., Koppel, H., & Sääksvuori, L. (2010). Applying Quadratic Scoring Rule transparently in multiple choice settings: a note. Technical report, Jena Economic Research Paper.

Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets.

*Journal of Experimental Psychology*,*75*(3), 354–359.Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions.

*Journal of Experimental Psychology*,*81*(3), 561–565.Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method.

*Behavioral Science*,*9*(3), 226.Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences.

*Handbook of Experimental Economics Results*,*1*, 1087–1097.Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem?

*Experimental Economics*,*13*(4), 412–438.Brier, G. W. (1950). Verification of forecasts expressed in terms of probability.

*Monthly Weather Review*,*78*(1), 1–3.Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.),

*Bayesian statistics 5*(pp. 513–519). Oxford: Oxford University Press.Costa-Gomes, M. A., Huck, S. and Weizsacker, G. (2012). Beliefs and actions in the trust game: creating instrumental variables to estimate the causal effect. WZB Discussion Paper, 2012–302.

Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play.

*Journal of Economic Behavior & Organization*,*41*(3), 299–314.Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation.

*Journal of Personality and Social Psychology*,*35*(1), 1.De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item.

*British Journal of Mathematical and Statistical Psychology*,*18*(1), 87–123.De Finetti, B. (1970). Logical foundations and measurement of subjective probability.

*Acta Psychologica*,*34*, 129–145.De Finetti, B. (1974).

*Theory of probability*(Vol. 1). New York: Wiley.Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence.

*Journal of Development Economics*,*94*(2), 151–163.Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment.

*Organizational Behavior and Human Performance*,*10*, 108–117.Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given.

*Experimental Economics*,*260*(2000), 241–260.Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality.

*Organizational Behavior and Human Decision Processes*,*55*, 78–94.Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting.

*Organizational Behavior and Human Performance*,*29*(3), 352–369.Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict?

*Applied Economics Letters*,*18*(3), 267–272.Friedman, D. (1983). Effective scoring rules for probabilistic forecasts.

*Management Science*,*29*(4), 447–454.Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments.

*Experimental Economics*,*13*(3), 364–377.Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions.

*Journal of the American Statistical Association*,*100*(470), 680–701.Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats.

*Psychological Review*,*102*(4), 684–704.Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation.

*Journal of the American Statistical Association*,*102*(477), 359–378.Good, I. (1952). Rational decisions.

*Journal of the Royal Statistical Society. Series B*,*14*(1), 107–114.Grether, D. (1981). Financial incentive effects and individual decision-making. California Institute of Technology, Working Paper 401.

Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs.

*Journal of Economic Behavior & Organization*,*55*(1), 25–30.Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study.

*Journal of Risk and Uncertainty*,*2*, 161–180.Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries.

*Journal of Economic Behavior and Organization*,*101*, 128–140.Harrison, G. W., Martínez-correa, J., Swarthout, J. T., & Ulm, E. R. (2013a).

*Scoring rules for subjective probability distributions*. Manuscript, Georgia State University.Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration.

*Journal of Economic Behavior and Organization*,*94*, 145–159.Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral.

*Experimental Economics*,*12*(2), 133–158.Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games.

*Review of Economic Studies*,*76*, 181–221.Hoffmann, T. (2013).

*The Effect of Belief Elicitation on Game Play*. Manuscript, Mannheim University.Hollard, G., Massoni, S. and Vergnaud, J. (2010). Subjective beliefs formation and elicitation rules : Experimental evidence. Centre d’Economie de la Sorbonne Working Paper, 2010.88.

Holt, C. (2006).

*Markets, games and strategic behavior*. Boston: Pearson/Addison-Wesley.Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects.

*The American Economic Review*,*92*(5), 1644.Hossain, T., & Okui, R. (2013). The binarized scoring rule.

*The Review of Economic Studies*,*80*(3), 984–1001.Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs.

*Journal of Economic Behavior & Organization*,*47*(1), 71–85.Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs.

*Journal of Risk and Uncertainty*,*30*(2), 169–188.Hurley, T. T. M., Peterson, N., & Shogren, J. J. F. (2007).

*Belief elicitation: An experimental comparison of scoring rule and prediction methods*. Manuscript, University of Minnesota.Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable.

*Journal of Risk and Uncertainty*,*8*, 5–20.Jenkinson, D. (2005).

*The elicitation of probabilities—A review of the statistical literature*. Manuscript, University of Sheffield.Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules.

*Organizational Behavior and Human Performance*,*9*(2), 307–317.Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments.

*Operations Research*,*57*(5), 1287–1297.Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities.

*Journal of the American Statistical Association*,*83*(402), 357–363.Karni, E. (2009). A mechanism for eliciting probabilities.

*Econometrica*,*77*(2), 603–606.Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities.

*Theory and Decision*,*38*, 313–320.Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets.

*Journal of Economic Behavior & Organization*,*83*(2), 195–208.Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs.

*Journal of Multi-Criteria Decision Analysis*,*101*, 101–113.Krawczyk, M. (2011).

*Overconfident for real? Proper scoring for confidence intervals*. Manuscript, University of Warsaw.Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights.

*ACM SIGecom Exchanges*,*7*(3), 1–5.Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments.

*Journal of the Royal Statistical Society. Series A (General)*,*142*(2), 146.Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples.

*Medical Decision Making*,*21*, 37–44.Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability.

*Econometrica*,*60*(4), 745–780.Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response.

*European Economic Review*,*46*, 880–891.Manski, C. (2004). Measuring expectations.

*Econometrica*,*72*(5), 1329–1376.Manski, C. F. (2006). Interpreting the predictions of prediction markets.

*Economics Letters*,*91*(3), 425–429.Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions.

*Management Science*,*22*(10), 1087–1096.McCarthy, J. (1956). Measures of the value of information.

*Proceedings of the National Academy of Sciences of the United States of America*,*42*(9), 654–655.McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling.

*Econometrica*,*58*(6), 1321–1339.Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method.

*Management Science*,*51*(9), 1359–1373.Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs.

*Econometrica*,*70*(3), 971–1005.Offerman, T., & Palley, A. B. (2013).

*Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents*. Manuscript, University of Amsterdam.Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods.

*The Economic Journal*,*106*(437), 817–845.Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians.

*Review of Economic Studies*,*76*(4), 1461–1489.Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games.

*Journal of Economic Behavior & Organization*,*71*(2), 98–109.Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task.

*Journal of Experimental Psychology*,*72*(3), 346–354.Prelec, D. (2004). A bayesian truth serum for subjective data.

*Science*,*306*, 462–466.Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence.

*Organizational Behavior and Human Decision Processes*,*76*(3), 277–297.Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.),

*The foundations of mathematics and other logical essays*(pp. 156–198). New York (1931): Harcourt.Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory.

Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test.

*Games and Economic Behavior*,*67*(2), 616–632.Savage, L. J. (1954).

*The foundation of statistics*. New York: Wiley.Savage, L. J. (1971). Elicitation of personal probabilities and expectations.

*Journal of the American Statistical Association*,*66*(336), 783–801.Schervish, M. (1989). A general method for comparing probability assessors.

*The Annals of Statistics*,*17*(4), 1856–1879.Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224.

Schlag, K. H., & van der Weele, J. J. (2012).

*Incentives for interval elicitation*. Manuscript, Vienna University.Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality.

*Theoretical Economics Letters*,*03*(1), 38–42.Schmalensee, R. (1976). An experimental study of expectation formation.

*Econometrica*,*44*(1), 17–41.Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory.

*Annual Review of Economics*,*6*, 103–128.Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements.

*Organizational Behavior and Human Performance*,*2*, 84–104.Seghers, R. C., Fryback, D. G., & Goodman, B. C. (1973). Relative variance preferences in a choice-among-bets paradigm. Technical report, DTIC Document.

Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule.

*Experimental Economics*,*62*, 43–62.Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse.

*Theory and Decision*,*46*, 211–249.Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures.

*Psychometrika*,*31*(2), 125–145.Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games.

*Experimental Economics*,*16*(3), 414–425.Smith, C. (1961). Consistency in statistical inference and decision.

*Journal of the Royal Statistical Society. Series B*,*23*(1), 1–37.Sonnemans, J., & Offerman, T. T. T. (2001).

*Is the quadratic scoring rule behaviorally incentive compatible?*. Manuscript, University of Amsterdam.Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability.

*Acta Psychologica*,*34*, 146–159.Toda, M. (1951). Measurement of intuitive probability by a method of game.

*Japanese Journal of Psychology*,*22*, 29–40.Toda, M. (1963). Measurement of subjective probability distribution. Report No. 3(3), State College, Pennsylvania, Institute for Research, Division of Mathematical Psychology.

Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums.

*The Economic Journal*. doi:10.1111/ecoj.12160.Tversky, A., & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability.

*Psychological Review*,*101*(4), 547.Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities.

*British Journal of Mathematical and Statistical Psychology*,*26*(1), 18–30.Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment.

*Organizational Behavior and Human Performance*,*9*(460–481), 460–481.Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models.

*IEEE Transactions on Systems, Man, and Cybernetics, Part B*,*32*(1), 38–43.Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups.

*Economics Letters*,*111*(1), 30–33.Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time.

*Journal of Behavioral Decision Making*,*6*(4), 283–296.Wilcox, N. T. and Feltovich, N. (2000). Thinking like a game theorist: Comment. University of Houston Department of Economics working paper.

Winkler, R. (1996). Scoring rules and the evaluation of probabilities.

*Test*,*5*(1), 1–60.Winkler, R., & Murphy, A. (1968). ”Good” probability assessors.

*Journal of Applied Meteorology*,*7*, 751.Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score.

*Journal of Applied Meteorology*,*9*, 143–148.Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures.

*The Meteorological Magazine*,*108*(1288), 317–329.Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200.

Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off.

*Journal of Experimental Psychology: General*,*124*(4), 424–432.Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation.

*Journal of Behavioral Decision Making*,*10*, 21–32.Zizzo, D. J. (2009). Experimenter demand effects in economic experiments.

*Experimental Economics*,*13*(1), 75–98.

## Acknowledgments

We would like to thank Peter Wakker, Theo Offerman, Glenn Harrison, Gerhard Sorger and two anonymous referees for useful comments.

## Author information

### Authors and Affiliations

### Corresponding author

## Appendix

### Appendix

###
*Proof*

(*Proof of Proposition* 1) We use the characterization of Schervish (1989). To simplify exposition assume that \(\nu \) has no point masses and admits a piecewise continuous density \(f\), hence \(S\left( r,1\right) =S\left( 1,1\right) -\int _{r}^{1}\left( 1-c\right) f\left( c\right) dc\) and \(S\left( r,0\right) =S\left( 0,0\right) -\int _{0}^{r}cf\left( c\right) dc\). Consequently, if \(EX=p\) then

and

So \(f\left( r\right) \) describes the strength of the local incentives to tell the truth for reports that are close to \(r\).

Now note that

Assume now w.l.o.g. that the scoring rule gives payoffs in \(\left[ 0,k\right] \) (i.e. \(\omega _1=0\) and \(\omega _2=k\)). For instance, the quadratic scoring rule would be represented as \(S^{QSR}\left( r,1\right) =k\left( 1-\left( 1-r\right) ^{2}\right) \) and \(S^{QSR}\left( r,0\right) =k\left( 1-r^{2}\right) \). It is easy to show that for the QSR, \(f(r)=2k\). Note that

Comparing (10) and (11) it follows that \(f\equiv 2k\) if \(f\left( c\right) \ge 2k\) for all \(c\).

## Rights and permissions

## About this article

### Cite this article

Schlag, K.H., Tremewan, J. & van der Weele, J.J. A penny for your thoughts: a survey of methods for eliciting beliefs.
*Exp Econ* **18**, 457–490 (2015). https://doi.org/10.1007/s10683-014-9416-x

Received:

Revised:

Accepted:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s10683-014-9416-x

### Keywords

- Belief elicitation
- Subjective beliefs
- Scoring rules
- Experimental design

### JEL Classification

- C83
- C91
- D83