Abstract
This paper describes a general principle that can be used to elicit honest opinions, even if risk attitudes are unknown.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Several mechanisms, often called proper scoring rules, incentivize an expert to honestly report his subjective probabilities. However, these mechanisms often assume that the expert is risk neutral or that the expert’s preferences over risk are known. Recently, Karni (2009) showed a novel mechanism that induces truth reporting even if the expert’s preferences over risk are not known. Lambert (2011) characterizes the mechanisms that induce truthful-revelation with unknown preferences. In this paper, we describe a very simple principle that can also be used to dispose the assumption that the expert’s preferences are known when beliefs are elicited.
There are two monetary rewards. The expert receives the greater reward with odds proportional to a proper scoring rule. Payments based on this principle delivers incentives for an expert to truthfully reveal what he knows, even if his attitudes toward risk are unknown.
We present two examples to illustrate the use of this principle. In the first example, we consider a traditional proper scoring rules (the Brier Score). In the second example, we consider Prelec (2004) “Truth Serum” which elicits honest opinions in the absence of any data. In both cases, the original mechanism is based on known preferences. We show how to transform these mechanisms, with the principle mentioned above, so that the assumption of risk-neutrality (or known attitudes towards risk) are disposed. While we provide no general results, we hope that these two examples illustrate the basic principle clearly.
2 The principle of eliciting beliefs by paying in chance
An event \(E\) may or may not occur next period. An expert’s (Bob) subjective probability of \(E\) is of interest. The question is how to elicit the Bob’s subjective probability \(p\) of event \(E\). Let’s say that the expert is offered the following monetary rewards
This payment scheme \(S\) is a Brier Score, Brier (1950). The critical property of the Brier Score is that it is proper. That is, if the expert maximizes his expected payoff
then he optimally chooses \(\hat{p}=p.\) De Finetti (1962) describes proper scoring rules as devices that “oblige each participant to express his true feelings, because any departure from his own personal probability results in a diminution of his own average score as he sees it”. That is, a proper score induces a risk-neutral expert to reveal his subjective beliefs. Several proper scoring rules exist [see Bernardo (1979); Good (1997); Hendrickson and Buehler (1971); Kiefer (2010); Lindley (1982); Matheson and Winkler (1976); McCarthy (1956); Savage (1971); Schervish (1989); Van Lenthe (1993); Winkler (1994)].
The main difficulty with these proper scoring rules is that they assume risk-neutrality or, alternatively, that the risk-attitudes of the expert are known [i.e., the utilities of the expert must be known if the rewards \(S(\hat{p},1)\) and \(S(\hat{p},0)\) are interpreted as utils]. If Bob is assumed to be risk-neutral when he is risk-averse, then the Brier Score may induce him to misrepresent his views [see Winkler and Murphy (1970); Kadane and Winkler (1988)]. Hence, it is significant to construct devices that oblige each participant to express his true beliefs, regardless of his attitudes toward risk.
One approach is to learn the experts’ preferences through experiments [see Offerman et al (2009); Jaffray and Karni (1999)]. Another approach was recently proposed by Karni (2009). He constructs a single-stage random scoring rule that induces Bob to honestly reveal his beliefs, regardless of his attitudes toward risk. In fact, Karni (2009) only assumes probabilistic sophistication and dominance [see Machina and Schmeidler (1995)]. So, let \(x\) and \(y\) be two monetary rewards with \(x>y\). Karni (2009) assumes that Bob prefers a lottery that delivers \(x\) dollars with probability \(\mu \) and \(y\) dollars with probability \((1-\mu )\) to a lottery that delivers \(x\) dollars with probability \(\mu ^{\prime }\) and \(y\) dollars with probability \((1-\mu ^{\prime })\) if and only if \(\mu \) is greater than \(\mu ^{\prime }\). That is, Bob prefers a higher chance of a greater monetary reward.
Now consider the following alternative random scoring rule. If Bob announces probability \(\hat{p}\) and \(E\) occurs then Bob receives the lottery
If Bob announces probability \(\hat{p}\) and \(E\) does not occur then Bob receives the lottery
where
That is, Bob receives \(x\) with odds proportional to his Brier Score (the division by \(2\) is merely to ensure that \(P(\hat{p},1)\in [0,1]\) and \(P(\hat{p},0)\in [0,1]\)). Thus, assume Bob believes that \(E\) occurs with probability \(p\) and he announces \(\hat{p}\). Then, he obtains \(x\) with probability
Thus, if \(S\) is a proper score (as in the case of the Brier Score) then it is optimal for Bob to truthfully reveals his subjective probability \(p\), regardless of his attitudes toward risk. As in Karni’s mechanism, probabilistic sophistication and dominance are the only underlying assumptions.
This example illustrates the principle of eliciting beliefs by paying in chance. The basic procedure is as follows: start with any proper score and normalize it (with some linear, strictly increasing transformation function) so that the score is always between 0 and 1. The higher monetary reward \(x\) is delivered with a probability given by the normalized proper score. So, Bob has a better chance of obtaining a greater reward if his score is higher. It then follows that it is optimal for Bob to truthfully reveals his opinions, regardless of his attitudes toward risk.
Lambert (2011) provides a characterization of scoring rules that induce truthful-revelation with unknown preferences. The random scoring rule described above is subsumed in his characterization [see also Allen (1987); Hossain and Okui (2010); Schlag and van der Weele (2010)].
It is easy to see that the principle of eliciting beliefs by paying in chance can be applied to any bounded, proper score. Like Karni (2009) mechanism, this principle can be extended to many events and many experts, by running separate mechanisms to different experts and events. In addition, it also extends in a straightforward way to the case of multiple data points.
Importantly, this principle can be used in multi-agents, game-theoretic settings. We illustrate this point with Prelec (2004) “Truth Serum”. Consider several experts who face a subjective question such as “Is Picasso the best painter of the \(20^{\text{ th }}\) century?” No objective data is observed on this matter, but each expert has an opinion (say yes or no) on it. The problem is how to induce each expert to announce his honest opinion. Prelec assumes that the opinions of each expert are realizations of a random variable with a common distribution. He designs a Bayesian game in which the action space of expert \(i,\) \(A_{i}\), is to say yes or no and a forecast of fraction of experts who answers yes. A pure strategy, for player \(i\), maps his actual opinions (yes or no) into his action space \(A_{i}\). Player \(i\)’s payoffs is given by a specific utility function
which he constructs. The key point in his paper is that there exists a Nash equilibrium in which all experts announce their opinions honestly.
As in the case of the Brier Score, if the payoffs given \(U_{i}\) are provided in monetary terms then risk-neutrality is assumed. If the payoffs are assumed to be utils then it is assumed that the expert’s attitudes toward risk are known. However, let \(\tau \) be a linear, strictly increasing function such that, for every player \(i\) and for every action profile \(a\in \prod \nolimits _{k}A_{k}\), \(\tau (U_{i}(a))\) is between \(0\) and \(1\).Footnote 1 Now consider the game in which given an action profile \(a\in \prod \nolimits _{k}A_{k}\), player \(i\) receives \(x\) with probability \(\tau (U_{i}(a))\) and \(y\) with probability \(1-\tau (U_{i}(a))\), where \(x>y\) are monetary rewards. The randomizations of nature determining between \(x\) and \(y\) is independent for each agent and each action profile. It is straightforward to show that the same truth-revealing Nash equilibrium in Prelec’s game remains a Nash equilibrium in this modified game. This follows regardless of the expert’s attitudes toward risks, provided that probabilistic sophistication and dominance are satisfied.
3 Conclusion
A simple way to elicit experts’ opinions when experts’ attitudes toward risk are unknown is to pay the expert in chance: he receives the greater reward with odds based on a proper method of eliciting beliefs that assume risk-neutrality.
Notes
This linear function always exist if the payoffs are uniformly bounded. Let’s say that \(\left| U_{i}(a)\right| \le M\) for some \(M>0\). The linear function could then be
$$\begin{aligned} \tau (z)=\frac{z+M}{2M}. \end{aligned}$$The payoffs in Prelec’s paper are based on the Kullback-Leibler measure of relative entropy and, therefore are not bounded. However, similar results can be obtained if payoffs are based on alternative loss functions, such as quadratic loss, that are bounded.
References
Allen, F.: Discovering personal probabilities when utility functions are unknown. Manag. Sci. 33, 542–544 (1987)
Bernardo, J.: Expected information as expected utility. Ann. Stat. 7, 686–690 (1979)
De Finetti, B.: Does it make sense to speak of good probability appraisals? In: Good, I.J. (ed.), The scientist speculates: an anthology of parly-baked ideas, pp. 356–364. Heineman, London (1962)
Good, I.: Rational decisions. Springer, Berlin (1997)
Hendrickson, A., Buehler, R.: Proper scores for probability forecasters. Ann. Math. Stat. 42, 1916–1921 (1971)
Hossain, T., Okui, R.: The binarized scoring rule of belief elicitation. Mimeo, UK (2010)
Jaffray, J., Karni, E.: Elicitation of subjective probabilities when the initial endowmenet is unobservable. J. Risk Uncertain. 18, 5–20 (1999)
Kadane, J., Winkler, R.: Separating probability elicitation from utilities. J. Am. Stat. Assoc. 83, 357–363 (1988)
Karni, E.: A mechanism for eliciting probabilities. Econometrica 77, 603–606 (2009)
Kiefer, N.: Incentive-compatible elicitation of quantiles. Mimeo, UK (2010)
Lambert, N.: Probability elicitation for agents with arbitrary risk preferences. Mimeo, UK (2011)
Lindley, D.: Scoring rules and the inevitability of probability. Int. Stat. Rev. 50, 1–11 (1982)
Machina, M., Schmeidler, D.: Bayes without Bernoulli: simple conditions for probabilistic sophistication choice. J. Econ. Theory 49, 479–486 (1995)
Matheson, J., Winkler, R.: Scoring rules for continuous probability distributions. Manag. Sci. 22, 1087–1096 (1976)
McCarthy, J.: Measures of the value of information. In: Proceedings of the National Academy of Sciences 42(9), pp. 654–655 (1956)
Offerman, T., Sonnemans, J., van de Kuilen, G., Wakker, P.: A truth serum for non-bayesians: correcting proper scoring rules for risk attitudes. Rev. Econ. Stud. 76, 1461–1489 (2009)
Prelec, D.: A bayesian truth serum for subjective data. Science 306(5695), 462–466 (2004)
Savage, L.: Elicitation of personal probabilities and expectations. J. Am. Stat. Assoc. 66, 783–801 (1971)
Schervish, M.: A general method for comparing probability assessors. Ann. Stat. 17, 1856–1879 (1989)
Schlag, K., van der Weele, J.: Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Mimeo, UK (2010)
Van Lenthe, J.: A blueprint of ELI, a new method for eliciting subjective probability distributions. Behav. Res. Methods Instrum. Comput. 25, 425–433 (1993)
Winkler, R.: Evaluating probabilities: asymmetric scoring rules. Manag. Sci. 40, 1395–1405 (1994)
Winkler, R., Murphy, A.: Nonlinear utility and the probability score. J. Appl. Meteorol. 9(1), 143–148 (1970)
Acknowledgments
Financial support from the National Science Foundation is gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sandroni, A., Shmaya, E. Eliciting beliefs by paying in chance. Econ Theory Bull 1, 33–37 (2013). https://doi.org/10.1007/s40505-013-0009-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40505-013-0009-1