We consider an agent who must report a subjective belief about the chances of an uncertain future event \(A\). Her true belief is that event \(A\) will occur (\(X=1\)) with probability \(p\) and its complement \({\bar{A}}\) will occur (\(X=0\)) with probability \(1-p\). She submits a reported probability \(r \in [0,1]\) that \(A\) will occur and receives a payoff according to an L-adjusted QSR, a generalization of the asymmetric QSR introduced by Winkler (1994).
Definition 1
(L-adjusted Quadratic Scoring Rule) The L-adjusted asymmetric QSR is defined by
$$\begin{aligned} S_L(X,r) = \left\{ \begin{array}{ll} \frac{(1-c)^2-(1-r)^2}{c^2 L} & {\text{if }}\; {A} \; {\text{ occurs \; and }}\;r < c, \\ \frac{c^2-r^2}{c^2} & {\text{if }}\; \bar{A} \; {\text{ occurs \; and }}\;r < c,\\ \frac{(1-c)^2-(1-r)^2}{(1-c)^2} & {\text{if }}\; {A} \; {\text{ occurs \; and }}\; r \ge c, \\ \frac{c^2-r^2}{(1-c)^2 L} & {\text{if }}\; \bar{A} \; {\text{ occurs \; and }}\; r \ge c. \end{array}\right. \end{aligned}$$
In general, the L-adjusted QSR can be centered around any baseline probability \(c\) of the event \(A\) occurring,Footnote 2 but for most of the paper we will focus on the typical case of a symmetric baseline \(c=1/2\). When \(L=1\) this scoring rule reduces to the asymmetric QSR and when \(L=1\) and \(c=1/2\) it reduces to the classical binary QSR.
The pattern of reporting behavior that previous studies have observed cannot be explained by classical expected utility theory. Therefore, to understand how an agent will respond to this risky payoff function, we apply a prospect theory model of risk preferences. Prospect theory applies psychological principles to incorporate several important and frequently observed behavioral tendencies into the neoclassical expected utility model of preferences. This more flexible formulation provides a useful descriptive model of choice under risk (Camerer 2000) and generally includes four main behavioral components:
-
1.
Reference Dependence The agent evaluates outcomes as differences relative to a reference point rather than in absolute levels.
-
2.
Loss Aversion Outcomes that fall below the reference point (“losses”) are felt more intensely than equivalent outcomes above the reference point (“gains”).
-
3.
Risk Aversion in Gains, Risk Seeking in Losses, and Diminishing Sensitivity to Both Gains and Losses The agent tends to prefer a sure moderate-sized outcome over an equal chance of a large gain or zero gain, but prefers an equal chance of taking a large loss or avoiding the loss altogether over a sure moderate-sized loss. In addition, the marginal effect of changes in the outcome for the agent diminishes as the outcome moves away from the reference point.
-
4.
Probability Weighting The agent overweights probabilities close to 0 and underweights probabilities close to 1.
Of critical importance in applying prospect theory to model choices under risk is the determination of the reference point. Often the reference point is implicitly set to equal 0, but this assumption may not be realistic when all outcomes are positive, as is typical in practice when rewarding subjects for providing reports according to a QSR. For example, if the reference point were taken to be 0, then all outcomes in our experiment would be viewed as “gains” and the prospect theory model would be unable to explain the observed reporting behavior.
Instead, we argue that even in settings where all outcomes are nominally positive, an agent may still feel elation or disappointment based on whether the payoff she receives falls above (a “gain”) or below (a “loss”) what she expected at the time that she submitted her report. To model this, we assume that the agent possesses a reference-dependent utility function of the form of Palley (2015), in which the agent develops an expectation \(E\) about her outcome \(S\) from the scoring rule, and this expected outcome then forms a natural reference point for her to evaluate the outcome that she ultimately receives. This utility function extends existing models of an endogenously determined reference point (see, e.g., Shalev (2000)) to accommodate the case of an agent with prospect-theory-type preferences. This model will provide a parsimonious explanation for the behavior that is observed, and most importantly, can be readily used to provide a solution to the problem and insight into why it works.
Specifically, we assume that when the agent’s outcome exceeds this expectation, she feels an additional gain of \((S-E)^\alpha \), where \(\alpha \in (0,1]\) specifies the curvature of her risk preferences. When her outcome falls below her expectation, she perceives this as an additional loss equal to \(- \lambda (E-S)^\alpha \), where \(\lambda \ge 1\) additionally parameterizes the agent’s degree of loss aversion. Mathematically, this utility function is specified by
$$\begin{aligned} v(S,E) = \left\{ \begin{array}{ll} E - \lambda (E-S)^\alpha \quad {\text{if}}\; S < E \\ E + (S-E)^\alpha \quad {\text{if}}\; S \ge E. \end{array}\right. \end{aligned}$$
If \(\alpha =1\), this formulation coincides with the loss-averse utility function detailed in Shalev (2000). If \(\alpha =\lambda =1\), then this simplifies to the risk-neutral objective of maximizing expected payoff that the definition of a proper scoring rule implicitly assumes.Footnote 3
In addition, we assume that the agent applies probability weighting functions \(w_+(p)\) and \(w_-(p)\) for scores that fall above and below \(E\) (positive and negative events), respectively. \(w_+(\cdot )\) and \(w_-(\cdot )\) are assumed to be strictly increasing with \(w_+(0) = w_-(0) = 0\), \(w_+(1) = w_-(1) = 1\), and \(w_+(p)+w_-(1-p)=1\) for all \(p \in [0,1]\).Footnote 4 The ex ante expected-valuation that an agent receives from responding to a binary scoring rule is then given by a probability-weighted sum over the possible scores; \(V(E) = \sum _S w(p_S) v(S,E).\)
Footnote 5
Until this point, we still have not specified the details of how the reference point \(E\) is determined. The motivating intuition we follow here is that the agent’s expected-valuation of the prospect should be consistent with her expectation about the prospect. In other words, if she uses \(E\) as her reference point in determining \(V(E)\), then the resulting expected-valuation should simply equal \(E\) itself. Specifically, we assume that the reference point \(E\) is determined endogenously according to the consistency equation \(V(E)=E\), as in Palley (2015). In this sense, for a given prospect, a consistent reference point \(E\) is the expectation that perfectly balances the agent’s potential gains against her potential losses, weighted according to her beliefs of their respective likelihoods.
A consistent reference point \(E\) is the natural evaluation of a prospect for an agent who carefully contemplates the possible outcomes and anticipates her possible ex post feelings, providing a summary measure of how the agent evaluates the risk in an ex ante sense. An agent who initially forms a reference point \(R\) higher than \(E\) will find that her expected losses \(-\lambda (R-S)^\alpha \) outweigh her expected gains \((S-R)^\alpha \), causing her to adjust her expectation downward.
Conversely, an agent whose reference point is initially lower than \(E\) will find that her expected gains outweigh her expected losses, causing her adjust her reference point upward. A thoughtful agent will thus converge to a unique consistent expectation \(E\). This notion of expectations as an endogenously determined reference point is introduced and developed in the models of Bell (1985), Loomes and Sugden (1986), Gul (1991), Shalev (2000), and Koszegi and Rabin (2006, 2007).
Note that the relationship between the reference point and the valuation function possesses an intentional “circularity,” which is an important part of the model. For any prospect, there exists only one unique reference point E that satisfies V(E) = E, and this is the reference point that represents the agent’s ex ante valuation of a given prospect. It is this equation that ensures the consistency of the valuation function and the reference point, and which pins down the appropriate expectation E.
Figure 1 displays an example of this reference point formation process. We see that a loss-averse agent with subjective beliefs of \(p=0.7\) would derive an ex ante expectation of \(-0.17\) from truthfully reporting \(r=0.7\) in response to a QSR, while deriving an ex ante expectation of \(-0.05\) from reporting \(r=0.6\). Both of these reports are therefore dominated by reporting the baseline \(r=0.5\), which yields an outcome of 0 with certainty and a corresponding ex ante expectation of \(0\). Whereas a risk-neutral agent would prefer to report \(r=0.7\), which yields the highest expected score, the loss-averse agent in this case will prefer to report \(r=0.5\).
We assume that the agent seeks to maximize her expected outcome \(E\) over all possible reports \(r \in [0,1]\), subject to the consistency requirement, which essentially means that the agent will consider her ex post prospects when she chooses her report and forms her ex ante expectation about her outcome. The timeline of events is displayed in Fig. 2.
Proposition 1
The optimal consistent report function when
\(c=0.5\)
is given by
$$\begin{aligned} r_L^*(p) = \left\{ \begin{array}{ll} \frac{ \varLambda (p)^{\frac{1}{\alpha}} }{ \varLambda (p)^{{\frac{1}{\alpha }}} + L }, & p < \min \left\{ w_-^{-1} \left( \frac{L^{\alpha }}{\lambda +L^{\alpha }} \right) , \frac{1}{2} \right\} \\ \frac{1}{2}, & \min \left\{ w_-^{-1} \left( \frac{L^{\alpha }}{\lambda +L^{\alpha }} \right) , \frac{1}{2} \right\} \le p \le \max \left\{ w_+^{-1} \left( \frac{ \lambda }{ L^\alpha + \lambda } \right) , \frac{1}{2} \right\} \\ \frac{ L }{ \varLambda (1-p)^{{\frac{1}{\alpha }}} + L }, & p > \max \left\{ w_+^{-1} \left( \frac{ \lambda }{ L^\alpha + \lambda } \right) , \frac{1}{2} \right\} , \end{array}\right. \end{aligned}$$
where
\(\varLambda (p) = \frac{ \lambda w_-(p) }{w_+(1-p)}\)
is the agent’s loss-weighted odds ratio of event
\(A. \)
The optimal consistent response function for more general (asymmetric) baseline probabilities \(c\) can be found in Appendix 1.Footnote 6
Proposition 2
For any positive linear rescaling of the payoffs
\(\tilde{S}_L(r) \equiv a S_L(r) + b\), \(a>0, b \in {\mathbb{R}}\), the optimal consistent report remains
\(\tilde{r}_L^*(p) = r_L^*(p)\)
and the corresponding optimal
ex ante
expected outcome is rescaled according to
\(\tilde{E}^*(p) = a E^*(p) +b\).
In other words, in contrast to the predictions of the cumulative prospect theory model with a fixed reference point and many classical utility formulations, the agent’s behavior will be invariant to positive linear rescaling of the payoffs. This means, for example, that the agent’s optimal behavior would not change if the decision maker decided to pay her in a different currency with exchange rate \(a\):1 or pay her an additional fixed fee \(b\) for providing the report.
Figure 3 displays the shape of optimal reports as a function of the agent’s beliefs \(p\) in response to the classical QSR. For a large region of moderate beliefs near \(1/2\), the agent will prefer to simply report \(1/2\) in order to receive a payoff of \(0\) with certainty. While the width of this region depends jointly on \(\lambda \), \(\alpha \), \(w_+(\cdot )\), and \(w_-(\cdot )\), it is largely driven by the loss aversion parameter \(\lambda \). The shape of the optimal consistent report function closely mirrors the theoretical results of Offerman et al. (2009). Here the decision maker cannot simply provide the agent with the classical QSR and then infer her true beliefs from her report because the resulting response function \(r^*(p)\) is not invertible. All beliefs \(p\) in the interval \([w_-^{-1} \left( \frac{1}{\lambda +1} \right) \le p \le w_+^{-1} \left( \frac{\lambda }{\lambda +1} \right) ]\) are mapped to the conservative risk-free report of \(1/2\) (this is the flat region of the optimal report function). This means that observing a report of \(1/2\), which may happen quite frequently if the agent is loss-averse and has moderate beliefs, tells the decision maker only that the agent’s beliefs lie somewhere within that interval.
Determining the \(L\)-adjustment
To recover true beliefs, the decision maker needs to instead adjust the scoring rule to eliminate the “flat region” of conservative reports of \(1/2\), which will allow him to invert the agent’s optimal report function \(r^*(p)\) and estimate \(p\) according to \(r^{*-1}(r)\). Sensitivity analysis suggests that loss aversion accounts for the largest proportion of this conservative behavior. The best way to counteract this phenomenon, then, is to adjust the scoring rule so that negative outcomes are less severe by a factor of \(\frac{1}{L}\). By computing the value \(L^*\) that solves
$$\begin{aligned} w_-^{-1} \left( \frac{L^{\alpha }}{\lambda +L^{\alpha }} \right) = w_+^{-1} \left( \frac{ \lambda }{ L^\alpha + \lambda } \right) , \end{aligned}$$
(1)
the decision maker can squeeze the endpoints of the “flat region” of conservative reports of \(1/2\) together and recover the agent’s true beliefs.
Corollary 1
The optimal adjustment when
\(c=0.5\)
is given by
$$\begin{aligned} L^* = \varLambda (1/2)^{1/\alpha } = \left( \frac{\lambda w_-(1/2)}{w_+(1/2)} \right) ^{1/\alpha }. \end{aligned}$$
This calibration of \(L = L^*\) eliminates the agent’s incentive to provide these uninformative reports even for very moderate beliefs close to \(1/2\), and also removes almost all of her distortion in the optimal reporting function. After receiving her report, the decision maker can apply the inverse of the optimal report function to the observed report \(r\) to recover the agent’s exact truthful beliefs \(p = r_L^{*-1}(r)\). In the absence of utility curvature and probability weighting (\(\alpha =1\) and \(w(p)=p\)), the optimal adjustment is simply equal to the loss aversion parameter (\(L^* = \lambda \)) and the inversion step is unnecessary because the optimal report function is truthful (\(r_\lambda ^*(p)=p\)).
In practice, an agent’s report may include a noisy error term \(\epsilon \), so that the agent reports \(r_L^*(p) + \epsilon \) instead. This means that the inferred beliefs will also contain an error of \( r_L^{*-1}(r_L^*(p) + \epsilon ) - r_L^{*-1}(r_L^*(p)). \) However, since \(r_L^{*-1}(\cdot )\) is differentiable and close to the identity function for a broad range of reasonable parameter values, the resulting error in inferred beliefs simply scales roughly equally to the size of the original reporting error. Another concern with the L-adjustment method is that it may become laborious if agents are very heterogeneous. In such a setting, the model parameters \(\alpha \), \(\lambda \), and \(w(p)\) and the corresponding \(L^*\) would have to be estimated individually. Our experimental results show, however, that heterogeneity is only of secondary importance and that our method does a remarkable job even without a correction of individual differences.
Figure 4 displays the optimal reports in response to an L-adjusted scoring rule, which is calibrated to average parameter estimates \(\lambda =2.4\), \(\alpha =0.8\), \(\delta _+ = 0.8\), \(\gamma _+ = 0.7\), \(\delta _- = 1.1\) and \(\gamma _- = 0.7\) (yielding \(L^*=3.7\)) from the studies discussed in footnotes 4 and 5 for the general population, for an agent with various actual loss aversion parameters \(\lambda \), utility curvature parameters \(\alpha \), and both with and without probability weighting. As might be expected, given that the adjustment is primarily designed to address distortions due to loss aversion, optimal report functions are most sensitive to misestimation of the parameter of loss aversion \(\lambda \), and are less sensitive to variations in \(\alpha \) and the probability weighting functions. This suggests that if the decision maker does not want to assess individual parameters, the most important measurement to focus on is \(\lambda \). We also see that if \(L^*\) is miscalibrated due to errors in parameter estimates, he may observe reports both above and below the true beliefs \(p\), depending on whether the \(\lambda \) estimate is too high or too low.
Next, note that any remaining difference between the optimal report function in response to the \(L^*\)-adjusted rule and truthful reporting, which in theory could be corrected by applying \(r^{*-1}(\cdot )\) to the observed report, would be completely swamped by any noise in reports and the distortionary effects of errors in the parameter estimates. As a result, in practice there is very little benefit to attempting to carry out this second inversion step on the reports \(r\). A more practical approach is to simplify the assessment process by eliminating this second inversion step and taking the reported probability as our estimate of the agent’s true beliefs. In doing so, the decision maker should keep in mind the remaining potential for distortion, which is mainly caused by incorrect estimation of the agent’s parameters, and understand that her reports may be somewhat noisy due to this miscalibration.
If the decision maker wishes to avoid the potentially laborious process of individually assessing parameter values for each agent beforehand, a simple approach is to simply present the agent with the L-adjusted QSR with \(L^*=3.7\) and take her resulting report as the estimate of her true beliefs. If the decision maker does want to spend some time and effort to estimate the agent’s parameter values ahead of time, he should focus on accurately assessing her loss-aversion parameter \(\lambda \), since this offers a fair amount of flexibility in calibrating the scoring rule, and variation in the other parameter values has a less significant effect on the optimal report function.
Below we outline a simple approach that the decision maker can use to estimate \(\lambda \) and \(L\) on an individual basis: first, assume that the agent’s utility curvature is \(\alpha =1\) and probability weighting function is \(w(p)=p\). This implies that for a 50–50 lottery between receiving \(x_1\) and \(x_2\), where \(x_1 \le x_2\), her consistent expectation is given byFootnote 7
$$\begin{aligned} E = ( x_2 + \lambda x_1 ) / ( 1 + \lambda ). \end{aligned}$$
(2)
Next, the agent is offered a choice from a carefully designed set of coin flips that offer different payoffs depending on whether the coin ends up heads or tails. We assume that the agent makes this choice so that she maximizes the consistent expectation given in Eq. 2, meaning that she will prefer different lotteries for different values of \(\lambda \). Specifically, the set of coin flips offered is designed so that each lottery is the most preferred option for a specific interval of possible \(\lambda \) values. Once the agent selects her most preferred lottery from this set, the decision maker can use her choice to make inferences about her loss aversion parameter, for example, by taking the midpoint of the interval of \(\lambda \) values for which that flip is the most preferred. As noted in the discussion of Corollary 1, under these assumptions, the optimal \(L\)-adjustment is then simply equal to that \(\lambda \) estimate. An example of such a set of coin flip lotteries and the \(\lambda \) parameters implied by each can be found in the description of the IC treatment in Sect. 3.