Noise and bias in eliciting preferences

Abstract

In the context of eliciting preferences for decision making under risk, we analyse the features of four different elicitation methods—pairwise choice, willingness-to-pay, willingness-to-accept, and the Becker-DeGroot-Marschak mechanism—and estimate noise, bias and risk attitudes for two different preference functionals, Expected Utility and Rank-Dependent Expected Utility. It is well-known that methods differ in terms of the bias in the elicitation; it is rather less well-known that methods differ in terms of their noisiness. It has also been reported that risk attitudes are not stable across different elicitation methods. Our results suggest that elicited preferences should only be used in the context in which they were elicited, and the bias in the certainty-equivalent methods should be kept in mind when making predictions based on the elicited preferences. Moreover, conclusions should be moderated to take into account the various methods’ noise, which is generally lowest in the case of pairwise choice.

Appendices

Appendix 1

Table 10 The lotteries in the experiment and median responses

Appendix 2: Technical details

This Appendix describes the mathematics lying behind the estimation and the GAUSS programs used in the estimation.¹³ We concentrate on the EU estimates. Those for RDEU are the same, mutatis mutandis. We assume throughout that the subjects make monetary evaluations of the various gambles with a normally distributed error.

In the experiment there were 4 possible outcomes £0, £10, £30 and £40. We denote the utilities of these by u ₁ , u ₂ , u ₃ and u ₄ .

We assume a CRRA utility function:

$$ u(x) = {{{x^{1 - r}}} \mathord{\left/{\vphantom {{{x^{1 - r}}} {\left( {1 - r} \right)}}} \right.} {\left( {1 - r} \right)}} $$

(1)

where we have normalised the function so that u ₁   =  0 and u ₄   =  1. A risk-neutral person has r  =  1. The inverse of the utility function is

$$ x = {u^{ - 1}}(u) = \left( {1 - r} \right){u^{{1 \mathord{\left/{\vphantom {1 {\left( {1 - r} \right)}}} \right.} {\left( {1 - r} \right)}}}} $$

2.1 Estimation using the certainty equivalent data

Denote by c _j the certainty equivalent reported by the subject on question number j (j  =  1,..,J) . Let us drop the subscript j to save notational clutter. Suppose the probabilities on the question are p ₁ , p ₂ , p ₃ and p ₄ . Then the Expected utility of the gamble (for given parameters) is

$$ EUG = {p_1}{u_1} + {p_2}{u_2} + {p_3}{u_3} + {p_4}{u_4} = {0.25^r}{p_2} + {0.75^r}{p_3} + {p_4} $$

(2)

Hence the true certainty equivalent, t, of the gamble G is given by

$$ t = 40{\left( {{{0.25}^r}{p_2} + {{0.75}^r}{p_3} + {{0.75}^r}{p_3} + {p_4}} \right)^{{1 \mathord{\left/{\vphantom {1 r}} \right.} r}}} $$

(3)

The difference between the stated certainty equivalent and the true one

$$ c - t = c - 40{\left( {{{0.25}^r}{p_2} + {{0.75}^r}{p_3} + {p_4}} \right)^{{1 \mathord{\left/{\vphantom {1 r}} \right.} r}}} $$

We assume that this difference c − t is error, normally distributed with standard deviation s. The normal pdf of this is:

$$ f(x) = \frac{1}{{\sqrt {2{\pi^2}} }}\exp \left[ { - \frac{{{x^2}}}{{2{s^2}}}} \right] $$

(4)

The log of the pdf is therefore

$$ \ln f(x) = - {x^2}/2{s^2} - 0.5\ln \left( {2\pi {s^2}} \right) $$

(5)

It follows that the log of the probability density of the difference is:

$$ \ln f\left( {c - t} \right) = - {\left( {c - t} \right)^2}/2{s^2} - \ln (s) - 0.5\ln \left( {2\pi } \right) $$

(6)

This is the contribution to the log-likelihood from the certainty equivalent data.

2.2 Estimation using the preference data

Again we assume that subjects make normally distributed errors when evaluating lotteries. When comparing two lotteries they compare the estimated certainty equivalents. Suppose we have two gambles L and R. The Expected Utilities are EUL and EUR. Their monetary evaluations are ML  =  u ⁻¹ (EUL) and are MR  =  u ⁻¹ (EUR) The treatment is different according to whether the subject reports indifference or not.

• 1) The subjectneverreports indifference. In this case, we have that L is reported as preferred to R if $ML - MR + \varepsilon \geqslant 0$ and that R is reported as preferred if $ML - MR + \varepsilon < 0$. Hence the probability that L is reported as preferred is ${\text{Prob}}\left( {\varepsilon \geqslant MR - ML} \right)$ and the probability that R is reported as preferred is ${\text{Prob}}\left( {\varepsilon < MR - ML} \right)$. Hence the probability of L (R) is:

$$ \begin{gathered} \begin{array}{*{20}{c}} {{\text{Prob}}\left( {\varepsilon \geqslant MR - ML} \right)}{\left( {{\text{Prob}}\left( {\varepsilon < {\text{ }}MR - ML} \right)} \right)} \\ \end{array} \hfill \\ {\text{ }} \hfill \\ \end{gathered} $$

(9)

Now we need to find expressions for the probabilities. If we denote the normal cdf by (x/s) (this is the integral of (4)) we can then write that the probability of L (R) is

$$ \begin{gathered} \begin{array}{*{20}{c}} {1 - \Psi \left( {{{\left( {MR - ML} \right)} \mathord{\left/{\vphantom {{\left( {MR - ML} \right)} s}} \right.} s}} \right)}{\left( {\Psi \left( {{{\left( {MR - ML} \right)} \mathord{\left/{\vphantom {{\left( {MR - ML} \right)} s}} \right.} s}} \right)} \right)} \\ \end{array} \hfill \\ \hfill \\ \end{gathered} $$

(10)

Hence the log-likelihood is

$$ \begin{gathered} \begin{array}{*{20}{c}} {\ln \left( {1 - \Psi \left( {MR - ML} \right)} \right)}{\left( {\ln \left( {\Psi \left( {MR - ML} \right)} \right)} \right)} \\ \end{array} \hfill \\ \hfill \\ \end{gathered} $$

(11)

• 2) The subjectsometimesreports indifference. This is almost the same but we need some story about when the subject reports indifference. We say that L is reported as preferred if $ML - MR + \varepsilon \geqslant \tau$, that R is reported as preferred if $ML - MR + \varepsilon < - \tau$, and that indifference is reported when $- \tau \leqslant ML - MR + \varepsilon < \tau$. Hence the probability that L is reported as preferred is ${\text{Prob}}\left( {\varepsilon \geqslant MR - ML + \tau } \right)$, the probability that R is reported as preferred is ${\text{Prob}}\left( {\varepsilon < MR - ML - \tau } \right)$, and the probability that indifference is reported is ${\text{Prob}}\left( {MR - ML - \tau \leqslant \varepsilon < MR - ML + \tau } \right)$.

2.3 Estimating bias in the certainty equivalents

We simply assume that there is a true valuation V and a reported valuation v which are related by

$$ V = a + b\nu $$

Here the parameters a and b determine the bias in the reporting of the certainty equivalents. If a  =  0 and b  =  1 there is no bias. In the text tables we report the estimated values of a and b for each of the certainty equivalent methods. We assume no bias in the pairwise choice elicitation method.