Skip to main content
Log in

Updating beliefs with imperfect signals: Experimental evidence

  • Published:
Journal of Risk and Uncertainty Aims and scope Submit manuscript

Abstract

We conduct an experiment on individual choice under risk in which we study belief updating when an agent receives a signal that restricts the number of possible states of the world. Subjects observe a sample drawn from an urn and form initial beliefs about the urn’s composition. We then elicit how beliefs are modified after subjects receive a signal that restricts the set of the possible urns from which the observed sample could have been drawn. We find that this type of signal increases the frequency of correct assessments and that prediction accuracy is higher for lower levels of risk. We also show that prediction accuracy is higher after invalidating signals (i.e. signals that contradict the initial belief). This pattern is explained by the lower level of risk associated with invalidating signals. Finally, we find evidence for a lack of persistence of choices under high risk.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. A standard is defined as a minimum level to achieve.

  2. For instance, consider an urn containing balls of two colors: 2/3 are red and 1/3 are white. A first subject draws 6 balls (5 are red and 1 is white), and a second subject draws 20 balls (12 are red and 8 are white). Asking individuals about who of the two subjects should feel more confident that the urn from which the sample has been drawn contains 2/3 of red balls, they answer the first one.

  3. We thank an anonymous referee for having suggested this article.

  4. As an illustration, see the instructions in Appendix A.

  5. Given that the urn and the sample composition are random, the numbers 6, 10 and 14 are chosen so as to ensure enough observations in each case.

  6. The objective of the paper is not to test if the subjects apply Bayes’ rule in forming their beliefs. Instead, our goal is to study how individuals update their beliefs after observing a pass/fail test, regardless of the law or heuristic they use in their revision process. Nevertheless, we think that predictions computed with Bayes’ rule represent an interesting point of comparison.

  7. For instance, if the observed sample contains 6 yellow balls out of 10, the most likely number of yellow balls contained in the urn is 12.

  8. In the post experimental questionnaire 28% of the subjects declare having used the representativeness heuristics to make their decisions.

  9. Test of equality of distributions on matched data.

  10. The three categories are: (i) the proportion of yellow balls is lower than 0.1, (ii) the proportion of yellow balls is between 0.1 and 0.9, and (iii) the proportion of yellow balls is higher than 0.9.

  11. Indeed, the coefficient of correlation between the number of possible urns after a signal and invalidating signals is negative and high (−0.301).

  12. A closer look at the data makes it apparent that the subjects are less likely to change their prediction when the observed sample is composed of around 100% or 0% of yellow balls and subjects play the optimal strategy (their predictions are close to 20 or 0 yellow balls respectively) and regardless of the sample size.

References

  • Ashton, A. H., & Ashton, R. H. (1988). Sequential belief revision in auditing. Accounting Review, 63, 623–641.

    Google Scholar 

  • Bazerman, M. (1990). Judgement in managerial decision making. New York: Wiley.

    Google Scholar 

  • Camerer, C. (1995). Individual decision making. In J. H. Kagel, & A. E. Roth (Eds.), Handbook of experimental economics. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Charness, G., & Levin, D. (2005). When optimal choices feel wrong: A laboratory study of Bayesian updating, complexity, and affect. American Economic Review, 95, 1300–1309.

    Article  Google Scholar 

  • Friedman, D. (1998). Monty Hall’s three doors: Construction and deconstruction of a choice anomaly. American Economic Review, 88(4), 933–946.

    Google Scholar 

  • Greiner, B. (2004). An online recruitment system for economic experiments. Forschung und wissenschaftliches Rechnen 2003. GWDG Bericht (Vol. 63, pp. 79–93).

  • Grether, D. (1980). Bayes rule as a descriptive model: The representativeness heuristic. Quarterly Journal of Economics, 95, 537–557.

    Article  Google Scholar 

  • Hoffman, R. M., Kagel, J. H., & Levin, D. (2011). Simultaneous versus sequential information processing. Economics Letters, 112, 16–18.

    Article  Google Scholar 

  • Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgement under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.

    Google Scholar 

  • Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237–251.

    Article  Google Scholar 

  • Ouwersloot, H., Nijkamp, P., & Rietveld, P. (1998). Errors in probability updating behaviour: Measurement and impact analysis. Journal of Economic Psychology, 19, 535–563.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105–110.

    Article  Google Scholar 

  • Zeiliger, R. (2000). A presentation of regate, internet based software for experimental economics. http://www.gate.cnrs.fr/∼zeiliger/regate/RegateIntro.ppt, GATE, Lyon.

Download references

Acknowledgements

We are grateful to Stéphane Robin, Isabelle Vialle, Marie Claire Villeval and Anne Rozan for positive and constructive discussions. We thank the editor, Kip Viscusi and the anonymous referee for their helpful comments. We are also grateful to participants at the GATE research seminars, the ESA meeting in Washington, the JMA conference in Angers and the AFSE annual conference in Paris. Financial support from the “Chaire Universitaire en Economie Expérimentale : Finance industrielle et design de réseaux” and GATE are gratefully acknowledged.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to François Poinas.

Appendices

Appendix A: Instructions (original in French)

You are about to participate in an experiment during which you can earn money. Your earnings will depend on your decisions taken during the experiment. Your earnings will be paid to you in cash in private and in a separate room.

There are 50 periods in the experiment. Each period has two parts. In each period, you see a partially revealed urn containing 20 balls. The balls may be yellow or blue.

In each part, you have to predict the total number of yellow balls in the urn. Your earnings depend on your prediction of the total number of yellow balls. You increase your earnings if you find the exact total number of yellow balls in the urn.

Period description

An urn is composed of 20 balls: yellow or blue. The number of yellow balls is randomly determined between 0 and 20. There are equal chances that the urn is composed of 0 yellow balls, or 1 yellow ball,..., or 20 yellow balls. The urn stays the same during the period. You can observe only 6, 10 or 14 balls in the urn. The visible balls are randomly drawn from the urn.

Each period has 2 parts. The set of visible balls is identical for the 2 parts.

  • PART 1   You have to predict the total number of yellow balls in the urn. Enter your answer with the scroll bar and validate it.

    figure a
  • PART 2   The urn stays the same as in the 1st part. You receive complementary information on the urn’s composition. This information indicates if the urn is composed with at most, or at least 6, 10 or 14 yellow balls. Then, you have to predict a second time the total number of yellow balls in the urn. Enter your answer with the scroll bars, even if your answer does not differ from your prediction in the 1st part, and validate it.

    figure b

Description of the added information on the urn’s composition

You can receive one of the following pieces of information:

  • ≥ 6, there are AT LEAST 6 yellow balls in the urn.

  • < 6, there are LESS THAN 6 yellow balls in the urn.

  • ≥ 10, there are AT LEAST 10 yellow balls in the urn.

  • < 10, there are LESS THAN 10 yellow balls in the urn.

  • ≥ 14, there are AT LEAST 14 yellow balls in the urn.

  • < 14, there are LESS THAN 14 yellow balls in the urn.

All six possibilities are not simultaneously possible for a given composition of the urn. For example, when an urn has a total number of 8 yellow balls, only three things out of six are possible:

  • ≥ 6, there are AT LEAST 6 yellow balls in the urn.

  • < 10, there are LESS THAN 10 yellow balls in the urn.

  • < 14, there are LESS THAN 14 yellow balls in the urn.

The computer draws randomly and shows one of the possible pieces of information.

At the end of this 2nd part, you observe the complete urn and the predictions you made in part 1 and in part 2.

A new period starts automatically when you validate the last screen. At each new period, your task remains the same, but the following elements can be different:

  • the urn’s composition

  • the number of visible balls

  • the information on the maximum or minimum total number of yellow balls in the urn.

Payment

Three periods over the 50 played are selected for your payoff. The three periods are randomly drawn by the computer at the end of the experiment. They can be different among the participants. Thus, you do not know in advance which periods will be selected. Each period has the same chance to be selected.

For each of the three selected periods, only one part is chosen for your payoff. When you enter the payment room, you determine the part selected for the payoff by tossing a coin:

  • if you toss tails, the 1st part is selected for your payoff.

  • if you toss heads, the 2nd part is selected for your payoff.

To summarize, three periods are selected randomly by the computer and, for each of the selected periods, you determine which part will be paid by tossing a coin. Thus, three parts are selected for your payoff. Your payoff depends on the number of exact predictions you have made for these three parts:

  • If none of your three predictions is exact: your payoff is €0.

  • If one of your three predictions is exact: your payoff is €10.

  • If two of your three predictions are exact: your payoff is €15.

  • If all your three predictions are exact: your payoff is €20.

In all the cases, you will receive €10 more for your participation in this experiment.

It is forbidden to communicate with other subjects during the experiment. If you have any question regarding these instructions, please raise your hand.

Appendix B: Predicted probabilities

2.1 B.1 Predicted probabilities in part 1

An urn is composed of 20 balls and the number of yellow balls contained in the urn is chosen randomly (from 0 to 20). Therefore, there are 21 possible urns, with each one having the same probability (1/21) of being drawn. Then, the probability to observe sample S, given the urn from which the sample is drawn is U i is:

$$ P(S|U_i)=\frac{C_{y_u}^{y_s}*C_{n_u-y_u}^{n_s-y_s}}{C_{n_u}^{n_s}} $$

where n u (resp. n s ) is the number of balls contained in urn U i , n u  = 20, (resp. in sample S) and y u (resp. y s ) represents the number of yellow balls contained in urn U i (resp. in sample S). Then, according to Bayes’ rule and given an observed sample S, the probability that this sample is drawn from urn U i is equal to:

$$ P(U_i|S)=\frac{P(S|U_i)\cdot P(U_i)}{\sum_{x=1}^{21}P(S|U_{x})\cdot P(U_{x})} $$

Given an observed sample S, the distribution of probabilities over the 21 possible urns is obtained using this formula. The optimal choice for the subject is thus to give as a prediction the number of yellow balls contained in the urn having the highest probability.

2.2 B.2 Predicted Probabilities in part 2

Now, we have to compute the probability that a particular observed sample is drawn from each of the 21 possible urns, after a signal is received. According to Bayes’ rule, the probability that sample S is drawn from urn U i given signal I (I stands for information) is:

$$ P(U_i|S,I)=\frac{P(S,I|U_i)\cdot P(U_i)}{\sum_{x=1}^{21}P(S,I|U_x)\cdot P(U_x)} $$

The signal sent to the agent is only based on the composition of the urn. In particular, it is independent from the composition of the observed sample drawn from the urn in part 1. Therefore, P(S,I|U i ) = P(S|U i P(I|U i ). P(I|U i ) is computed according to the following logic. For a particular urn, among the six existing signals, only three can be sent to the subject. Indeed, the number of yellow balls contained in an urn cannot be both inferior and superior to the same number. Since the signal is randomly chosen, the probability that a signal is sent is equal to one third if the signal is a possible one, and 0 if the signal is inconsistent with the composition of the urn. This leads to the following:

$$ P(U_i|S,I)=\frac{P(S|U_i)\cdot P(I|U_i)\cdot P(U_i)}{\sum_{x=1}^{21}P(S|U_x)\cdot P(I|U_x)\cdot P(U_x)} $$

with

$$ P(I|U_i)=\left\{ \begin{array}{ll} 1/3&\textrm{ if signal }I\textrm{ is consistent with the composition of urn }U_i\\ 0&\textrm{ if not} \end{array} \right. $$

As is the case in the first part, the optimal choice for the subject is to give as a prediction the number of yellow balls contained in the urn having the highest probability, given the sample and the signal.

Appendix C: Example of optimal prediction

Figures 2 and 3 display the probability distribution that different predictions are correct for an urn where the observed sample is composed of 6 yellow balls and 4 blue balls.

Fig. 2
figure 2

Optimal predictions for a validating signal

Fig. 3
figure 3

Optimal predictions for an invalidating signal

In both figures, the optimal prediction in the first part (represented by light bars) is 12 yellow balls as it is the number of yellow balls associated with the urn having the highest probability to be correct. Figure 2 represents the case where the subject receives a signal that validates his initial belief (indeed the signal states that “there are at least 10 yellow balls”). Each one of the urns remaining in the new restricted set has a higher probability of being correct. Thus, the optimal prediction in part 2 (represented by dark bars) remains the same (12) and its probability of being correct increases. Figure 3 represents the case where the signal invalidates the first prediction. The signal states that “there are less than 10 yellow balls” in the urn and proves the optimal prediction to be wrong. In this case, the optimal prediction in the second part corresponds to 9, which is the closest number to 12 included in the new restricted set.

Appendix D: Expected revenue

The expected gain for one subject depends both on the number of correct predictions he gives and on the likelihood that the period(s) at which he makes correct predictions is(are) drawn at the end of the experiment. Three periods are drawn. The probability that, among those three periods, a subject makes i correct predictions (and therefore makes 3 − i wrong predictions) is given by:

$$ P(i)=\frac{C^{i}_{n}*C^{3-i}_{n-c}}{C^{3}_{n}},\textrm{ for }i=0,\ldots,3 $$

where n represents the total number of periods played and c the total number of correct predictions.

The expected gain is the average of the gains associated with 0, 1, 2 and 3 correct answers drawn, weighted by the probability that the corresponding number of correct answers is drawn:

$$ ER(n,c) = \frac{C^{0}_{c}*C^{3}_{n-c}}{C^{3}_{n}}*10 + \frac{C^{1}_{c}*C^{2}_{n-c}}{C^{3}_{n}}*20 + \frac{C^{2}_{c}*C^{1}_{n-c}}{C^{3}_{n}}*25 + \frac{C^{3}_{c}*C^{0}_{n-c}}{C^{3}_{n}}*30 $$

Given that, for the whole experience, the subjects made 7900 predictions, among which 1492 were correct ones, the total expected revenue is ER(7900,1492) = €15.16. If every subject had played with the optimal strategy (consisting in giving as a prediction the urn associated with the highest probability), they would have made 2340 correct predictions. In this case, the expected revenue would have been ER(7900,2340) = €17.70.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Poinas, F., Rosaz, J. & Roussillon, B. Updating beliefs with imperfect signals: Experimental evidence. J Risk Uncertain 44, 219–241 (2012). https://doi.org/10.1007/s11166-012-9143-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11166-012-9143-7

Keywords

JEL Classification

Navigation