In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments
- 605 Downloads
In this paper, we use an experimental design to compare the performance of elicitation rules for subjective beliefs. Contrary to previous works in which elicited beliefs are compared to an objective benchmark, we consider a purely subjective belief framework (confidence in one’s own performance in a cognitive task and a perceptual task). The performance of different elicitation rules is assessed according to the accuracy of stated beliefs in predicting success. We measure this accuracy using two main factors: calibration and discrimination. For each of them, we propose two statistical indexes and we compare the rules’ performances for each measurement. The matching probability method provides more accurate beliefs in terms of discrimination, while the quadratic scoring rule reduces overconfidence and the free rule, a simple rule with no incentives, which succeeds in eliciting accurate beliefs. Nevertheless, the matching probability appears to be the best mechanism for eliciting beliefs due to its performances in terms of calibration and discrimination, but also its ability to elicit consistent beliefs across measures and across tasks, as well as its empirical and theoretical properties.
KeywordsBelief elicitation Scoring rules Confidence Calibration Discrimination Incentives
The authors are grateful for insightful comments by Karim N’Diaye, Steve Fleming, Thibault Gajdos and Peter Wakker; participants at the ESE Conference in Rotterdam, the EBIM Workshop in Paris, the LabSi Workshop in Sienna, the FUR XIV in Newcastle, the ESA 2010 in Copenhagen, the SABE 2010 in San Diego, the EMPG 2011 in Paris, and the North American ES Winter Meeting 2012 in Chicago.
- Andersen, S., Fountain, J., Harrison, G., & Rutstrom, E. (2010). Estimating subjective probabilities. CEAR Working Paper.Google Scholar
- Baillon, A., & Bleichrodt, H. (2015). Testing ambiguity models through the measurement of probabilities for gains and losses. American Economic Journal: Microeconomics (forthcoming), 7(2), 77–100.Google Scholar
- Dimmock, S., Kouwenberg, R., & Wakker, P. (2011). Ambiguity attitudes and portfolio choice: Evidence from a large representative survey. Netspar Discussion Paper No 06/2011-054.Google Scholar
- Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.Google Scholar
- Holt, C. (2006). Markets, games, and strategic behavior: Recipes for interactive learning. Reading: Addison-Wesley.Google Scholar
- LaValle, I. H. (1978). Fundamentals of decision analysis. New York: Holt, Rinehart and Winston.Google Scholar
- Massoni, S. (2009). A direct revelation mechanism for elicitating confidence in perceptual and cognitive tasks: An experimental study. Master’s Thesis, Université Paris 1.Google Scholar
- Mobius, M., Niederle, M., Niehaus, P., & Rosenblat, T. (2011). Managing self-confidence: Theory and experimental evidence. NBER Working Paper No 17014.Google Scholar
- Raiffa, H. (1968). Decision analysis. London: Addison-Wesley.Google Scholar
- Trautmann, S., & van de Kuilen, G. (2015). Belief elicitation: A horse race among truth serums. The Economic Journal (forthcoming).Google Scholar
- Winkler, R. L. (1972). An introduction to Bayesian inference and decision theory. New York: Holt, Rinehart and Winston.Google Scholar