Abstract
Scoring rules are traditional techniques to measure the association between a reported belief and an observed outcome. The condition that a scoring rule is proper means that an agent maximizes his expected score when he reports a belief that equals his true belief. The implicit assumption that the agent is risk neutral is, however, often unrealistic, at least when the underlying agent is a human. Modern decision theories based on rank-dependent utilities, such as cumulative prospect theory, have been shown to be more effective at describing how human beings make decisions under risk and uncertainty. Traditional proper scoring rules are, however, incompatible with cumulative prospect theory because they fail to satisfy a property called comonotonicity. In this paper, we provide novel insights on why comonotonicity is crucial to make proper scoring rules indeed proper when eliciting beliefs from cumulative prospect theory agents. After suggesting strategies to create comonotonic proper scoring rules, we propose calibration procedures to obtain an agent’s true belief by removing the influence of the agent’s value function and weighting functions from his reported belief, when beliefs are elicited by means of comonotonic proper scoring rules.
Similar content being viewed by others
References
Abdellaoui M (2000) Parameter-free elicitation of utility and probability weighting functions. Manage Sci 46(11):1497–1512
Abdellaoui M, Bleichrodt H, L’Haridon O (2008) A tractable method to measure utility and loss aversion under prospect theory. J Risk Uncert 36(3):245–266
Allais M (1953) Violations of the betweenness axiom and nonlinearity in probability. Econometrica 21:503–546
Allen F (1987) Discovering personal probabilities when utility functions are unknown. Manage Sci 33(4):542–544
Armantier O, Treich N (2013) Eliciting beliefs: proper scoring rules, incentives, stakes and hedging. Eur Econ Rev 62:17–40
Barberis N, Xiong W (2009) What drives the disposition effect? An analysis of a long-standing preference-based explanation. J Fin 64(2):751–784
Bickel JE (2010) Scoring rules and decision analysis education. Decis Anal 7(4):346–357
Birnbaum MH (1997) Violations of monotonicity in judgment and decision making. In: Marley AAJ (ed) Choice, decision, and measurement: essays in Honor of R. Duncan Luce, pp 73–100
Birnbaum MH (2008) New paradoxes of risky decision making. Psychol Rev 115(2):463–501
Camerer CF (2004) Prospect theory in the wild: evidence from the field. In: Camerer CF, Loewenstein G, Rabin M (eds) Advances in behavioral economics, chapter 5, pp 148–161
Carvalho A (2015) Tailored proper scoring rules elicit decision weights. Judgm Decis Mak 10(1):86–96
Carvalho A (2016a) A note on Sandroni–Shmaya belief elicitation mechanism. J Predict Mark 10(2):14–21
Carvalho A (2016b) An overview of applications of proper scoring rules. Decis Anal 13(4):223–242
Carvalho A (2017) On a participation structure that ensures representative prices in prediction markets. Decis Support Syst 104:13–25
Carvalho A, Larson K (2012) Sharing rewards among strangers based on peer evaluations. Decis Anal 9(3):253–273
Chambers CP (2008) Proper scoring rules for general decision models. Games Econ Behav 63(1):32–40
Friedman D (1983) Effective scoring rules for probabilistic forecasts. Manage Sci 29(4):447–454
Gilboa I (1987) Expected utility with purely subjective non-additive probabilities. J Math Econ 16(1):65–88
Gneiting T, Raftery AE (2007) Strictly Proper scoring rules, prediction, and estimation. J Am Stat Assoc 102(477):359–378
Gonzalez R, Wu G (1999) On the shape of the probability weighting function. Cogn Psychol 38:129–166
Hanson R (2003) Combinatorial information market design. Inf Syst Front 5(1):107–119
Hines G, Larson K (2010) Preference elicitation for risky prospects. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pp 889–896
Hossain T, Okui R (2013) The binarized scoring rule. Rev Econ Stud 80(3):984–1001
Johnstone DJ, Jose VRR, Winkler RL (2011) Tailored scoring rules for probabilities. Decis Anal 8(4):256–268
Kadane JB, Winkler RL (1988) Separating probability elicitation from utilities. J Am Stat Assoc 83(402):357–363
Karni E (2009) A mechanism for eliciting probabilities. Econometrica 77(2):603–606
Kothiyal A, Spinu V, Wakker PP (2011) Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. J Multi-Crit Decis Anal 17(3–4):101–113
Nakazono Y (2013) Strategic behavior of federal open market committee board members: evidence from members’ forecasts. J Econ Behav Organ 93:62–70
Offerman T, Palley AB (2016) Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents. Exp Econ 19(1):1–30
Offerman T, Sonnemans J, Van De Kuilen G, Wakker PP (2009) A truth serum for Non-Bayesians: correcting proper scoring rules for risk attitudes. Rev Econ Stud 76(4):1461–1489
Parkes DC, Wellman MP (2015) Economic reasoning and artificial intelligence. Science 349(6245):267–272
Perny P, Viappiani P, Boukhatem A (2016) Incremental preference elicitation for decision making under risk with the rank-dependent utility model. Uncert Artif Intell
Quiggin J (1982) A theory of anticipated utility. J Econ Behav Organ 3(4):323–343
Sandroni A, Shmaya E (2013) Eliciting beliefs by paying in chance. Econ Theory Bull 1(1):33–37
Savage LJ (1971) Elicitation of personal probabilities and expectations. J Am Stat Assoc 66(336):783–801
Schervish MJ (1989) A general method for comparing probability assessors. Ann Stat 17(4):1856–1879
Schlag KH, Tremewan J, Van der Weele JJ (2015) A penny for your thoughts: a survey of methods for eliciting beliefs. Exp Econ 3(1):457–490
Schlag KH, van der Weele JJ (2013) Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theor Econ Lett 3(1):38–42
Schmeidler D (1989) Subjective probability and expected utility without additivity. Econometrica 57(3):571–587
Schotter A, Trevino I (2014) Belief elicitation in the laboratory. Annu Rev Econ 6:103–128
Selten R, Sadrieh A, Abbink K (1999) Money does not induce risk neutral behavior, but binary lotteries do even worse. Theor Decis 46(3):211–249
Spiegelhalter DJ (1986) Probabilistic prediction in patient management and clinical trials. Stat Med 5(5):421–433
Starmer C (2000) Developments in non-expected utility theory: the hunt for a descriptive theory of choice under risk. J Econ Lit 38(2):332–382
Tetlock PE (2005) Expert political judgment: how good is it? How can we know? Princeton University Press
Toda M (1963) Measurement of subjective probability distributions. Technical Report ESD-TDR-63-407, Decision Sciences Laboratory, Electronic Systems Division, Air Force Systems Command, United States Air Force, Bedford, MA
Tversky A, Kahneman D (1992) Advances in prospect theory: cumulative representation of uncertainty. J Risk Uncert 5(4):297–323
Wakker P, Deneffe D (1996) Eliciting von Neumann–Morgenstern utilities when probabilities are distorted or unknown. Manage Sci 42(8):1131–1150
Wakker PP (2010) Prospect theory: for risk and ambiguity. Cambridge university press
Weber BJ, Chapman GB (2005) Playing for peanuts: why is risk seeking more common for low-stakes gambles? Organ Behav Hum Decis Process 97(1):31–46
Winkler RL, Murphy AH (1968) “Good” probability assessors. J Appl Meteorol 7(5):751–758
Winkler RL, Murphy AH (1970) Nonlinear utility and the probability score. J Appl Meteorol 9:143–148
Acknowledgements
The authors thank Jonathan Baron and two anonymous reviewers for useful comments. The authors also thank the Natural Sciences and Engineering Research Council of Canada for funding this research.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Carvalho, A., Dimitrov, S. & Larson, K. On proper scoring rules and cumulative prospect theory. EURO J Decis Process 6, 343–376 (2018). https://doi.org/10.1007/s40070-018-0081-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40070-018-0081-8