Abstract
This paper considers pick-the-winners contests as a simple method for harnessing the wisdom of crowds to produce probability forecasts (and be used as a business forecasting tool). Pick-the-winners contests are those in which players pick the outcomes of selected future binary events with a prize going to the player with the most correct picks. In contrast to soliciting probability forecasts from experts (for which competition among the forecasters leads to exaggerated and less accurate probabilities), this paper shows that competition among players is to be encouraged because it improves the accuracy of the resulting crowd probability forecasts. This improvement comes because the competition not only discourages the overbetting of favorites that occurs if prizes are awarded based on absolute performance, but also helps mitigate the public knowledge bias that occurs if everyone reports truthfully. In addition to the theoretical arguments, the paper analyzes picks from 6 years of pick-the-winners contests involving hundreds of players and 1037 college lacrosse games to find that the crowd proportions outperformed two benchmark sets of probabilities.
Similar content being viewed by others
References
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.
Chen, K., Fine, L. R., & Huberman, B. A. (2004). Eliminating public knowledge biases in information-aggregation mechanisms. Management Science, 50, 983–994.
Clair, B., & Letscher, D. (2007). Optimal strategies for sports betting pools. Operations Research, 55(6), 1163–1177.
Gaba, A., Tsetlin, L., & Winkler, R. L. (2004). Modifying variability and correlations in winner-take-all contests. Operations Research, 52(3), 384–395.
Johnson, N. L., & Kotz, S. (1969). Discrete distributions. New York: Wiley.
Johnson, N. L., & Kotz, S. (1970). Continuous univariate distributions-2. New York: Wiley.
Laster, D., Bennett, P., & Geoum, I. S. (1999). Rational bias in macroeconomic forecasts. The Quarterly Journal of Economics, 114(1), 293–318.
Lichtendahl, K. C., Jr., & Winkler, R. L. (2007). Probability elicitation, scoring rules, and competition among forecasters. Management Science, 53(11), 1745–1755.
Lichtendahl, K. C., Jr., Grushka-Cockayne, Y., & Pfeifer, P. E. (2013). The wisdom of competing crowds. Operations Research, 61(6), 1383–1398.
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: the state of the art in 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: heuristics and biases (pp. 306–344). Cambridge: Cambridge University Press.
McCrea, S. M., & Hirt, E. A. (2009). March madness: probability matching in prediction of the NCAA basketball tournament. Journal of Applied Social Psychology, 39(12), 2809–2839.
Metrick, A. (1996). March madness? Strategic behavior in NCAA basketball tournament betting pools. Journal of Economic and Behavior Organization, 30, 159–172.
Niemi, J. B., Carlin, B. P., & Alexander, J. M. (2008). Contrarian strategies for NCAA tournament pools: a cure for march madness? Chance, 21(1), 35–42.
Osborne, M. J. (2004). An introduction to game theory. Oxford: Oxford University Press.
Ottaviani, M., & Sørensen, P. N. (2006). The strategy of professional forecasting. Journal of Financial Economics, 81, 441–446.
Pfeifer, P. E., Grushka-Cockayne, Y., & Lichtendahl, K. C, Jr. (2014). The promise of prediction contests. The American Statistician, 68(4), 264–270.
Surowiecki, J. (2006). The wisdom of crowds. New York: Anchor Books.
Winston, W. L. (2004). Microsoft excel data analysis and business modeling. Winston: Wayne L.
Wolfers, J., & Zitzewitz, E. (2004). Prediction markets. Journal of Economic Perspectives, American Economic Association, 18(2), 107–126.
Acknowledgments
The author acknowledges helpful comments on this paper provided by Yael Grushka-Cockayne, Kenneth C. Lichtendahl Jr. and anonymous referees.
Author information
Authors and Affiliations
Corresponding author
Appendix: Derivation of MSE of truthful-probability-reporting and private-information-randomizing crowd probability forecasts
Appendix: Derivation of MSE of truthful-probability-reporting and private-information-randomizing crowd probability forecasts
Recall that the probability forecaster i assigns to a game (dispensing with the j subscript) is
where \(N=m_0 +m +n.\)
If a crowd of k forecasters each report their probabilities to a planner who averages them, the crowd proportion becomes
where \(\bar{r} _k \) is the average of the k private signals.
Writing the forecast error as
we note that the forecast error for given P is a linear combination of four independent components, the second of which is a constant (for given P), and the other three have expectation of zero. The expected squared forecast error given P will then be
given that the expected value of all cross-product terms is zero, and given the properties of the binomial distribution. We rewrite this as
Since P follows the beta distribution with parameters \(r_{0}, m_{0} - r_{0}\), we know that \({E[P]} = r_{0}/m_{0}\), \({E}[P^{2}] = r_{0}(r_{0}+1)/[m_{0}(m_{0}+1)]\), and \({E}[(r_{0}/m_{0}-P)^{2}] = {\mathrm{Var}}(P)=r_{0}(m_{0} - r_{0})/[m_{0}^{2}(m_{0} + 1)]\) (see, for example, Johnson and Kotz 1970, p. 40) so that
For a private-information-randomizing crowd, the crowd proportion will behave as if it were the average of k independent \(r_{i}\) from players with private information consisting of \(n= 1\) trial. To see this compare a player who first observes \(r_{i}\) heads in n flips of a coin and then picks 1 with probability \(r_{i}/n\) to a player who simply sets his pick equal to the outcome from one flip of the coin. In both cases, the players pick 1 independently (from all other players) with probability P (the actual probability the coin comes up heads).
This means that
where \(\bar{r}_k\) is equivalent to the average of k independent game outcomes. So that
which has expected value given P of
and expected value
Rights and permissions
About this article
Cite this article
Pfeifer, P.E. The promise of pick-the-winners contests for producing crowd probability forecasts. Theory Decis 81, 255–278 (2016). https://doi.org/10.1007/s11238-015-9533-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11238-015-9533-9