Skip to main content
Log in

The promise of pick-the-winners contests for producing crowd probability forecasts

  • Published:
Theory and Decision Aims and scope Submit manuscript

Abstract

This paper considers pick-the-winners contests as a simple method for harnessing the wisdom of crowds to produce probability forecasts (and be used as a business forecasting tool). Pick-the-winners contests are those in which players pick the outcomes of selected future binary events with a prize going to the player with the most correct picks. In contrast to soliciting probability forecasts from experts (for which competition among the forecasters leads to exaggerated and less accurate probabilities), this paper shows that competition among players is to be encouraged because it improves the accuracy of the resulting crowd probability forecasts. This improvement comes because the competition not only discourages the overbetting of favorites that occurs if prizes are awarded based on absolute performance, but also helps mitigate the public knowledge bias that occurs if everyone reports truthfully. In addition to the theoretical arguments, the paper analyzes picks from 6 years of pick-the-winners contests involving hundreds of players and 1037 college lacrosse games to find that the crowd proportions outperformed two benchmark sets of probabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.

    Article  Google Scholar 

  • Chen, K., Fine, L. R., & Huberman, B. A. (2004). Eliminating public knowledge biases in information-aggregation mechanisms. Management Science, 50, 983–994.

    Article  Google Scholar 

  • Clair, B., & Letscher, D. (2007). Optimal strategies for sports betting pools. Operations Research, 55(6), 1163–1177.

    Article  Google Scholar 

  • Gaba, A., Tsetlin, L., & Winkler, R. L. (2004). Modifying variability and correlations in winner-take-all contests. Operations Research, 52(3), 384–395.

    Article  Google Scholar 

  • Johnson, N. L., & Kotz, S. (1969). Discrete distributions. New York: Wiley.

    Google Scholar 

  • Johnson, N. L., & Kotz, S. (1970). Continuous univariate distributions-2. New York: Wiley.

    Google Scholar 

  • Laster, D., Bennett, P., & Geoum, I. S. (1999). Rational bias in macroeconomic forecasts. The Quarterly Journal of Economics, 114(1), 293–318.

    Article  Google Scholar 

  • Lichtendahl, K. C., Jr., & Winkler, R. L. (2007). Probability elicitation, scoring rules, and competition among forecasters. Management Science, 53(11), 1745–1755.

  • Lichtendahl, K. C., Jr., Grushka-Cockayne, Y., & Pfeifer, P. E. (2013). The wisdom of competing crowds. Operations Research, 61(6), 1383–1398.

  • Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: the state of the art in 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: heuristics and biases (pp. 306–344). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • McCrea, S. M., & Hirt, E. A. (2009). March madness: probability matching in prediction of the NCAA basketball tournament. Journal of Applied Social Psychology, 39(12), 2809–2839.

    Article  Google Scholar 

  • Metrick, A. (1996). March madness? Strategic behavior in NCAA basketball tournament betting pools. Journal of Economic and Behavior Organization, 30, 159–172.

    Article  Google Scholar 

  • Niemi, J. B., Carlin, B. P., & Alexander, J. M. (2008). Contrarian strategies for NCAA tournament pools: a cure for march madness? Chance, 21(1), 35–42.

    Article  Google Scholar 

  • Osborne, M. J. (2004). An introduction to game theory. Oxford: Oxford University Press.

    Google Scholar 

  • Ottaviani, M., & Sørensen, P. N. (2006). The strategy of professional forecasting. Journal of Financial Economics, 81, 441–446.

    Article  Google Scholar 

  • Pfeifer, P. E., Grushka-Cockayne, Y., & Lichtendahl, K. C, Jr. (2014). The promise of prediction contests. The American Statistician, 68(4), 264–270.

    Article  Google Scholar 

  • Surowiecki, J. (2006). The wisdom of crowds. New York: Anchor Books.

    Google Scholar 

  • Winston, W. L. (2004). Microsoft excel data analysis and business modeling. Winston: Wayne L.

    Google Scholar 

  • Wolfers, J., & Zitzewitz, E. (2004). Prediction markets. Journal of Economic Perspectives, American Economic Association, 18(2), 107–126.

    Article  Google Scholar 

Download references

Acknowledgments

The author acknowledges helpful comments on this paper provided by Yael Grushka-Cockayne, Kenneth C. Lichtendahl Jr. and anonymous referees.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Phillip E. Pfeifer.

Appendix: Derivation of MSE of truthful-probability-reporting and private-information-randomizing crowd probability forecasts

Appendix: Derivation of MSE of truthful-probability-reporting and private-information-randomizing crowd probability forecasts

Recall that the probability forecaster i assigns to a game (dispensing with the j subscript) is

$$\begin{aligned} \hat{p} _i =\left( {\frac{r_0 +r+r_i }{N}} \right) , \end{aligned}$$

where \(N=m_0 +m +n.\)

If a crowd of k forecasters each report their probabilities to a planner who averages them, the crowd proportion becomes

$$\begin{aligned} \bar{R} _k =\left( {\frac{1}{N}} \right) \left( {r_0 +r+\bar{r} _k } \right) , \end{aligned}$$

where \(\bar{r} _k \) is the average of the k private signals.

Writing the forecast error as

$$\begin{aligned}{}[ {X-\bar{R} _k } ]= & {} \left( {X-P} \right) -\left( {\bar{R} _k -P} \right) \\= & {} \left( {X-P} \right) -\left( {\frac{1}{N}} \right) \left[ {\left( {r_0 -m_0 P} \right) +\left( {r-mP} \right) +\left( {\bar{r} _k -nP} \right) } \right] , \end{aligned}$$

we note that the forecast error for given P is a linear combination of four independent components, the second of which is a constant (for given P), and the other three have expectation of zero. The expected squared forecast error given P will then be

$$\begin{aligned} E( {[ {X-\bar{R} _k } ]^{2}\left| P \right. } )= & {} P\left( {1-P} \right) \\&+\frac{1}{N^{2}}\left[ m_0 ^{2}\left( {\frac{r_0 }{m_0 }-P} \right) ^{2}+mP\left( {1-P} \right) +\frac{n}{k}P\left( {1-P} \right) \right] , \end{aligned}$$

given that the expected value of all cross-product terms is zero, and given the properties of the binomial distribution. We rewrite this as

$$\begin{aligned} =\left( {1+\frac{m+\frac{n}{k}}{N^{2}}} \right) P\left( {1-P} \right) +\frac{m_0 ^{2}}{N^{2}}\left( {\frac{r_0 }{m_0 }-P} \right) ^{2}. \end{aligned}$$

Since P follows the beta distribution with parameters \(r_{0}, m_{0} - r_{0}\), we know that \({E[P]} = r_{0}/m_{0}\), \({E}[P^{2}] = r_{0}(r_{0}+1)/[m_{0}(m_{0}+1)]\), and \({E}[(r_{0}/m_{0}-P)^{2}] = {\mathrm{Var}}(P)=r_{0}(m_{0} - r_{0})/[m_{0}^{2}(m_{0} + 1)]\) (see, for example, Johnson and Kotz 1970, p. 40) so that

$$\begin{aligned}&\mathrm{MSE}_{\text {truthful-probability reporting}}\\&\quad =\left( {1+\frac{m+\frac{n}{k}}{N^{2}}} \right) \left[ {\frac{r_0 \left( {m_0 -r_0 } \right) }{m_0 \left( {m_0 +1} \right) }} \right] +\frac{m_0 ^{2}}{N^{2}}\left[ {\frac{r_0 \left( {m_0 -r_0 } \right) }{m_0 ^{2}\left( {m_0 +1} \right) }} \right] \\&\quad =\left( {1+\frac{m_0 +m+\frac{n}{k}}{N^{2}}} \right) \left[ {\frac{r_0 \left( {m_0 -r_0 } \right) }{m_0 \left( {m_0 +1} \right) }} \right] . \end{aligned}$$

For a private-information-randomizing crowd, the crowd proportion will behave as if it were the average of k independent \(r_{i}\) from players with private information consisting of \(n= 1\) trial. To see this compare a player who first observes \(r_{i}\) heads in n flips of a coin and then picks 1 with probability \(r_{i}/n\) to a player who simply sets his pick equal to the outcome from one flip of the coin. In both cases, the players pick 1 independently (from all other players) with probability P (the actual probability the coin comes up heads).

This means that

$$\begin{aligned} \bar{R} _k =\bar{r} _k , \end{aligned}$$

where \(\bar{r}_k\) is equivalent to the average of k independent game outcomes. So that

$$\begin{aligned}{}[ {X-\bar{R} _k } ]^{2}=\left( {X-P} \right) ^{2}+\left( {\bar{r}_k -P} \right) ^{2}-2\left( {X-P} \right) \left( {\bar{r}_k -P} \right) , \end{aligned}$$

which has expected value given P of

$$\begin{aligned} \left[ {1+\left( {\frac{1}{k}} \right) } \right] P\left( {1-P} \right) \end{aligned}$$

and expected value

$$\begin{aligned} \mathrm{MSE}_{\text {private-information randomization}} =\left( {1+\frac{1}{k}} \right) \left[ {\frac{r_0 \left( {m_0 -r_0 } \right) }{m_0 \left( {m_0 +1} \right) }} \right] . \end{aligned}$$

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pfeifer, P.E. The promise of pick-the-winners contests for producing crowd probability forecasts. Theory Decis 81, 255–278 (2016). https://doi.org/10.1007/s11238-015-9533-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11238-015-9533-9

Keywords

Navigation