The Wisdom of Crowds: Methods of Human Judgement Aggregation

  • Aidan LyonEmail author
  • Eric Pacuit


Although the idea is an old one, there has been a recent boom in research into the Wisdom of Crowds, and this appears to be at least partly due to the now widespread availability of the Internet, and the advent of social media and Web 2.0 applications. In this paper, we start by laying out a simple conceptual framework for thinking about the Wisdom of the Crowds. We identify six core aspects that are part of any instance of the Wisdom of the Crowds. One of these aspects, called aggregation, is the main focus of this paper. An aggregation method is the method of bringing the many contributions of a crowd together into a collective output. We discuss three different types of aggregation methods: mathematical aggregation, group deliberation and prediction markets.


Aggregation Method Individual Judgement Judgement Aggregation Prediction Market Collective Judgement 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Amrstrong JS (2006) Should the forecasting process eliminate face-to-face meetings? Int J Appl Forecast 5:3–8Google Scholar
  2. Andler D (2012) What has Collective Wisdom to do with Wisdom? In: Landemore H, Elster J (eds) Collective wisdom. Cambridge University Press, Cambridge, pp 72–84Google Scholar
  3. Armstrong J (2001a) Principles of forecasting. Kluwer Academic, BostonCrossRefGoogle Scholar
  4. Armstrong S (2001b) Combining forecasts. In: Armstrong S (ed) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic, NorwellCrossRefGoogle Scholar
  5. Arrow KJ, Forsythe R, Gorham M, Hahn R, Hanson R, Ledyard JO, Levmore S, Litan R, Milgrom P, Nelson FD, Neumann GR, Ottaviani M, Schelling TC, Shiller RJ, Smith VL, Snowberg E, Sunstein CR, Tetlock PC, Tetlock PE, Varian HR, Wolfers J, Zitzewitz E (2008) The promise of prediciton markets. Science 320:877–878CrossRefGoogle Scholar
  6. Asan G, Sanver R (2002) Another characterization of majority rule. Econ Lett 75(3):409–413Google Scholar
  7. Bao P, Gerber E, Gergle D, Hoffman D (2010) Momentum: getting and staying on topic during a brainstorm. In: Proceedings of the SIGCHI conference on human factors in computing systems, Atlanta, pp 1233–1236Google Scholar
  8. Black D (1963) The theory of committees and elections. Springer, Norwell, MAGoogle Scholar
  9. Brownstein JS, Freifeld CC, Madoff LC (2009) Digital disease detection—harnessing the web for public health surveillance. N Engl J Med 360(21):2153–2157CrossRefGoogle Scholar
  10. Burgman M, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Twardy C (2011) Expert status and performance. PLoS One 6(7):e22998CrossRefGoogle Scholar
  11. Cariani F (2011) Judgment aggregation. Philos Compass 6(1):22–32CrossRefGoogle Scholar
  12. Chen Y, Pennock D (2010) Designing markets for prediction. AI Mag 31(4):42–52MathSciNetGoogle Scholar
  13. Chen Y, Dimitrov S, Sami R, Reeves D, Pennock D, Hanson R, Fortnow L, Gonen R (2010) Gaming prediction markets: equilibrium strategies with a market maker. Algorithmica 58(4):930–969MathSciNetCrossRefzbMATHGoogle Scholar
  14. Clemen RT (2008) Comment on cooke’s classical method. Reliab Eng Syst Saf 93(5):760–765CrossRefGoogle Scholar
  15. Collier N, Kawazoe A, Jin L, Shigematsu M, Dien D, Barrero R, Takeuchi K, Kawtrakul A (2006) A multilingual ontology for infectious disease surveillance: rationale, design and challenges. Lang Resour Eval 40(3):405–413Google Scholar
  16. Condorcet M (1785) Essai sur l’application de l’analyse á la probabilité des décisions rendues á la pluralité des voix. Paris: l’Imprimerie Royale. (Reprint, 1972, Chelsea, New York)Google Scholar
  17. Cooke RM (1991) Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New YorkGoogle Scholar
  18. Diehl M, Stroebe W (1987) Productivity loss in brainstorming groups: toward the solution of a riddle. J Personal Soc Psychol 53(3):497–509CrossRefGoogle Scholar
  19. Dietrich F (2012) Judgment aggregation and the discursive dilemma. In: Encyclopedia of philosophy and the social sciences. SageGoogle Scholar
  20. Fidler F, Wintle B, Thomason N (2013) Groups making wise judgements. Final report to IARPA (Intelligence Advanced Research Projects Activity)Google Scholar
  21. Frongillo R, Della Penna N, Reid M (2012) “Interpreting prediction markets: a stochastic approach.” In: Proceedings of neural information processing systemsGoogle Scholar
  22. Galton F (1907a) Letters to the editor: the ballot-box. Nature 75:900–1Google Scholar
  23. Galton F (1907b) Vox Populi. Nature 75:450–1CrossRefGoogle Scholar
  24. Gerber E (2009) Using improvisation to enhance the effectiveness of brainstorming. In: Proceedings of the SIGCHI conference on human factors in computing systems, Boston, pp 97–104Google Scholar
  25. Gigone D, Hastie R (1993) The common knowledge effect: informaiton sharing and group judgments. J Personal Soc Psychol 65(5):959–974CrossRefGoogle Scholar
  26. Graefe A, Armstrong JS (2011) Comparing face-to-face meetings, nominal groups, delphi and prediction markets on an estimation task. Int J Forecast 27(1):183–195CrossRefGoogle Scholar
  27. Grofman B, Owen G, Feld SL (1983) Thirteen theorems in search of the truth. Theory Decis 15(3):261–278MathSciNetCrossRefzbMATHGoogle Scholar
  28. Hanson R, Oprea R (2009) A manipulator can aid prediction market accuracy. Economica 76(302): 304–314Google Scholar
  29. Hanson R, Oprea R, Porter D (2006) Information aggregation and manipulation in an experimental market. J Econ Behav Organ 60(4):449–459Google Scholar
  30. Herzog SM, Hertwig R (2009) The wisdom of many in one mind improving individual judgments with dialectical bootstrapping. Psychol Sci 20(2):231–237CrossRefGoogle Scholar
  31. Hourihan KL, Benjamin AS (2010) Smaller is better (when sampling from the crowd within): low memory-span individuals benefit more from multiple opportunities for estimation. J Exp Psychol Learn Mem Cogn 36(4):1068CrossRefGoogle Scholar
  32. Keller M, Blench M, Tolentino H, Freifeld C, Mandl K, Mawudeku A, Eysenbach G, Brownstein J (2009) Use of unstructured event-based reports for global infectious disease surveillance. Emerg Infect Dis 15(5):689CrossRefGoogle Scholar
  33. Klayman J, Soll J, González-Vallejo C, Barlas S (1999) Overconfidence: it depends on how, what, and whom you ask. Organ behav Hum Decis Process 79(3):216–247CrossRefGoogle Scholar
  34. Koriat A (2012) When are two heads better than one and why? Science 336:360–2CrossRefGoogle Scholar
  35. Ladha KK (1992) The condorcet Jury theorem, free speech, and correlated votes. Am J Political Sci 36(3):617–634Google Scholar
  36. Landemore H, Elster J (2012) Collective wisdom: principes and mechanisms. Cambridge University Press, New YorkCrossRefGoogle Scholar
  37. Lehrer K, Wagner C (1981) Rational consensus in science and society: a philosophical and mathematical study, Vol 24. D Reidel, DordrechtCrossRefzbMATHGoogle Scholar
  38. Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley, ReadingzbMATHGoogle Scholar
  39. List C (2012) The theory of judgment aggregation: an introductory review. Synthese 187(1):179–207MathSciNetCrossRefzbMATHGoogle Scholar
  40. List C, Goodin RE (2002) Epistemic democracy: generalizing the condorcet Jury theorem. J Political Philos 9(3):277–306CrossRefGoogle Scholar
  41. List C, Pettit P (2002) Aggregating sets of judgments: an impossibility result. Econ Philos 18(01):89–110Google Scholar
  42. Lo, A (2007) Efficient market hypothesis, in The New Palgrave: A Dictionary of Economics, L. Blume, S. Durlauf (eds), Palgrave MacmillanGoogle Scholar
  43. Loewer B, Laddaga R (1985) Destroying the consensus. Synthese 62(1):79–95CrossRefGoogle Scholar
  44. Lyon A, Fidler F, Burgman M (2012a) Judgement swapping and aggregation. In: 2012 AAAI fall symposium seriesGoogle Scholar
  45. Lyon A, Nunn M, Grossel G, Burgman M (2012b) Comparison of web-based biosecurity intelligence systems: biocaster, epispider and healthmap. Transboundary Emerg Dis 59(3):223–232CrossRefGoogle Scholar
  46. Lyon A, Grossel G, Nunn M, Burgman M (2013) Using internet intelligence to manage biosecurity risks: a case study for aquatic animal health. Divers Distrib 19(5–6):640–650CrossRefGoogle Scholar
  47. Manski C (2006) Interpreting the predictions of prediction markets. Econ Lett 91(3):425–429CrossRefGoogle Scholar
  48. Maskin E (1995) Majority rule, social welfare functions and game forms. In: Choice, welfare and development: a festschrift in honour of Amartya K. Sen. Oxford University Press, Oxford, pp 100–109Google Scholar
  49. May K (1952) A set of independent necessary and sufficient conditions for simply majority decision. Econometrica 20(4):680–684CrossRefzbMATHGoogle Scholar
  50. Nielsen M (2011) Reinventing discovery: the new era of networked science. Princeton University Press, PrincetonGoogle Scholar
  51. Nuwer R (2013) Software could make rare diseases easier to spot. New Sci 218(2913):21CrossRefGoogle Scholar
  52. Othman A, Sandholm T (2010) When do markets with simple agents fail? In: Proceedings of the 9th international conference on autonomous agents and multiagent systems, AAMAS ’10, Toronto, vol 1, pp 865–872Google Scholar
  53. Pacuit E (2012) Voting methods. In: Zalta EN (ed) The stanford encyclopedia of philosophy. (Winter 2012 edn)Google Scholar
  54. Page S (2008) The difference: how the power of diversity creates better groups, firms, schools, and societies (new edn). Princeton University Press, PrincetonGoogle Scholar
  55. Pennock DM, Sami R (2007) Computational aspects of prediction markets. In: Nisan N, Roughgarden T, Tardos E, Vazirani V (eds) Algorithmic game theory. Cambridge University Press, Cambridge/New YorkGoogle Scholar
  56. Plummer B (2012) How to swing the prediction markets and boost Mitt Romney’s fortunes, The Washington Post Wonkblog, October 23Google Scholar
  57. Polymath DHJ (2012) A new proof of the density Hales-Jewett theorem. Ann Math 175(3):1283–1327MathSciNetCrossRefzbMATHGoogle Scholar
  58. Regan HM, Colyvan M, Markovchick-Nicholls L (2006) A formal model for consensus and negotiation in environmental management. J Environ Manag 80(2):167–176CrossRefGoogle Scholar
  59. Rothschild D (2009) Forecasting elections: comparing prediction markets, polls and their biases. Public Opin Q 73(5):895–916CrossRefGoogle Scholar
  60. Sunstein C (2011) Deliberating groups versus prediction markets or Hayek’s challenge to habermas. In: Social epistemology: essential readings. Oxford University Press, Oxford/New York, pp 314–337Google Scholar
  61. Surowiecki J (2005) The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday, New YorkGoogle Scholar
  62. Thaler R, Ziemba W (1988) Anomalies: parimutuel betting markets: racetracks and lotteries. J Econ Perspect 2(2):161–174CrossRefGoogle Scholar
  63. Von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) Recaptcha: human-based character recognition via web security measures. Science 321(5895):1465–1468MathSciNetCrossRefzbMATHGoogle Scholar
  64. Vul E, Pashler H (2008) Measuring the crowd within probabilistic representations within individuals. Psychol Sci 19(7):645–647CrossRefGoogle Scholar
  65. Woeginger G (2003) A new characterizaiton of the majority rule. Econ Lett 81(1):89–94MathSciNetCrossRefzbMATHGoogle Scholar
  66. Wolfers J, Zitzewitz E (2004) Prediction markets. Journal of Economic Perspectives, 18(2):107–126Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.PhilosophyUniversity of MarylandCollege ParkUSA
  2. 2.Munich Centre for Mathematical PhilosophyLudwig Maximilian University of MunichMunichGermany

Personalised recommendations