Skip to main content
SpringerLink
Log in

The Wisdom of Crowds: Methods of Human Judgement Aggregation

  • Chapter
  • First Online:
Handbook of Human Computation

Abstract

Although the idea is an old one, there has been a recent boom in research into the Wisdom of Crowds, and this appears to be at least partly due to the now widespread availability of the Internet, and the advent of social media and Web 2.0 applications. In this paper, we start by laying out a simple conceptual framework for thinking about the Wisdom of the Crowds. We identify six core aspects that are part of any instance of the Wisdom of the Crowds. One of these aspects, called aggregation, is the main focus of this paper. An aggregation method is the method of bringing the many contributions of a crowd together into a collective output. We discuss three different types of aggregation methods: mathematical aggregation, group deliberation and prediction markets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We say “so-called”, because examples of the Wisdom of Crowds often have little to do with the notion of wisdom that philosophers care about (see Andler (2012) for further discussion), and they often involve only a group of people—even just a handful—and not a crowd in the usual sense of the word. Nevertheless, we will stick with the words that seem to have stuck.

  2. 2.

    Psychologists have found that even just single person can function as a crowd of individuals (see e.g., Vul and Pashler 2008; Herzog and Hertwig 2009; Hourihan and Benjamin 2010).

  3. 3.

    For example, it should rain on 90 % of the days that your crowd says there is a 90 % chance of rain.

  4. 4.

    This is because N and N → G entail G, so if the former two propositions are true, the latter has to be true.

  5. 5.

    As we noted in section “Thinking About the Wisdom of Crowds”, not all deliberation groups are instances of judgement aggregation. For example, the crowd could simply meet to share information and then still give different individual judgements, which could then be aggregated using one of the methods described in sections “Thinking About the Wisdom of Crowds” or “Prediction Markets”.

References

  • Amrstrong JS (2006) Should the forecasting process eliminate face-to-face meetings? Int J Appl Forecast 5:3–8

    Google Scholar 

  • Andler D (2012) What has Collective Wisdom to do with Wisdom? In: Landemore H, Elster J (eds) Collective wisdom. Cambridge University Press, Cambridge, pp 72–84

    Google Scholar 

  • Armstrong J (2001a) Principles of forecasting. Kluwer Academic, Boston

    Book  Google Scholar 

  • Armstrong S (2001b) Combining forecasts. In: Armstrong S (ed) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic, Norwell

    Chapter  Google Scholar 

  • Arrow KJ, Forsythe R, Gorham M, Hahn R, Hanson R, Ledyard JO, Levmore S, Litan R, Milgrom P, Nelson FD, Neumann GR, Ottaviani M, Schelling TC, Shiller RJ, Smith VL, Snowberg E, Sunstein CR, Tetlock PC, Tetlock PE, Varian HR, Wolfers J, Zitzewitz E (2008) The promise of prediciton markets. Science 320:877–878

    Article  Google Scholar 

  • Asan G, Sanver R (2002) Another characterization of majority rule. Econ Lett 75(3):409–413

    Google Scholar 

  • Bao P, Gerber E, Gergle D, Hoffman D (2010) Momentum: getting and staying on topic during a brainstorm. In: Proceedings of the SIGCHI conference on human factors in computing systems, Atlanta, pp 1233–1236

    Google Scholar 

  • Black D (1963) The theory of committees and elections. Springer, Norwell, MA

    Google Scholar 

  • Brownstein JS, Freifeld CC, Madoff LC (2009) Digital disease detection—harnessing the web for public health surveillance. N Engl J Med 360(21):2153–2157

    Article  Google Scholar 

  • Burgman M, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Twardy C (2011) Expert status and performance. PLoS One 6(7):e22998

    Article  Google Scholar 

  • Cariani F (2011) Judgment aggregation. Philos Compass 6(1):22–32

    Article  Google Scholar 

  • Chen Y, Pennock D (2010) Designing markets for prediction. AI Mag 31(4):42–52

    MathSciNet  Google Scholar 

  • Chen Y, Dimitrov S, Sami R, Reeves D, Pennock D, Hanson R, Fortnow L, Gonen R (2010) Gaming prediction markets: equilibrium strategies with a market maker. Algorithmica 58(4):930–969

    Article  MathSciNet  MATH  Google Scholar 

  • Clemen RT (2008) Comment on cooke’s classical method. Reliab Eng Syst Saf 93(5):760–765

    Article  Google Scholar 

  • Collier N, Kawazoe A, Jin L, Shigematsu M, Dien D, Barrero R, Takeuchi K, Kawtrakul A (2006) A multilingual ontology for infectious disease surveillance: rationale, design and challenges. Lang Resour Eval 40(3):405–413

    Google Scholar 

  • Condorcet M (1785) Essai sur l’application de l’analyse á la probabilité des décisions rendues á la pluralité des voix. Paris: l’Imprimerie Royale. (Reprint, 1972, Chelsea, New York)

    Google Scholar 

  • Cooke RM (1991) Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New York

    Google Scholar 

  • Diehl M, Stroebe W (1987) Productivity loss in brainstorming groups: toward the solution of a riddle. J Personal Soc Psychol 53(3):497–509

    Article  Google Scholar 

  • Dietrich F (2012) Judgment aggregation and the discursive dilemma. In: Encyclopedia of philosophy and the social sciences. Sage

    Google Scholar 

  • Fidler F, Wintle B, Thomason N (2013) Groups making wise judgements. Final report to IARPA (Intelligence Advanced Research Projects Activity)

    Google Scholar 

  • Frongillo R, Della Penna N, Reid M (2012) “Interpreting prediction markets: a stochastic approach.” In: Proceedings of neural information processing systems

    Google Scholar 

  • Galton F (1907a) Letters to the editor: the ballot-box. Nature 75:900–1

    Google Scholar 

  • Galton F (1907b) Vox Populi. Nature 75:450–1

    Article  Google Scholar 

  • Gerber E (2009) Using improvisation to enhance the effectiveness of brainstorming. In: Proceedings of the SIGCHI conference on human factors in computing systems, Boston, pp 97–104

    Google Scholar 

  • Gigone D, Hastie R (1993) The common knowledge effect: informaiton sharing and group judgments. J Personal Soc Psychol 65(5):959–974

    Article  Google Scholar 

  • Graefe A, Armstrong JS (2011) Comparing face-to-face meetings, nominal groups, delphi and prediction markets on an estimation task. Int J Forecast 27(1):183–195

    Article  Google Scholar 

  • Grofman B, Owen G, Feld SL (1983) Thirteen theorems in search of the truth. Theory Decis 15(3):261–278

    Article  MathSciNet  MATH  Google Scholar 

  • Hanson R, Oprea R (2009) A manipulator can aid prediction market accuracy. Economica 76(302): 304–314

    Google Scholar 

  • Hanson R, Oprea R, Porter D (2006) Information aggregation and manipulation in an experimental market. J Econ Behav Organ 60(4):449–459

    Google Scholar 

  • Herzog SM, Hertwig R (2009) The wisdom of many in one mind improving individual judgments with dialectical bootstrapping. Psychol Sci 20(2):231–237

    Article  Google Scholar 

  • Hourihan KL, Benjamin AS (2010) Smaller is better (when sampling from the crowd within): low memory-span individuals benefit more from multiple opportunities for estimation. J Exp Psychol Learn Mem Cogn 36(4):1068

    Article  Google Scholar 

  • Keller M, Blench M, Tolentino H, Freifeld C, Mandl K, Mawudeku A, Eysenbach G, Brownstein J (2009) Use of unstructured event-based reports for global infectious disease surveillance. Emerg Infect Dis 15(5):689

    Article  Google Scholar 

  • Klayman J, Soll J, González-Vallejo C, Barlas S (1999) Overconfidence: it depends on how, what, and whom you ask. Organ behav Hum Decis Process 79(3):216–247

    Article  Google Scholar 

  • Koriat A (2012) When are two heads better than one and why? Science 336:360–2

    Article  Google Scholar 

  • Ladha KK (1992) The condorcet Jury theorem, free speech, and correlated votes. Am J Political Sci 36(3):617–634

    Google Scholar 

  • Landemore H, Elster J (2012) Collective wisdom: principes and mechanisms. Cambridge University Press, New York

    Book  Google Scholar 

  • Lehrer K, Wagner C (1981) Rational consensus in science and society: a philosophical and mathematical study, Vol 24. D Reidel, Dordrecht

    Book  MATH  Google Scholar 

  • Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley, Reading

    MATH  Google Scholar 

  • List C (2012) The theory of judgment aggregation: an introductory review. Synthese 187(1):179–207

    Article  MathSciNet  MATH  Google Scholar 

  • List C, Goodin RE (2002) Epistemic democracy: generalizing the condorcet Jury theorem. J Political Philos 9(3):277–306

    Article  Google Scholar 

  • List C, Pettit P (2002) Aggregating sets of judgments: an impossibility result. Econ Philos 18(01):89–110

    Google Scholar 

  • Lo, A (2007) Efficient market hypothesis, in The New Palgrave: A Dictionary of Economics, L. Blume, S. Durlauf (eds), Palgrave Macmillan

    Google Scholar 

  • Loewer B, Laddaga R (1985) Destroying the consensus. Synthese 62(1):79–95

    Article  Google Scholar 

  • Lyon A, Fidler F, Burgman M (2012a) Judgement swapping and aggregation. In: 2012 AAAI fall symposium series

    Google Scholar 

  • Lyon A, Nunn M, Grossel G, Burgman M (2012b) Comparison of web-based biosecurity intelligence systems: biocaster, epispider and healthmap. Transboundary Emerg Dis 59(3):223–232

    Article  Google Scholar 

  • Lyon A, Grossel G, Nunn M, Burgman M (2013) Using internet intelligence to manage biosecurity risks: a case study for aquatic animal health. Divers Distrib 19(5–6):640–650

    Article  Google Scholar 

  • Manski C (2006) Interpreting the predictions of prediction markets. Econ Lett 91(3):425–429

    Article  Google Scholar 

  • Maskin E (1995) Majority rule, social welfare functions and game forms. In: Choice, welfare and development: a festschrift in honour of Amartya K. Sen. Oxford University Press, Oxford, pp 100–109

    Google Scholar 

  • May K (1952) A set of independent necessary and sufficient conditions for simply majority decision. Econometrica 20(4):680–684

    Article  MATH  Google Scholar 

  • Nielsen M (2011) Reinventing discovery: the new era of networked science. Princeton University Press, Princeton

    Google Scholar 

  • Nuwer R (2013) Software could make rare diseases easier to spot. New Sci 218(2913):21

    Article  Google Scholar 

  • Othman A, Sandholm T (2010) When do markets with simple agents fail? In: Proceedings of the 9th international conference on autonomous agents and multiagent systems, AAMAS ’10, Toronto, vol 1, pp 865–872

    Google Scholar 

  • Pacuit E (2012) Voting methods. In: Zalta EN (ed) The stanford encyclopedia of philosophy. (Winter 2012 edn)

    Google Scholar 

  • Page S (2008) The difference: how the power of diversity creates better groups, firms, schools, and societies (new edn). Princeton University Press, Princeton

    Google Scholar 

  • Pennock DM, Sami R (2007) Computational aspects of prediction markets. In: Nisan N, Roughgarden T, Tardos E, Vazirani V (eds) Algorithmic game theory. Cambridge University Press, Cambridge/New York

    Google Scholar 

  • Plummer B (2012) How to swing the prediction markets and boost Mitt Romney’s fortunes, The Washington Post Wonkblog, October 23

    Google Scholar 

  • Polymath DHJ (2012) A new proof of the density Hales-Jewett theorem. Ann Math 175(3):1283–1327

    Article  MathSciNet  MATH  Google Scholar 

  • Regan HM, Colyvan M, Markovchick-Nicholls L (2006) A formal model for consensus and negotiation in environmental management. J Environ Manag 80(2):167–176

    Article  Google Scholar 

  • Rothschild D (2009) Forecasting elections: comparing prediction markets, polls and their biases. Public Opin Q 73(5):895–916

    Article  Google Scholar 

  • Sunstein C (2011) Deliberating groups versus prediction markets or Hayek’s challenge to habermas. In: Social epistemology: essential readings. Oxford University Press, Oxford/New York, pp 314–337

    Google Scholar 

  • Surowiecki J (2005) The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday, New York

    Google Scholar 

  • Thaler R, Ziemba W (1988) Anomalies: parimutuel betting markets: racetracks and lotteries. J Econ Perspect 2(2):161–174

    Article  Google Scholar 

  • Von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) Recaptcha: human-based character recognition via web security measures. Science 321(5895):1465–1468

    Article  MathSciNet  MATH  Google Scholar 

  • Vul E, Pashler H (2008) Measuring the crowd within probabilistic representations within individuals. Psychol Sci 19(7):645–647

    Article  Google Scholar 

  • Woeginger G (2003) A new characterizaiton of the majority rule. Econ Lett 81(1):89–94

    Article  MathSciNet  MATH  Google Scholar 

  • Wolfers J, Zitzewitz E (2004) Prediction markets. Journal of Economic Perspectives, 18(2):107–126

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Philosophy, University of Maryland, College Park, MD, USA

    Aidan Lyon & Eric Pacuit

  2. Munich Centre for Mathematical Philosophy, Ludwig Maximilian University of Munich, Munich, Germany

    Aidan Lyon

Authors
  1. Aidan Lyon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Pacuit
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aidan Lyon .

Editor information

Editors and Affiliations

  1. ThinkSplash, LLC, Fairfax, Virginia, USA

    Pietro Michelucci

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Lyon, A., Pacuit, E. (2013). The Wisdom of Crowds: Methods of Human Judgement Aggregation. In: Michelucci, P. (eds) Handbook of Human Computation. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8806-4_47

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-8806-4_47

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-8805-7

  • Online ISBN: 978-1-4614-8806-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation