Theory and Decision

, Volume 85, Issue 1, pp 99–116 | Cite as

The wisdom of collective grading and the effects of epistemic and semantic diversity

  • Aidan Lyon
  • Michael Morreau


A computer simulation is used to study collective judgements that an expert panel reaches on the basis of qualitative probability judgements contributed by individual members. The simulated panel displays a strong and robust crowd wisdom effect. The panel’s performance is better when members contribute precise probability estimates instead of qualitative judgements, but not by much. Surprisingly, it does not always hurt for panel members to interpret the probability expressions differently. Indeed, coordinating their understandings can be much worse.


Qualitative probability Scores and grades Wisdom of crowds Computer modeling 



We thank for helpful comments Luc Bovens, Wlodek Rabinowicz and an anonymous reviewer, as well as audiences at Bristol and York Universities, the London School of Economics and the Munich Center for Mathematical Philosophy.


  1. Alcantud, J. C. R., & Laruelle, A. (2014). Dis&approval voting: A characterization. Social Choice and Welfare, 43(1), 1–10.CrossRefGoogle Scholar
  2. Aleskerov, F., Yakuba, V., & Yuzbashev, D. (2007). A ‘threshold aggregation’ of three-graded rankings. Mathematical Social Sciences, 53(1), 106–110.CrossRefGoogle Scholar
  3. Arrow, K. J. (1951). Social choice and individual values. New York: Wiley (2nd ed. 1963).Google Scholar
  4. Balinski, M., & Laraki, R. (2007). A theory of measuring, electing, and ranking. Proceedings of the National Academy of Science United States of America, 104(21), 8720–8725.CrossRefGoogle Scholar
  5. Balinski, M., & Laraki, R. (2011). Majority judgement. Cambridge: MIT Press.Google Scholar
  6. Beisbart, C., & Bovens, L. (2007). Welfarist evaluations of decision rules for boards of representatives. Social Choice and Welfare, 29(4), 581–608.CrossRefGoogle Scholar
  7. Beisbart, C., Bovens, L., & Hartmann, S. (2005). A utilitarian assessment of alternative decision rules in the council of ministers. European Union Politics, 6(4), 395–419.CrossRefGoogle Scholar
  8. Beisbart, C., & Hartmann, S. (2010). Welfarist evaluations of decision rules under interstate utility dependencies. Social Choice and Welfare, 34(2), 315–344.CrossRefGoogle Scholar
  9. Brams, S. J., & Fishburn, P. C. (1978). Approval voting. American Political Science Review, 72(3), 831–847.CrossRefGoogle Scholar
  10. Condorcet, J.-A.-N. d. C. (1785). Essai sur l’application de l’analyse a la probabilite des decisions rendues a la pluralite des voix [microform]/par M. le Marquis de Condorcet. Imprimerie royale Paris.Google Scholar
  11. Gaertner, W., & Xu, Y. (2012). A general scoring rule. Mathematical Social Sciences, 63(3), 193–196.CrossRefGoogle Scholar
  12. Galton, F. (1907). Vox Populi. Nature, 75, 450–1.CrossRefGoogle Scholar
  13. Hubbard, D., & Evans, D. (2010). Problems with scoring methods and ordinal scales in risk assessment. IBM Journal of Research and Development, 54, 246–255.CrossRefGoogle Scholar
  14. List, C. (2013). Social choice theory. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Winter 2013 ed.).Google Scholar
  15. Lyon, A. (forthcoming). Collective wisdom. Journal of Philosophy.Google Scholar
  16. Lyon, A., & Pacuit, E. (2013). The Wisdom of crowds: Methods of human judgement aggregation. In: Michelucci P (Ed.), Springer handbook of human computation. (pp. 599–614). Springer.Google Scholar
  17. Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., et al. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106–1115.CrossRefGoogle Scholar
  18. Morgan, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Science United States of America, 111(20), 7176–7184.CrossRefGoogle Scholar
  19. Morreau, M. (2016). Grading in groups. Economics and Philosophy, 32(2), 323–352.CrossRefGoogle Scholar
  20. Nielsen, M. (2011). Reinventing discovery: The new era of networked science. Princeton: Princeton University Press.CrossRefGoogle Scholar
  21. Ohnishi, M., Fukui, T., Matsui, K., Hira, K., Shinozuka, M., Ezaki, H., et al. (2002). Interpretation of and preference for probability expressions among Japanese patients and physicians. Family Practice, 19(1), 7–11.CrossRefGoogle Scholar
  22. Page, S. (2008). The difference. Princeton: Princeton University Press.Google Scholar
  23. Pivato, M. (2016). Asymptotic utilitarianism in scoring rules. Social Choice and Welfare, 47(2), 431–458.CrossRefGoogle Scholar
  24. Sunstein, C. R. (2006). Deliberating groups versus prediction markets (or hayek’s challenge to habermas). Episteme, 3(03), 192–213.Google Scholar
  25. Surowiecki, J. (2004). The Wisdom of Crowds. New York: Doubleday.Google Scholar
  26. Tetlock, P. (2005). Expert political judgment: How good is it? How can we know?. Princeton: Princeton University Press.Google Scholar
  27. Wallsten, T. S., Budescu, D. V., Rapoport, A., Zwick, R., & Forsyth, B. (1986). Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General, 155(4), 348–365.CrossRefGoogle Scholar
  28. Wardekker, J. A., van der Sluijs, J. P., Janssen, P. H. M., Kloprogge, P., & Petersen, A. C. (2008). Uncertainty communication in environmental assessments: views from the Dutch science-policy interface. Environmental Science and Policy, 11, 627–641.CrossRefGoogle Scholar
  29. Wintle, B., Mascaro, S., Fidler, F., McBride, M., Burgman, M., Flander, L., Saw, G., Twardy, C., Lyon, A., & Manning, B. (2012). The intelligence game: Assessing Delphi Groups and structured question formats. In Proceedings of the 5th Australian Security and Intelligence Conference.
  30. Wolf, M., Krause, J., Carney, P. A., Bogart, A., & Kurvers, R. H. (2015). Collective intelligence meets medical decision-making: The collective outperforms the best radiologist. PloS One, 10(8), e0134269.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.University of Maryland at College ParkCollege ParkUSA
  2. 2.UiT-The Arctic University of NorwayTromsøNorway

Personalised recommendations