The wisdom of collective grading and the effects of epistemic and semantic diversity
A computer simulation is used to study collective judgements that an expert panel reaches on the basis of qualitative probability judgements contributed by individual members. The simulated panel displays a strong and robust crowd wisdom effect. The panel’s performance is better when members contribute precise probability estimates instead of qualitative judgements, but not by much. Surprisingly, it does not always hurt for panel members to interpret the probability expressions differently. Indeed, coordinating their understandings can be much worse.
KeywordsQualitative probability Scores and grades Wisdom of crowds Computer modeling
We thank for helpful comments Luc Bovens, Wlodek Rabinowicz and an anonymous reviewer, as well as audiences at Bristol and York Universities, the London School of Economics and the Munich Center for Mathematical Philosophy.
- Arrow, K. J. (1951). Social choice and individual values. New York: Wiley (2nd ed. 1963).Google Scholar
- Balinski, M., & Laraki, R. (2011). Majority judgement. Cambridge: MIT Press.Google Scholar
- Condorcet, J.-A.-N. d. C. (1785). Essai sur l’application de l’analyse a la probabilite des decisions rendues a la pluralite des voix [microform]/par M. le Marquis de Condorcet. Imprimerie royale Paris.Google Scholar
- List, C. (2013). Social choice theory. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Winter 2013 ed.).Google Scholar
- Lyon, A. (forthcoming). Collective wisdom. Journal of Philosophy.Google Scholar
- Lyon, A., & Pacuit, E. (2013). The Wisdom of crowds: Methods of human judgement aggregation. In: Michelucci P (Ed.), Springer handbook of human computation. (pp. 599–614). Springer.Google Scholar
- Page, S. (2008). The difference. Princeton: Princeton University Press.Google Scholar
- Sunstein, C. R. (2006). Deliberating groups versus prediction markets (or hayek’s challenge to habermas). Episteme, 3(03), 192–213.Google Scholar
- Surowiecki, J. (2004). The Wisdom of Crowds. New York: Doubleday.Google Scholar
- Tetlock, P. (2005). Expert political judgment: How good is it? How can we know?. Princeton: Princeton University Press.Google Scholar
- Wintle, B., Mascaro, S., Fidler, F., McBride, M., Burgman, M., Flander, L., Saw, G., Twardy, C., Lyon, A., & Manning, B. (2012). The intelligence game: Assessing Delphi Groups and structured question formats. In Proceedings of the 5th Australian Security and Intelligence Conference. http://ro.ecu.edu.au/cgi/viewcontent.cgi?article=1025&context=asi.