Abstract
We apply cognitive modeling to improve the wisdom of the crowd in a spatial knowledge task. Participants provided point estimates for where 48 US cities are located and then, using the point estimate as a center point, chose a radius large enough that they believed the resulting circle was certain to contain the city’s location. Simple and radius-weighted arithmetic averages of the individuals’ point estimates produced more accurate group answers than the majority of individuals. These statistical aggregates, however, assume there are no differences in individual expertise nor in the difficulty of locating different cities. Accordingly, we develop a set of cognitive models to infer group estimates that make various assumptions about individual expertise and differences in city difficulty. The model-based estimates generally outperform the statistical averages. The models are especially accurate if they allow for individual differences in expertise that can vary city by city. We replicate this finding by applying the same cognitive models to data reported by Mayer and Heck (2023) in which participants provided point estimates for the locations of European cities.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data and Code Availability
The US cities’ data set is available on an Open Science Framework project page at https://www.osf.io/ve8t9/. The Mayer and Heck (2023) data set is available on an Open Science Framework project page at https://www.osf.io/jbzk7/. The JAGS code, code for any analyses presented in this paper, and other supplementary materials are available on an Open Science Framework project page at https://www.osf.io/ve8t9/.
References
Abi-Zeid, I., & Frost, J. R. (2005). SARPlan: A decision support system for Canadian Search and Rescue operations. European Journal of Operational Research, 162(3), 630–653. https://doi.org/10.1016/j.ejor.2003.10.029
Anders, R., Oravecz, Z., & Batchelder, W. H. (2014). Cultural consensus theory for continuous responses: A latent appraisal model for information pooling. Journal of Mathematical Psychology, 61, 1–13. https://doi.org/10.1016/j.jmp.2014.06.001
Aydin, B. I., Yilmaz, Y. S., Li, Y., Li, Q., Gao, J., & Demirbas, M. (2014). Crowdsourcing for multiple-choice question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 28(2), 2946–2953. https://doi.org/10.1609/aaai.v28i2.19016
Bennett, S. T., Benjamin, A. S., Mistry, P. K., & Steyvers, M. (2018). Making a wiser crowd: Benefits of individual metacognitive control on crowd performance. Computational Brain & Behavior, 1, 90–99. https://doi.org/10.1007/s42113-018-0006-4
Brooks, S. P., & Gelman, A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434–455. https://doi.org/10.1080/10618600.1998.10474787
Budescu, D. V., & Chen, E. (2014). Identifying expertise to extract the wisdom of crowds. Management Science, 61(2), 267–280. https://doi.org/10.1287/mnsc.2014.1909
Butler, D., Butler, R., & Eakins, J. (2021). Expert performance and crowd wisdom: Evidence from English Premier League predictions. European Journal of Operational Research, 288(1), 170–182. https://doi.org/10.1016/j.ejor.2020.05.034
Da, Z., & Huang, X. (2019). Harnessing the wisdom of crowds. Management Science, 66(5), 1847–1867. https://doi.org/10.1287/mnsc.2019.3294
Danileiko, I., & Lee, M. D. (2018). A model-based approach to the wisdom of the crowd in category learning. Cognitive Science, 42(S3), 861–883. https://doi.org/10.1111/cogs.12561
Davis-Stober, C. P., Budescu, D. V., Broomell, S. B., & Dana, J. (2015). The composition of optimally wise crowds. Decision Analysis, 12(3), 130–143. https://doi.org/10.1287/deca.2015.0315
Davis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014). When is a crowd wise? Decision, 1(2), 79–101. https://doi.org/10.1037/dec0000004
Deary, I. J. (2020). Intelligence: A very short introduction. Oxford University Press, 2nd edition. https://doi.org/10.1093/actrade/9780198796206.001.0001
Fiechter, J. L. & Kornell, N. (2021). How the wisdom of crowds, and of the crowd within, are affected by expertise. Cognitive Research: Principles and Implications, 6(5). https://doi.org/10.1186/s41235-021-00273-6
Friedman, A., Brown, N. R., & Mcgaffey, A. P. (2002). A basis for bias in geographical judgments. Psychonomic Bulletin & Review, 9(1), 151–159. https://doi.org/10.3758/bf03196272
Friedman, A., Kerkman, D. D., & Brown, N. R. (2002). Spatial location judgments: A cross-national comparison of estimation bias in subjective North American geography. Psychonomic Bulletin & Review, 9(3), 615–623. https://doi.org/10.3758/bf03196321
Friedman, A., Kerkman, D. D., Brown, N. R., Stea, D., & Cappello, H. M. (2005). Cross-cultural similarities and differences in North Americans’ geographic location judgments. Psychonomic Bulletin & Review, 12(6), 1054–1060. https://doi.org/10.3758/bf03206443
Friedman, A., Mohr, C., & Brugger, P. (2012). Representational pseudoneglect and reference points both influence geographic location estimates. Psychonomic Bulletin & Review, 19, 277–284. https://doi.org/10.3758/s13423-011-0202-x
Galton, F. (1907). Vox populi. Nature, 75, 450–451. https://doi.org/10.1038/075450a0
Goldstein, D. G., McAfee, R. P., & Suri, S. (2014). The wisdom of smaller, smarter crowds. In Proceedings of the fifteenth ACM conference on economics and computation, EC ’14 (pp. 471–488). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2600057.2602886
Gordon, K. (1924). Group judgments in the field of lifted weights. Journal of Experimental Psychology, 7(5), 398–400. https://doi.org/10.1037/h0074666
Herzog, S. M., & Hertwig, R. (2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20(2), 231–237. https://doi.org/10.1111/j.1467-9280.2009.02271.x
Herzog, S. M., & Hertwig, R. (2014). Think twice and then: Combining or choosing in dialectical bootstrapping? Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(1), 218–232. https://doi.org/10.1037/a0034054
Himmelstein, M., Budescu, D. V., & Ho, E. H. (2023). The wisdom of many in few: Finding individuals who are as wise as the crowd. Journal of Experimental Psychology: General, 152(5), 1223–1244. https://doi.org/10.1037/xge0001340
Hora, S. C. (2004). Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 50(5), 597–604. https://doi.org/10.1287/mnsc.1040.0205
Keck, S., & Tang, W. (2020). Enhancing the wisdom of the crowd with cognitive-process diversity: The benefits of aggregating intuitive and analytical judgments. Psychological Science, 31(10), 1272–1282. https://doi.org/10.1177/0956797620941840
Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77(3), 217–273. https://doi.org/10.1016/0001-6918(91)90036-y
Klugman, S. F. (1947). Group and individual judgments for anticipated events. The Journal of Social Psychology, 26(1), 21–28. https://doi.org/10.1080/00224545.1947.9921728
Knight, H. C. (1921). A comparison of the reliability of group and individual judgments. Master’s thesis, Columbia University.
Larrick, R. P., Burson, K. A., & Soll, J. B. (2007). Social comparison and confidence: When thinking you’re better than average predicts overconfidence (and when it does not). Organizational Behavior and Human Decision Processes, 102(1), 76–94. https://doi.org/10.1016/j.obhdp.2006.10.002
Lee, M. D. (2024 submitted). Using cognitive models to improve the wisdom of the crowd. Manuscript submitted for publication.
Lee, M. D., & Danileiko, I. (2014). Using cognitive models to combine probability estimates. Judgment and Decision Making, 9(3), 258–272. https://doi.org/10.1017/S1930297500005799
Lee, M. D., Steyvers, M., & Miller, B. (2014). A cognitive model for aggregating people’s rankings. PLoS ONE, 9(5). https://doi.org/10.1371/journal.pone.0096431
Lee, M. D., Steyvers, M., de Young, M., & Miller, B. (2012). Inferring expertise in knowledge and prediction ranking tasks. Topics in Cognitive Science, 4(1), 151–163. https://doi.org/10.1111/j.1756-8765.2011.01175.x
Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge University Press. https://doi.org/10.1017/cbo9781139087759
Lee, M. D., Zhang, S., & Shi, J. (2011). The wisdom of the crowd playing the price is right. Memory & Cognition, 39, 914–923. https://doi.org/10.3758/s13421-010-0059-7
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1977). Calibration of probabilities: The state of the art. In H. Jungermann & G. De Zeeuw (Eds.), Decision making and change in human affairs: Proceedings of the fifth research conference on subjective probability, utility, and decision making, Darmstadt, 1–4 September, 1975 (pp. 275–324). Springer Netherlands. https://doi.org/10.1007/978-94-010-1276-8_19
Lin, L., & Goodrich, M. A. (2010). A Bayesian approach to modeling lost person behaviors based on terrain features in Wilderness Search and Rescue. Computational and Mathematical Organization Theory, 16, 300–323. https://doi.org/10.1007/s10588-010-9066-2
Lyon, A. & Pacuit, E. (2013). The wisdom of crowds: Methods of human judgement aggregation. In P. Michelucci (Ed.), Handbook of Human Computation (pp. 599–614). Springer, New York. https://doi.org/10.1007/978-1-4614-8806-4_47
Mannes, A. E., Soll, J. B., & Larrick, R. P. (2014). The wisdom of select crowds. Journal of Personality and Social Psychology, 107(2), 276–299. https://doi.org/10.1037/a0036677
Mayer, M., & Heck, D. W. (2023). Cultural consensus theory for two-dimensional location judgments. Journal of Mathematical Psychology, 113, 102742. https://doi.org/10.1016/j.jmp.2022.102742
McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37(1), 1–10. https://doi.org/10.1016/j.intell.2008.08.004
Montgomery, L. & Lee, M. D. (2022). The wisdom of the crowd and framing effects in spatial knowledge. In J. Culbertson, A. Perfors, H. Rabagliati, & V. Ramenzoni (Eds.), Proceedings of the annual meeting of the cognitive science society, vol. 44. https://escholarship.org/uc/item/0h95m7m4
Olsson, H. & Loveday, J. (2015). A comparison of small crowd selection methods. In D. C. Noelle, R. Dale, A. S. Warlaumont, J. Yoshimi, T. Matlock, C. D. Jennings, & P. P. Maglio (Eds.), Proceedings of the thirty-seventh annual meeting of the cognitive science society (pp. 1769–1774). Cognitive Science Society, Austin, TX. https://cognitivesciencesociety.org/wp-content/uploads/2019/03/cogsci15_proceedings.pdf
Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In K. Hornik, F. Leisch, & A. Zeileis (Eds.), Proceedings of the 3rd international workshop on Distributed Statistical Computing (DSC 2003), vol. 124 (pp. 1–10). Vienna, Austria. https://www.r-project.org/conferences/DSC-2003/Proceedings/Plummer.pdf
Prelec, D., Seung, H. S., & McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541(7638), 532–535. https://doi.org/10.1038/nature21054
Romney, A. K., Weller, S. C., & Batchelder, W. H. (1986). Culture as consensus: A theory of culture and informant accuracy. American Anthropologist, 88(2), 313–338. https://doi.org/10.1525/aa.1986.88.2.02a00020
Ronis, D. L., & Yates, J. F. (1987). Components of probability judment accuracy: Individual consistency and effects of subject matter and assessment method. Organizational Behavior and Human Decision Processes, 40, 193–218. https://doi.org/10.1016/0749-5978(87)90012-4
Schvaneveldt, R. W., Durso, F. T., Goldsmith, T. E., Breen, T. J., Cooke, N. M., Tucker, R. G., & De Maio, J. C. (1985). Measuring the structure of expertise. International Journal of Man-Machine Studies, 23(6), 699–728. https://doi.org/10.1016/s0020-7373(85)80064-x
Shaw, M. E. (1932). A comparison of individuals and small groups in the rational solution of complex problems. The American Journal of Psychology, 44(3), 491–504. https://doi.org/10.2307/1415351
Simoiu, C., Sumanth, C., Mysore, A., & Goel, S. (2019). Studying the “wisdom of crowds" at scale. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 171–179. https://doi.org/10.1609/hcomp.v7i1.5271
Steyvers, M., Miller, B., Hemmer, P., & Lee, M. (2009). The wisdom of crowds in the recollection of order information. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, & A. Culotta (Eds.), Advances in neural information processing systems, vol. 22 (pp. 1785–1793). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2009/file/4c27cea8526af8cfee3be5e183ac9605-Paper.pdf
Surowiecki, J. (2004). The wisdom of crowds (1st ed.). New York: Doubleday.
Thomas, B., Coon, J., Westfall, H. A., & Lee, M. D. (2021). Model-based wisdom of the crowd for sequential decision-making tasks. Cognitive Science, 45(7). https://doi.org/10.1111/cogs.13011
Turner, B. M., Steyvers, M., Merkle, E. C., Budescu, D. V., & Wallsten, T. S. (2014). Forecast aggregation via recalibration. Machine Learning, 95, 261–289. https://doi.org/10.1007/s10994-013-5401-4
Vul, E., & Pashler, H. (2008). Measuring the crowd within: Probabilistic representations within individuals. Psychological Science, 19(7), 645–647. https://doi.org/10.1111/j.1467-9280.2008.02136.x
Wallsten, T. S., Budescu, D. V., & Zwick, R. (1993). Comparing the calibration and coherence of numerical and verbal probability judgments. Management Science, 39(2), 176–190. https://doi.org/10.1287/mnsc.39.2.176
Wysokiński, M., Marcjan, R., & Dajda, J. (2014). Decision support software for search & rescue operations. Procedia Computer Science, 35, 776–785. https://doi.org/10.1016/j.procs.2014.08.160
Yi, S. K. M., Steyvers, M., Lee, M. D., & Dry, M. J. (2012). The wisdom of the crowd in combinatorial problems. Cognitive Science, 36, 452–470. https://doi.org/10.1111/j.1551-6709.2011.01223.x
Acknowledgements
We thank the Bayesian Cognitive Modeling lab at the University of California, Irvine, for their feedback, and Daniel Heck, Brandon Turner, and an anonymous reviewer for useful comments.
Funding
This research was supported by the US Air Force Research Laboratory’s Continuous Learning Branch. LEM’s collaboration was enabled through an appointment to the Oak Ridge Institute for Science and Education (ORISE) Summer Research Internship Program. MDL’s collaboration was enabled through an appointment to the ORISE Faculty Research Program. The views expressed in this paper are those of the authors and do not reflect the official policy or position of the US Air Force, Department of Defense, or the US Government.
Author information
Authors and Affiliations
Contributions
LEM and MDL conceived and designed the study. LEM, CMB, and MDL designed the experiment, and CMB programmed the experiment. LEM collected the experimental data. LEM, MDL, and JV developed the model. LEM performed the modeling analysis. CMB wrote the first draft of the Stimuli and Procedure subsections. LEM wrote the rest of the first draft. LEM and MDL contributed to revisions of the paper. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics Approval
This project was approved via exempt self-determination by the University of California Irvine (UCI) Institutional Review Board (IRB). This project also made use of de-identified archival data that was obtained from https://www.osf.io/jbzk7/.
Consent to Participate
Informed consent was obtained from all individuals who participated in the study.
Consent for Publication
Not applicable
Competing Interests
The authors declare no competing interests.
Additional information
Parts of this research were presented at the 2022 Annual Joint Meeting of the Society for Mathematical Psychology and the International Conference on Cognitive Modeling, the Meeting of the European Mathematical Psychology Group in 2022, and the 58th Edwards Bayesian Research Conference held in 2023.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Montgomery, L.E., Baldini, C.M., Vandekerckhove, J. et al. Where’s Waldo, Ohio? Using Cognitive Models to Improve the Aggregation of Spatial Knowledge. Comput Brain Behav (2024). https://doi.org/10.1007/s42113-024-00200-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s42113-024-00200-0