The Repeatability of Human Swarms

  • Gregg WillcoxEmail author
  • Louis Rosenberg
  • Colin Domnauer
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1130)


Swarm Intelligence (SI) is a natural phenomenon in which social organisms amplify their decision-making abilities by forming real-time systems that converge on optimized solutions. It has been studied extensively in schools of fish, flocks of birds, and swarms of bees. In recent years, a new technology called Artificial Swarm Intelligence (ASI) has enabled human groups to form similar systems over computer networks. While “human swarms” have been shown to be more accurate than traditional methods for tapping the intelligence of human groups, the present study tests the repeatability of the answers that human swarms generate. Ten groups of 20 to 25 participants were asked to give subjective ratings on a set of 25 opinion-based questions. The groups answered by working together in real-time, connected by swarming algorithms. The results show that groups answering as swarms produce repeatable results, reaching the same answer as other groups 67% of the time. Additional analysis found that the repeatability of each swarm was significantly correlated with a Conviction Index (CI) metric computed from the real-time swarming data (r2 = 0.33, p < 0.01). For swarms that converged upon a solution with a Conviction Index (CI) > 85%, the repeatability was found to be greater than 90% and the likelihood that another swarm randomly sampled from a similar population would generate the same response was greater than 95% (p < 0.05). This provides powerful guidelines for groups using ASI technology to generate optimized forecasts, insights, and decisions from human swarms sampled from general populations.


Artificial Swarm Intelligence Human swarms Repeatability Reliability Market research 



Thanks to Chris Hornbostel for his efforts in coordinating the swarms. Also, thanks to Unanimous AI for the use of the Swarm platform for this ongoing work. This work was partially funded by NSF Grant #1840937.


  1. 1.
    Galton, F.: Vox populi. Nature 75, 450–451 (1907)CrossRefGoogle Scholar
  2. 2.
    Steyvers, M., Lee, M.D., Miller, B., Hemmer, P.: The wisdom of crowds in the recollection of order information. In: Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C.K.I. (2009)Google Scholar
  3. 3.
    Tetlock, P.E., Gardner, D.: Superforecasting: The Art and Science of Prediction. Crown Publishing Group, New York (2015)Google Scholar
  4. 4.
    Dana, J., Atanasov, P., Tetlock, P., Mellers, B.: Are markets more accurate than polls? The surprising informational value of “just asking”. Judgm. Decis. Making 14(2), 135–147 (2019)Google Scholar
  5. 5.
    Rosenberg, L.B.: Human swarms, a real-time method for collective intelligence. In: Proceedings of the European Conference on Artificial Life, pp. 658–659 (2015)Google Scholar
  6. 6.
    Rosenberg, L.: Artificial swarm intelligence vs human experts. In: Clerk Maxwell, J. (ed.) International Joint Conference on Neural Networks (IJCNN). IEEE (2016). A Treatise on Electricity and Magnetism, 3rd edn, vol. 2, pp. 68–73. Oxford, Clarendon, (1892)Google Scholar
  7. 7.
    Rosenberg, L., Baltaxe, D., Pescetelli, N.: Crowds vs swarms, a comparison of intelligence. In: IEEE 2016 Swarm/Human Blended Intelligence (SHBI), Cleveland, OH, pp. 1–4 (2016)Google Scholar
  8. 8.
    Baltaxe, D., Rosenberg, L., Pescetelli, N.: Amplifying prediction accuracy using human swarms. In: Collective Intelligence 2017, New York, NY (2017)Google Scholar
  9. 9.
    Willcox, G., Rosenberg, L., Askay, D., Metcalf, L., Harris, E., Domnauer, C.: Artificial swarming shown to amplify accuracy of group decisions in subjective judgment tasks. In: Arai, K., Bhatia, R. (eds.) Advances in Information and Communication. FICC 2019. Lecture Notes in Networks and Systems, vol. 70. Springer, Cham (2020)Google Scholar
  10. 10.
    Rosenberg, L., Pescetelli N., Willcox, G.: Artificial swarm intelligence amplifies accuracy when predicting financial markets. In: 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), New York City, NY, pp. 58–62 (2017)Google Scholar
  11. 11.
    Rosenberg, L., Willcox, G.: Artificial swarm intelligence vs vegas betting markets. In: 2018 11th International Conference on Developments in eSystems Engineering (DeSE), Cambridge, United Kingdom, pp. 36–39 (2018)Google Scholar
  12. 12.
    Rosenberg, L., Lungren, M., Halabi, S., Willcox, G., Baltaxe, D., Lyons, M.: Artificial swarm intelligence employed to amplify diagnostic accuracy in radiology. In: 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), Vancouver, BC, pp. 1186–1191 (2018)Google Scholar
  13. 13.
    Metcalf, L., Askay, D.A., Rosenberg, L.B.: Keeping humans in the loop: pooling knowledge through artificial swarm intelligence to improve business decision making. Calif. Manag. Rev. (2019). Scholar
  14. 14.
    Lee, R.M., Blank, G.: The SAGE Handbook of Online Research Methods, 2nd edn. SAGE Publications, Thousand Oaks (2017). Edited by Nigel G. FieldingGoogle Scholar
  15. 15.
    Bartlett, J.E., et. al.: Organizational research: determining appropriate sample size in survey research. Inf. Technol. Learn. Perform. J. 19, 43–50 (2001)Google Scholar
  16. 16.
    Schumann, H., Willcox, G., Rosenberg, L., Pescetelli, N.: Human swarming amplifies accuracy and ROI when forecasting financial markets. In: IEEE International Conference on Humanized Computing and Communication (HCC), Laguna Hills, CA, pp. 77–82 (2019)Google Scholar
  17. 17.
    Willcox, G., Askay, D., Rosenberg, L., Metcalf, L., Kwong, B., Liu, R.: Measuring group personality with swarm AI. In: IEEE International Conference on Transdisciplinary AI (TransAI), Laguna Hills, CA, pp. 10–17 (2019)Google Scholar
  18. 18.
    Willcox, G., Rosenberg, L.: Group sales forecasting, polls vs swarms. In: Future Technology Conference (FTC), San Francisco, CA (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Gregg Willcox
    • 1
    Email author
  • Louis Rosenberg
    • 1
  • Colin Domnauer
    • 2
  1. 1.Unanimous AISan FranciscoUSA
  2. 2.University of CaliforniaBerkeleyUSA

Personalised recommendations