Journal of General Internal Medicine

, Volume 29, Issue 1, pp 187–203 | Cite as

Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review

  • Benjamin L. Ranard
  • Yoonhee P. Ha
  • Zachary F. Meisel
  • David A. Asch
  • Shawndra S. Hill
  • Lance B. Becker
  • Anne K. Seymour
  • Raina M. Merchant
Review Article

Abstract

Objective

Crowdsourcing research allows investigators to engage thousands of people to provide either data or data analysis. However, prior work has not documented the use of crowdsourcing in health and medical research. We sought to systematically review the literature to describe the scope of crowdsourcing in health research and to create a taxonomy to characterize past uses of this methodology for health and medical research.

Data sources

PubMed, Embase, and CINAHL through March 2013.

Study eligibility criteria

Primary peer-reviewed literature that used crowdsourcing for health research.

Study appraisal and synthesis methods

Two authors independently screened studies and abstracted data, including demographics of the crowd engaged and approaches to crowdsourcing.

Results

Twenty-one health-related studies utilizing crowdsourcing met eligibility criteria. Four distinct types of crowdsourcing tasks were identified: problem solving, data processing, surveillance/monitoring, and surveying. These studies collectively engaged a crowd of >136,395 people, yet few studies reported demographics of the crowd. Only one (5 %) reported age, sex, and race statistics, and seven (33 %) reported at least one of these descriptors. Most reports included data on crowdsourcing logistics such as the length of crowdsourcing (n = 18, 86 %) and time to complete crowdsourcing task (n = 15, 71 %). All articles (n = 21, 100 %) reported employing some method for validating or improving the quality of data reported from the crowd.

Limitations

Gray literature not searched and only a sample of online survey articles included.

Conclusions and implications of key findings

Utilizing crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science. Standardized guidelines are needed on crowdsourcing metrics that should be collected and reported to provide clarity and comparability in methods.

KEY WORDS

crowdsourcing crowd sourcing citizen scientist citizen science human computing 

References

  1. 1.
    Sobel D. Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time. New York: Walker Publishing Company; 1995.Google Scholar
  2. 2.
    National Audubon Society. History of the Christmas Bird Count [Internet]. 2012 [cited 2012 Dec 24]. Available from: http://birds.audubon.org/history-christmas-bird-count
  3. 3.
    ICT Data and Statistics Division. Measuring the Information Society 2012 [Internet]. Geneva, Switzerland: International Telecommunication Union; 2012 [cited 2012 Oct 14]. Available from: http://www.itu.int/ITU-D/ict/publications/idi/material/2012/MIS2012_without_Annex_4.pdf
  4. 4.
    Lintott C, Schawinski K, Bamford S, Slosar A, Land K, Thomas D, et al. Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies. Monthly Notices of the Royal Astronomical Society. 2011;410(1):166–78. doi:10.1111/j.1365-2966.2010.17432.x.CrossRefGoogle Scholar
  5. 5.
    Sayigh L, Quick N, Hastie G, Tyack P. Repeated call types in short-finned pilot whales, Globicephala macrorhynchus. Marine Mammal Science. 2012;29(2):312–24. doi:10.1111/j.1748-7692.2012.00577.x.CrossRefGoogle Scholar
  6. 6.
    Nature Publishing Group. Scientific American launches Citizen Science Whale-Song Project, Whale FM [Internet]. 2011 [cited 2012 Oct 14]. Available from: http://www.nature.com/press_releases/sciam-whale.html
  7. 7.
    Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation. 2009;142(10):2282–92. doi:10.1016/j.biocon.2009.05.006.CrossRefGoogle Scholar
  8. 8.
    Marris E. Supercomputing for the birds. Nature. 2010;466(7308):807. doi:10.1038/466807a. PubMed PMID: 20703280.PubMedCrossRefGoogle Scholar
  9. 9.
    Preamble to the Constitution of the World Health Organization as adopted by the International Health Conference, New York, 19–22 June, 1946; signed on 22 July 1946 by the representatives of 61 States (Official Records of the World Health Organization, no. 2, p. 100) and entered into force on 7 April 1948. [cited 2012 Jul 21]. Available from: http://www.who.int/about/definition/en/print.html
  10. 10.
    Reips U-D, Birnbaum MH. Behavioral research and data collection via the Internet. In: Proctor RW, Vu K-PL, eds. Handbook of human factors in Web design. 2nd ed. Mahwah, New Jersey: Lawrence Erlbaum Associates; 2011:563–85.CrossRefGoogle Scholar
  11. 11.
    Higgins JPT, Deeks JJ (editors). Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Internet]. The Cochrane Collaboration; 2011 [updated 2011 Mar; cited 2012 Aug 1]. Available from www.cochrane-handbook.org
  12. 12.
    Cooper S, Treuille A, Barbero J, Leaver-Fay A, Tuite K, Khatib F, et al. The challenge of designing scientific discovery games. California: Proceedings of the Fifth International Conference on the Foundations of Digital Games; Monterey; 2010:40–7. 1822354: ACM.Google Scholar
  13. 13.
    Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466(7307):756–60. doi:10.1038/nature09304. PubMed PMID: 20686574; PubMed Central PMCID: PMC2956414.PubMedCentralPubMedCrossRefGoogle Scholar
  14. 14.
    Freifeld CC, Chunara R, Mekaru SR, Chan EH, Kass-Hout T, Ayala Iacucci A, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS medicine. 2010;7(12):e1000376. doi:10.1371/journal.pmed.1000376. PubMed PMID: 21151888; PubMed Central PMCID: PMC2998443.PubMedCentralPubMedCrossRefGoogle Scholar
  15. 15.
    Behrend TS, Sharek DJ, Meade AW, Wiebe EN. The viability of crowdsourcing for survey research. Behavior research methods. 2011;43(3):800–13. doi:10.3758/s13428-011-0081-0. PubMed PMID: 21437749.PubMedCrossRefGoogle Scholar
  16. 16.
    Bender J, O'Grady L, Desphande A, Cortinois A, Saffie L, Husereau D, et al. Collaborative authoring: a case study of the use of a wiki as a tool to keep systematic reviews up to date. 2011.Google Scholar
  17. 17.
    Cooper S, Khatib F, Makedon I, Lu H, Barbero J, Baker D, et al. Analysis of social gameplay macros in the Foldit cookbook. France: Proceedings of the 6th International Conference on Foundations of Digital Games; Bordeaux; 2011:9–14. 2159367: ACM.Google Scholar
  18. 18.
    Khatib F, DiMaio F, Foldit Contenders G, Foldit Void Crushers G, Cooper S, Kazmierczyk M, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology. 2011;18(10):1175–7. doi:10.1038/nsmb.2119. PubMed PMID: 21926992.CrossRefGoogle Scholar
  19. 19.
    Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America. 2011;108(47):18949–53. doi:10.1073/pnas.1115898108. PubMed PMID: 22065763; PubMed Central PMCID: PMC3223433.PubMedCentralPubMedCrossRefGoogle Scholar
  20. 20.
    Chunara R, Chhaya V, Bane S, Mekaru SR, Chan EH, Freifeld CC, et al. Online reporting for malaria surveillance using micro-monetary incentives, in urban India 2010–2011. Malaria journal. 2012;11:43. doi:10.1186/1475-2875-11-43. PubMed PMID: 22330227; PubMed Central PMCID: PMC3305483.PubMedCentralPubMedCrossRefGoogle Scholar
  21. 21.
    Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F, Shen BW, et al. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology. 2012;30(2):190–2. doi:10.1038/nbt.2109. PubMed PMID: 22267011.PubMedCentralPubMedCrossRefGoogle Scholar
  22. 22.
    Jarmolowicz DP, Bickel WK, Carter AE, Franck CT, Mueller ET. Using crowdsourcing to examine relations between delay and probability discounting. Behavioural processes. 2012;91(3):308–12. doi:10.1016/j.beproc.2012.09.001. PubMed PMID: 22982370.PubMedCrossRefGoogle Scholar
  23. 23.
    Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, Wu C, et al. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One. 2012;7(3):e31362. doi:10.1371/journal.pone.0031362. PubMed PMID: 22412834; PubMed Central PMCID: PMC3296692.PubMedCentralPubMedCrossRefGoogle Scholar
  24. 24.
    Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. Journal of medical Internet research. 2012;14(6):e167. doi:10.2196/jmir.2338. PubMed PMID: 23196001; PubMed Central PMCID: PMC3510720.PubMedCentralPubMedCrossRefGoogle Scholar
  25. 25.
    Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One. 2012;7(5):e37245. doi:10.1371/journal.pone.0037245. PubMed PMID: 22606353; PubMed Central PMCID: PMC3350488.PubMedCentralPubMedCrossRefGoogle Scholar
  26. 26.
    Mavandadi S, Dimitrov S, Feng S, Yu F, Yu R, Sikora U, et al. Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platforms. Lab on a chip. 2012;12(20):4102–6. doi:10.1039/c2lc40614d. PubMed PMID: 22918378; PubMed Central PMCID: PMC3477593.PubMedCrossRefGoogle Scholar
  27. 27.
    McKenna MT, Wang S, Nguyen TB, Burns JE, Petrick N, Summers RM. Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence. Medical image analysis. 2012;16(6):1280–92. doi:10.1016/j.media.2012.04.007. PubMed PMID: 22705287; PubMed Central PMCID: PMC3443285.PubMedCentralPubMedCrossRefGoogle Scholar
  28. 28.
    Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, et al. Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography. Radiology. 2012;262(3):824–33. doi:10.1148/radiol.11110938. PubMed PMID: 22274839; PubMed Central PMCID: PMC3285221.PubMedCrossRefGoogle Scholar
  29. 29.
    Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. Journal of medical Internet research. 2012;14(3):e79. doi:10.2196/jmir.2063. PubMed PMID: 22664384.PubMedCentralPubMedCrossRefGoogle Scholar
  30. 30.
    Crump MJ, McDonnell JV, Gureckis TM. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS One. 2013;8(3):e57410. doi:10.1371/journal.pone.0057410. PubMed PMID: 23516406; PubMed Central PMCID: PMC3596391.PubMedCentralPubMedCrossRefGoogle Scholar
  31. 31.
    Hipp JA, Adlakha D, Eyler AA, Chang B, Pless R. Emerging technologies: webcams and crowd-sourcing to identify active transportation. American journal of preventive medicine. 2013;44(1):96–7. doi:10.1016/j.amepre.2012.09.051. PubMed PMID: 23253658.PubMedCrossRefGoogle Scholar
  32. 32.
    Merchant RM, Asch DA, Hershey JC, Griffis HM, Hill S, Saynisch O, et al. A crowdsourcing innovation challenge to locate and map automated external defibrillators. Circulation Cardiovascular quality and outcomes. 2013;6(2):229–36. doi:10.1161/CIRCOUTCOMES.113.000140. PubMed PMID: 23481522.PubMedCrossRefGoogle Scholar
  33. 33.
    Carpenter J. May the best analyst win. Science. 2011;331(6018):698–9. doi:10.1126/science.331.6018.698. PubMed PMID: 21311005.PubMedCrossRefGoogle Scholar
  34. 34.
    Hussein A. Brokering knowledge in biosciences with InnoCentive. Interview by Semahat S. Demir. IEEE engineering in medicine and biology magazine: the quarterly magazine of the Engineering in Medicine & Biology Society. 2003;22(4):26–7. PubMed PMID: 14515687.CrossRefGoogle Scholar
  35. 35.
    Wheat RE, Wang Y, Byrnes JE, Ranganathan J. Raising money for scientific research through crowdfunding. Trends in ecology & evolution. 2013;28(2):71–2. doi:10.1016/j.tree.2012.11.001. PubMed PMID: 23219380.CrossRefGoogle Scholar
  36. 36.
    Mason W, Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods. 2012;44(1):1–23. doi:10.3758/s13428-011-0124-6. PubMed PMID: 21717266.PubMedCrossRefGoogle Scholar
  37. 37.
    Signorini A, Segre AM, Polgreen PM. The use of Twitter to track levels of disease activity and public concern in the US during the influenza A H1N1 pandemic. PLoS One. 2011;6(5):e19467.PubMedCentralPubMedCrossRefGoogle Scholar
  38. 38.
    Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health research ecosystem. Journal of medical Internet research. 2012;14(2):e46. doi:10.2196/jmir.1988. PubMed PMID: 22397809.PubMedCentralPubMedCrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine 2013

Authors and Affiliations

  • Benjamin L. Ranard
    • 1
  • Yoonhee P. Ha
    • 1
  • Zachary F. Meisel
    • 3
    • 4
  • David A. Asch
    • 2
    • 3
    • 5
    • 6
  • Shawndra S. Hill
    • 6
  • Lance B. Becker
    • 4
  • Anne K. Seymour
    • 7
  • Raina M. Merchant
    • 2
    • 3
    • 4
  1. 1.Perelman School of MedicineUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.Penn Medicine Center for InnovationUniversity of PennsylvaniaPhiladelphiaUSA
  3. 3.The Leonard Davis Institute of Health EconomicsUniversity of PennsylvaniaPhiladelphiaUSA
  4. 4.Department of Emergency MedicineUniversity of PennsylvaniaPhiladelphiaUSA
  5. 5.Philadelphia Veterans Affairs Medical CenterPhiladelphiaUSA
  6. 6.The Wharton SchoolUniversity of PennsylvaniaPhiladelphiaUSA
  7. 7.University of Pennsylvania LibrariesUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations