Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review
- 3.7k Downloads
Crowdsourcing research allows investigators to engage thousands of people to provide either data or data analysis. However, prior work has not documented the use of crowdsourcing in health and medical research. We sought to systematically review the literature to describe the scope of crowdsourcing in health research and to create a taxonomy to characterize past uses of this methodology for health and medical research.
PubMed, Embase, and CINAHL through March 2013.
Study eligibility criteria
Primary peer-reviewed literature that used crowdsourcing for health research.
Study appraisal and synthesis methods
Two authors independently screened studies and abstracted data, including demographics of the crowd engaged and approaches to crowdsourcing.
Twenty-one health-related studies utilizing crowdsourcing met eligibility criteria. Four distinct types of crowdsourcing tasks were identified: problem solving, data processing, surveillance/monitoring, and surveying. These studies collectively engaged a crowd of >136,395 people, yet few studies reported demographics of the crowd. Only one (5 %) reported age, sex, and race statistics, and seven (33 %) reported at least one of these descriptors. Most reports included data on crowdsourcing logistics such as the length of crowdsourcing (n = 18, 86 %) and time to complete crowdsourcing task (n = 15, 71 %). All articles (n = 21, 100 %) reported employing some method for validating or improving the quality of data reported from the crowd.
Gray literature not searched and only a sample of online survey articles included.
Conclusions and implications of key findings
Utilizing crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science. Standardized guidelines are needed on crowdsourcing metrics that should be collected and reported to provide clarity and comparability in methods.
KEY WORDScrowdsourcing crowd sourcing citizen scientist citizen science human computing
The authors thank Hope Lappen, MLIS, for her assistance with designing the systematic literature search.
NIH, K23 grant 10714038 (Merchant).
Conflict of Interest
The authors declare that they do not have any conflicts of interest.
- 1.Sobel D. Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time. New York: Walker Publishing Company; 1995.Google Scholar
- 2.National Audubon Society. History of the Christmas Bird Count [Internet]. 2012 [cited 2012 Dec 24]. Available from: http://birds.audubon.org/history-christmas-bird-count
- 3.ICT Data and Statistics Division. Measuring the Information Society 2012 [Internet]. Geneva, Switzerland: International Telecommunication Union; 2012 [cited 2012 Oct 14]. Available from: http://www.itu.int/ITU-D/ict/publications/idi/material/2012/MIS2012_without_Annex_4.pdf
- 6.Nature Publishing Group. Scientific American launches Citizen Science Whale-Song Project, Whale FM [Internet]. 2011 [cited 2012 Oct 14]. Available from: http://www.nature.com/press_releases/sciam-whale.html
- 9.Preamble to the Constitution of the World Health Organization as adopted by the International Health Conference, New York, 19–22 June, 1946; signed on 22 July 1946 by the representatives of 61 States (Official Records of the World Health Organization, no. 2, p. 100) and entered into force on 7 April 1948. [cited 2012 Jul 21]. Available from: http://www.who.int/about/definition/en/print.html
- 11.Higgins JPT, Deeks JJ (editors). Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Internet]. The Cochrane Collaboration; 2011 [updated 2011 Mar; cited 2012 Aug 1]. Available from www.cochrane-handbook.org
- 12.Cooper S, Treuille A, Barbero J, Leaver-Fay A, Tuite K, Khatib F, et al. The challenge of designing scientific discovery games. California: Proceedings of the Fifth International Conference on the Foundations of Digital Games; Monterey; 2010:40–7. 1822354: ACM.Google Scholar
- 14.Freifeld CC, Chunara R, Mekaru SR, Chan EH, Kass-Hout T, Ayala Iacucci A, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS medicine. 2010;7(12):e1000376. doi: 10.1371/journal.pmed.1000376. PubMed PMID: 21151888; PubMed Central PMCID: PMC2998443.PubMedCentralPubMedCrossRefGoogle Scholar
- 16.Bender J, O'Grady L, Desphande A, Cortinois A, Saffie L, Husereau D, et al. Collaborative authoring: a case study of the use of a wiki as a tool to keep systematic reviews up to date. 2011.Google Scholar
- 17.Cooper S, Khatib F, Makedon I, Lu H, Barbero J, Baker D, et al. Analysis of social gameplay macros in the Foldit cookbook. France: Proceedings of the 6th International Conference on Foundations of Digital Games; Bordeaux; 2011:9–14. 2159367: ACM.Google Scholar
- 18.Khatib F, DiMaio F, Foldit Contenders G, Foldit Void Crushers G, Cooper S, Kazmierczyk M, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology. 2011;18(10):1175–7. doi: 10.1038/nsmb.2119. PubMed PMID: 21926992.CrossRefGoogle Scholar
- 19.Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America. 2011;108(47):18949–53. doi: 10.1073/pnas.1115898108. PubMed PMID: 22065763; PubMed Central PMCID: PMC3223433.PubMedCentralPubMedCrossRefGoogle Scholar
- 20.Chunara R, Chhaya V, Bane S, Mekaru SR, Chan EH, Freifeld CC, et al. Online reporting for malaria surveillance using micro-monetary incentives, in urban India 2010–2011. Malaria journal. 2012;11:43. doi: 10.1186/1475-2875-11-43. PubMed PMID: 22330227; PubMed Central PMCID: PMC3305483.PubMedCentralPubMedCrossRefGoogle Scholar
- 24.Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. Journal of medical Internet research. 2012;14(6):e167. doi: 10.2196/jmir.2338. PubMed PMID: 23196001; PubMed Central PMCID: PMC3510720.PubMedCentralPubMedCrossRefGoogle Scholar
- 25.Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One. 2012;7(5):e37245. doi: 10.1371/journal.pone.0037245. PubMed PMID: 22606353; PubMed Central PMCID: PMC3350488.PubMedCentralPubMedCrossRefGoogle Scholar
- 26.Mavandadi S, Dimitrov S, Feng S, Yu F, Yu R, Sikora U, et al. Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platforms. Lab on a chip. 2012;12(20):4102–6. doi: 10.1039/c2lc40614d. PubMed PMID: 22918378; PubMed Central PMCID: PMC3477593.PubMedCrossRefGoogle Scholar
- 27.McKenna MT, Wang S, Nguyen TB, Burns JE, Petrick N, Summers RM. Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence. Medical image analysis. 2012;16(6):1280–92. doi: 10.1016/j.media.2012.04.007. PubMed PMID: 22705287; PubMed Central PMCID: PMC3443285.PubMedCentralPubMedCrossRefGoogle Scholar
- 28.Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, et al. Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography. Radiology. 2012;262(3):824–33. doi: 10.1148/radiol.11110938. PubMed PMID: 22274839; PubMed Central PMCID: PMC3285221.PubMedCrossRefGoogle Scholar
- 32.Merchant RM, Asch DA, Hershey JC, Griffis HM, Hill S, Saynisch O, et al. A crowdsourcing innovation challenge to locate and map automated external defibrillators. Circulation Cardiovascular quality and outcomes. 2013;6(2):229–36. doi: 10.1161/CIRCOUTCOMES.113.000140. PubMed PMID: 23481522.PubMedCrossRefGoogle Scholar