Crisis Crowdsourcing Framework: Designing Strategic Configurations of Crowdsourcing for the Emergency Management Domain


Crowdsourcing is not a new practice but it is a concept that has gained substantial attention during recent disasters. Drawing from previous work in the crisis informatics, disaster sociology, and computer-supported cooperative work (CSCW) literature, this paper first explains recent conceptualizations of crowdsourcing and how crowdsourcing is a way of leveraging disaster convergence. The CSCW concept of “articulation work” is introduced as an interpretive frame for extracting the salient dimensions of “crisis crowdsourcing.” Then, a series of vignettes are presented to illustrate the evolution of crisis crowdsourcing that spontaneously emerged after the 2010 Haiti earthquake and evolved to more established forms of public engagement during crises. The best practices extracted from the vignettes clarified the efforts to formalize crisis crowdsourcing through the development of innovative interfaces designed to support the articulation work needed to facilitate spontaneous volunteer efforts. Extracting these best practices led to the development of a conceptual framework that unpacks the key dimensions of crisis crowdsourcing. The Crisis Crowdsourcing Framework is a systematic, problem-driven approach to determining the why, who, what, when, where, and how aspects of a crowdsourcing system. The framework also draws attention to the social, technological, organizational, and policy (STOP) interfaces that need to be designed to manage the articulation work involved with reducing the complexity of coordinating across these key dimensions. An example of how to apply the framework to design a crowdsourcing system is offered with a discussion on the implications for applying this framework as well as the limitations of this framework. Innovation is occurring at the social, technological, organizational, and policy interfaces enabling crowdsourcing to be operationalized and integrated into official products and services.

This is a preview of subscription content, log in to check access.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5


  1. Baker, Karen S. and Florence Millerand (2007). Articulation Work Supporting Information Infrastructure Design: Coordination, Categorization, and Assessment in Practice. In Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS '07). IEEE Computer Society, Washington, DC, USA, 242a.

  2. Bannon, Liam J. and Kjeld Schmidt (1989). CSCW: Four characters in search of a context. ECSCW ’89. Proceedings of the First European Conference on Computer Supported Cooperative Work, Gatwick, London, 13–15 September, 1989, pp. 358–372. - Reprinted in Studies in Computer Supported Cooperative Work. Theory, Practice and Design, J. M. Bowers and S. D. Benford, Eds. North-Holland, Amsterdam etc., 1991, pp. 3–16.

  3. Barrington, Luke, Shubharoop Ghosh, Marjorie Greene, Shay Har-Noy, Jay Berger, Stuart Gill, Albert Yu-Min Lin, and Charles Huyck. (2011). Crowdsourcing Earthquake Damage Assessment Using Remote Sensing Imagery. Annals of Geophysics - Citizen Empowered Seismology, vol. 54, no. 6, pp. 680–687.

    Google Scholar 

  4. Bonney, Rick, Caren B. Cooper, Janis Dickinson, Steve Kelling, Tina Phillips, Kenneth V. Rosenberg, and Jennifer Shirk (2009). Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience, vol. 59, no. 11, pp. 977–984.

    Article  Google Scholar 

  5. Boulos, Maged N. Kamel, Bernd Resch, David N. Crowley, John G. Breslin, Gunho Sohn, Russ Burtner, William A. Pike, Eduardo Jezierski, and Kuo-Yu Slayer Chuang (2011). Crowdsourcing, citizen sensing and sensor web technologies for public and environmental health surveillance and crisis management: trends, OGC standards and application examples. International Journal of Health Geographics, vol. 10, no. 67, pp. 1–29.

    Google Scholar 

  6. Bowker, Geoffrey C. and Susan Leigh Star (1991). Situations vs. standards in long-term, widescale decision-making: The case of the International Classification of Diseases. In J. F. Nunamaker, Jr. and R. H. Sprague, Jr. (eds.): Proceedings of the Twenty-Fourth Annual Hawaii International Conference on System Sciences, Kauai, Hawaii, January 7-11, 1991. IEEE Computer Society Press, vol. 4, pp. 73–81.

  7. Bowker, Geoffrey C. and Susan Leigh Star (1999). Sorting Things Out. MIT Press, Boston.

    Google Scholar 

  8. Bowser, Anne and Lea Shanley (2013). New Visions in Citizen Science. Case study series v3. Woodrow Wilson International Center for Scholars: Washington DC.

  9. Brabham, Daren C. (2008). Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. Convergence: The International Journal of Research into New Media Technologies, vol. 14, no. 1, pp. 75–90.

  10. Brabham, Daren C. (2012). Crowdsourcing: A Model for Leveraging Online Communities. In: A. Delwiche and J. Henderson (eds.), The Routledge Handbook of Participatory Culture. New York: Taylor & Francis Inc.

    Google Scholar 

  11. Capelo, Luis, Natalie Chang, and Andrej Verity (2012). Guidance for Collaborating with Volunteer & Technical Communities. Digital Humanitarian Network. Accessed 13 March 2013.

  12. Clarke, Lee (2002). Panic: Myth or Reality? Contexts, vol. 1, no. 3, pp. 21–26.

    Article  Google Scholar 

  13. Cobb, Camille, Ted McCarthy, Annuska Perkins, Ankitha Bharadwaj, Jared Comis, Brian Do, and Kate Starbird (2014). Designing for the Deluge: Understanding & Supporting the Distributed, Collaborative Work of Crisis Volunteers. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14). ACM, New York, NY, USA, 888–899.

  14. Cohn, Jeffrey P. (2008). Citizen science: Can volunteers do real research? BioScience, vol. 58, no. 3, pp. 192–197.

    Article  Google Scholar 

  15. Divitini, Monica and Tuomo Tuikka (1996). Steps for Developing Coordination Support. In Dahlbom, B., Ljungberg, F., Nuldén, U., Simon, K., Stage, J., Sørensen, C. (Eds.), Proceedings of IRIS 19. pp. 823–841. Gothenburg Studies of Informatics.

  16. Doran, Kara S., Hilary F. Stockdon, Kristin L. Sopkin, David M. Thompson, and Nathaniel G. Plant (2012). National assessment of hurricane-induced coastal erosion hazards: Mid-Atlantic Coast. U.S. Geological Survey Open-File Report 2013–1131,

  17. Dynes, Russell Rowe (1970). Organized Behavior in Disaster. Lexington, MA: Heath Lexington Books, D.C. Health and Company.

    Google Scholar 

  18. Earle, Paul S. and Peter M. Shearer (1994). Characterization of global seismograms using an automatic-picking algorithm. Bulletin of the Seismological Society of America, vol. 84, pp. 366–376.

  19. Earle, Paul S., Daniel C. Bowden, and Michelle Guy (2011). Twitter Earthquake Detection: Earthquake Monitoring in a Social World. Annals of Geophysics Special Section on Citizen Empowered Seismology, vol. 54, no. 6, pp. 708–715.

    Google Scholar 

  20. Earle, Paul S., Michelle Guy, Richard Buckmaster, Chris Ostrum, Scott Horvath, and Amy Vaughan (2010). OMG Earthquake! Can Twitter Improve Earthquake Response? Seismological Research Letters, vol. 81, no. 2, pp. 246–251.

    Article  Google Scholar 

  21. Elwood, Sarah (2008). Volunteered geographic information: Key questions, concepts and methods to guide emerging research and practice. GeoJournal, vol. 72, no. 3–4, pp. 133–135.

  22. Estellés-Arolas, Enrique and Fernando González-Ladrón-De-Guevara (2012). Towards an integrated crowdsourcing definition. Journal Information Science, vol. 38, no. 2, pp. 189–200.

  23. Fischer III, Henry W. (1998). Response to Disaster: Fact Versus Fiction & Perpetuation, 2nd ed. New York: University Press of America.

    Google Scholar 

  24. Fitzpatrick, Geraldine, William J. Tolone, and Simon M. Kaplan (1995). Work, locales and distributed social worlds. In Proceedings of the fourth conference on European Conference on Computer-Supported Cooperative Work (ECSCW'95), Hans Marmolin, Yngve Sundblad, and Kjeld Schmidt (Eds.). Kluwer Academic Publishers, Norwell, MA, pp. 1–16.

    Google Scholar 

  25. Fjuk, Annita, Markku I. Nuriminen, and Ole Smørdal (1997). Taking Articulation Work Seriously — An Activity Theoretical Approach. Turku Centre for Computer Science, TUCS Technical Report No 120.

  26. Fritz, Charles E. and John H. Mathewson (1957). Convergence Behavior in Disasters: A Problem in Social Control, Committee on Disaster Studies. Washington, DC: National Academy of Sciences, National Research Council.

    Google Scholar 

  27. Fujimura, Joan H. (1987). Constructing ‘Do-Able’ Problems in Cancer Research: Articulating Alignment. Social Studies of Science, vol. 17, no. 2, pp. 257–293.

    Article  Google Scholar 

  28. Gao, Huiji, Geoffrey Barbier, and Rebecca Goolsby (2011). Harnessing the Crowdsourcing Power of Social Media for Disaster Relief. IEEE Intelligent Systems, vol. 26, no. 3, pp. 10–14.

    Article  Google Scholar 

  29. Gerson, Elihu M. and Susan Leigh Star (1986). Analyzing due process in the workplace. ACM Transactions on Information Systems (TOIS), vol. 4, no. 3, pp. 257–270.

    Article  Google Scholar 

  30. Goodchild, Michael F. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, vol. 69, no. 4, pp. 211–221.

    Article  Google Scholar 

  31. Goodchild, Michael F. and J. Alan Glennon (2010). Crowdsourcing Geographic Information for Disaster Response—A Research Frontier. International Journal of Digital Earth, vol. 3, no. 3, pp. 231–241.

    Article  Google Scholar 

  32. Grinter, Rebecca E. (1996). Supporting Articulation Work Using Software Configuration Management Systems. Computer Supported Cooperative Work: The Journal of Collaborative Computing, vol. 5, pp. 447–465.

    Article  Google Scholar 

  33. Guy, Michelle, Paul Earle, Chris Ostrum, Kenny Gruchalla, and Scott Horvath (2010). Integration and Dissemination of Citizen Reported and Seismically Derived Earthquake Information via Social Network Technologies. In Proceedings of the Advances in Intelligent Data Analysis IX, vol. 6065, pp. 42–53.

  34. Hagar, Christine and C. Haythornthwaite (2005). Crisis, Farming & Community. The Journal of Community Informatics, vol. 1, no. 3, pp. 41–52.

    Google Scholar 

  35. Haklay, Mordechai (2010). How good is volunteered geographical information? A comparative study of OpenStreetMap and Ordnance survey datasets. Environment and Planning B: Planning and Design, vol. 37, pp. 682–703.

    Article  Google Scholar 

  36. Mordechai (Muki) Haklay, Alex Singleton, and Chris Parker (2008). Web Mapping 2.0—The Neogeography of the GeoWeb. Geography Compass, vol. 2, no. 6, pp. 2011–2039.

  37. Harvard Humanitarian Initiative. (2011). Disaster Relief 2.0: The Future of Information Sharing in Humanitarian Emergencies. Washington, D.C. and Berkshire, UK: UN Foundation & Vodafone Foundation Technology Partnership.

  38. Heath, Christian and Paul Luff (1992). Collaboration and Control: Crisis Management and Multimedia Technology in London Underground Line Control Rooms. Computer Supported Cooperative Work (CSCW), vol. 1, pp. 69–94.

    Article  Google Scholar 

  39. Heinzelman, Jessica and Carol Waters. (2010). Crowdsourcing Crisis Information in Disaster-Affected Haiti. Special Report. Washington, D.C.: United States Institute of Peace.

    Google Scholar 

  40. Heipke, Christian (2010). Crowdsourcing Geospatial Data. ISPRS Journal of Photogrammetry and Remote Sensing, vol. 65, no. 6, pp. 550–557.

    Article  Google Scholar 

  41. Hester, Vaughn, Aaron Shaw, and Lukas Biewald (2010). Scalable crisis relief: Crowdsourced SMS translation and categorization with Mission 4636. In Proceedings of the First ACM Symposium on Computing for Development (ACM DEV '10). ACM, New York, NY, USA, Article 15, 7 pages.

  42. Hines, Megan, Abigail Benson, David Govoni, Derek Masaki, Barbara Poore, Annie Simpson, and Steven Tessler (2013). Partnering for Science; Proceedings of the USGS Workshop on Citizen Science. U.S. Geological Survey Open-File Report 2013–1234,

  43. Howe, Jeff (2006). The Rise of Crowdsourcing. Wired Magazine, vol. 14, pp. 1–4.

    Google Scholar 

  44. Howe, Jeff (2008). Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. New York: Crown Business.

    Google Scholar 

  45. Hughes, Amanda and Leysia Palen, (2009). Twitter Adoption and Use in Mass Convergence and Emergency Events. Proceedings of the 2009 Information Systems for Crisis Response and Management Conference, (ISCRAM 2009).

  46. Hughes, Amanda, Leysia Palen, Jeannette Sutton, Sophia B. Liu, and Sarah Vieweg (2008). “Site-Seeing” in Disaster: An Examination of On-Line Social Convergence. Proceedings of the Information Systems for Crisis Response and Management Conference (ISCRAM 2008).

  47. Jackson, Steven P., William Mullen, Peggy Agouris, Andrew Crooks, Arie Croitoru, and Anthony Stefanidis (2013). Assessing Completeness and Spatial Error of Features in Volunteered Geographic Information. ISPRS International Journal of Geo-Information, vol. 2, no. 2, pp. 507–530.

    Article  Google Scholar 

  48. Kendra, James M. and Tricia Wachtendorf (2003). Reconsidering Convergence and Converger: Legitimacy in Response to the World Trade Center Disaster. Research in Social Problems and Public Policy, vol. 11, pp. 97–122.

  49. Kreps, Gary A. and Susan Lovegren Bosworth. (1994). Organizing, Role Enactment, and Disaster: A Structural Theory. Cranbury, NJ: Associated University Presses.

    Google Scholar 

  50. Lewis, Clayton H. (1982). Using the “Thinking Aloud” Method In Cognitive Interface Design. IBM Technical Research Report RC 9265, Yorktown Heights, NY.

    Google Scholar 

  51. Little, Greg, Lydia B. Chilton, Max Goldman, and Robert C. Miller (2010). Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, pp. 68–76.

  52. Liu, Sophia B. (2010). Trends in Distributed Curatorial Technology to Manage Data in a Networked World. UPGRADE Journal Emerging Information Technologies, vol. XI, no. 3, pp. 18–24.

  53. Liu, Sophia B. (2011). Grassroots Heritage: A Multi-Method Investigation of How Social Media Sustain the Living Heritage of Historic Crises. PhD dissertation. University of Colorado at Boulder, Colorado: Alliance for Technology, Learning, and Society (ATLAS) Institute.

  54. Liu, Sophia B. (2012). “Socially Distributed Curation of the Bhopal Disaster: A Case of Grassroots Heritage in the Crisis Context.” In Elisa Giaccardi (Ed.), Heritage and Social Media: Understanding and Experiencing Heritage in a Participatory Culture, Cambridge, UK: Routledge.

    Google Scholar 

  55. Liu, Sophia B. and Jen Ziemke. (2012). From Cultures of Participation to the Rise of Crisis Mapping in a Networked World. In Aaron Delwiche and Jennifer Henderson (eds), The Routledge Handbook of Participatory Cultures, New York: Routledge, pp. 185–196.

    Google Scholar 

  56. Liu, Sophia B. and Leysia Palen. (2010). The New Cartographers: Crisis Map Mashups and the Emergence of Neogeographic Practice. Cartography and Geographic Information Science (CaGIS) Journal, 7(1): 69–90.

    Article  Google Scholar 

  57. Liu, Sophia B., Barbara S. Poore, Richard J. Snell, Aubrey Goodman, Nathaniel G. Plant, Hilary Stockdon, Karen L. Morgan, and M. Dennis Krohn (2014). USGS iCoast – Did the Coast Change? Designing a Crisis Crowdsourcing App to Validate Coastal Change Models. In Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW Companion '14). ACM, New York, NY, USA, pp. 17–20.

  58. Liu, Sophia B., Beau Bouchard, Daniel Bowden, Michelle Guy, and Paul Earle (2012). USGS Tweet Earthquake Dispatch (@USGSted): Using Twitter for Earthquake Detection and Characterization. Poster presented at the American Geophysical Union (AGU) 2012 Annual Meeting for the “Citizen Empowered Seismology” paper session in San Francisco, CA.

  59. Maisonneuve, Nicolas and Bastien Chopard (2012). Crowdsourcing satellite imagery analysis: study of parallel and iterative models. Geographic Information Science, Lecture Notes in Computer Science, vol. 7478, pp. 116–131.

  60. Mark, Gloria, Mossaab Bagdouri, Leysia Palen, James Martin, Ban Al-Ani, and Kenneth Anderson (2012). Blogs as a collective war diary. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, pp. 37–46.

  61. Martineau, Eric (2012). A Typology of Crowdsourcing Participation Styles. Masters thesis. Concordia University.

  62. Meier, Patrick (2011). Verifying Crowdsourced Social Media Reports for Live Crisis Mapping: An Introduction to Information Forensics. iRevolution blog. Accessed on December 29, 2011.

  63. Meier, Patrick and Robert Munro (2010). The Unprecedented Role of SMS in Disaster Response: Learning from Haiti. SAIS Review of International Affairs, vol. 30, no. 2, pp. 91–13.

    Google Scholar 

  64. Mendonça, David (2007). Decision support for improvisation in response to extreme events: Learning from the response to the 2011 World Trade Center attack. Decision Support Systems, vol. 43, no. 3, pp. 952–967.

    Article  Google Scholar 

  65. Munro, Robert (2012). Crowdsourcing and the Crisis-affected Community: Lessons Learned and Looking Forward from Mission 4636. Journal of Information Retrieval.

  66. Munro, Robert, Tyler Schnoebelen, Schuyler Erle (2013). Quality Analysis After Action Report for the Crowdsourced Aerial Imagery Assessment Following Hurricane Sandy. Proceedings of the 10th International ISCRAM Conference. Available:

  67. Newman, Greg, Andrea Wiggins, Alycia Crall, Eric Graham, Sarah Newman, and Kevin Crowston (2012). The Future of Citizen Science: Emerging Technologies and Shifting Paradigms. Frontiers in Ecology and the Environment, vol. 19, no. 6, pp. 298–304.

    Article  Google Scholar 

  68. Newman, Greg, Jim Graham, Alycia Crall, and Melinda Laituri (2011). The Art and Science of Multi-Scale Citizen Science Support. Ecological Informatics, vol. 6, no. 3–4, pp. 217–227.

    Article  Google Scholar 

  69. Nourbakhsh, Illah, Randy Sargent, Anne Wright, Kathryn Cramer, Brian McClendon, and Michael Jones (2006). Mapping disaster zones. Nature, vol. 439, pp. 787–788.

    Article  Google Scholar 

  70. Palen, Leysia and Sophia B. Liu (2007). Citizen Communications in Crisis: Anticipating a Future of ICT-Supported Participation. Proceedings of the 2007 ACM Conference on Human Factors in Computing System (CHI 2007). ACM, New York, NY, pp. 727–736.

    Google Scholar 

  71. Palen, Leysia, Kenneth M. Anderson, Gloria Mark, James Martin, Douglas Sicker, Martha Palmer, and Dirk Grunwald (2010). A vision for technology-mediated support for public participation and assistance in mass emergencies and disasters. In Proceedings of the 2010 ACM-BCS Visions of Computer Science Conference (ACM-BCS '10). British Computer Society, Swinton, UK, Article 8, 12 pages.

  72. Palen, Leysia, Sarah Vieweg, Sophia B. Liu, and Amanda Hughes (2009). Crisis in a Networked World: Features of Computer- Mediated Communication in the April 16, 2007 Virginia Tech Event. Social Science Computing Review, Sage, vol. 27, no. 4, pp. 467–480.

  73. Plant, Nathaniel G. and Hilary F. Stockdon (2012). Probabilistic prediction of barrier-island response to hurricanes. Journal of Geophysical Research, vol. 117, no. F3, pp. 2156–2202.

    Article  Google Scholar 

  74. Qu, Yan, Chen Huang, Pengyi Zhang, and Jun Zhang (2011). Microblogging after a major disaster in China: A case study of the 2010 Yushu earthquake. In Proceedings of the ACM 2011 conference on Computer supported cooperative work (CSCW '11). ACM, New York, NY, USA, pp. 25–34.

  75. Quarantelli, E.L. and Russell R. Dynes (1977). Response to Social Crisis and Disaster. Annual Review of Sociology, vol. 3, pp. 23–49.

    Article  Google Scholar 

  76. Roberts, Shadrock, Stephanie Grosser, and D. Ben Swartley (2012). Crowdsourcing to Geocode Development Credit Authority Data: A Case Study. Washington, D.C.: U.S. Agency for International Development. fs/2012/USAIDCrowdsourcingCaseStudy.pdf. Accessed 13 March 2013.

  77. Roche, Stephane, Eliane Propeck-Zimmermann, and Boris Mericskay (2011). GeoWeb and Crisis Management: Issues and Perspectives of Volunteered Geographic Information. GeoJournal, Netherlands: Springer, vol. 78, no. 1, pp. 21–40.

  78. Rouse, Anne C. (2010). A Preliminary Taxonomy of Crowdsourcing. In Proceedings of the 2010 Australasian Conferences on Information Systems (ACIS) on World Wide Web Internet And Web Information Systems.

  79. Sarcevic, Aleksandra, Leysia Palen, Joanne White, Kate Starbird, Mossaab Bagdouri, and Kenneth Anderson (2012). “Beacons of Hope” in Decentralized Coordination: Learning from On-the-Ground Medical Twitterers During the 2010 Haiti Earthquake. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, pp. 47–56.

  80. Schenk, Eric and Claude Guittard (2011). Towards a Characterization of Crowdsourcing Practices. Journal of Innovation Economics, vol. 7, no. 1, pp. 93–107.

    Article  Google Scholar 

  81. Schmidt, Kjeld (1993). Cooperative Work and its Articulation: Requirement for Computer Support. Le Travail Humain, Presses Universitaires de France, vol. 57, no. 4, pp. 345–366

  82. Schmidt, Kjeld (1994). Modes and Meachanisms of Interaction in Cooperative Work: Outline of a Conceptual Framework. Risø National Laboratory, Roskilde, Denmark.

  83. Schmidt, Kjeld and Liam Bannon (1992). Taking CSCW seriously: Supporting articulation work. Computer Supported Cooperative Work, vol. 1, no. 1–2, pp. 7–40.

  84. Sharma, Ankit (2010). Crowdsourcing Critical Success Factor Model: Strategies to harness the collective intelligence of the crowd. Working paper. Accessed on 3 July 2010.

  85. Shirk, Jennifer L., Heidi L. Ballard, Candie C. Wilderman, Tina Phillips, Andrea Wiggins, Rebecca Jordan, Ellen McCallie, Matthew Minarchek, Bruce V. Lewenstein, Marianne E. Krasny and Rick Bonney (2012). Public participation in scientific research: A framework for deliberate design. Ecology and Society, vol. 17, no. 2, pp. 29.

  86. Simone, Carla, Monica Divitini, and Kjeld Schmidt (1995). A notation for malleable and interoperable coordination mechanisms for CSCW systems. In Proceedings of conference on Organizational computing systems (COCS '95), Nora Comstock and Clarence Ellis (eds), ACM, New York, NY, USA, pp. 44–54.

    Google Scholar 

  87. Sletholt, Magnus Thorstein, Jo Hannay, Dietmar Pfahl, Hans Christian Benestad, and Hans Petter Langtangen (2011). A literature review of agile practices and their effects in scientific software development. In Proceedings of the 4th International Workshop on Software Engineering for Computational Science and Engineering (SECSE '11). ACM, New York, NY, USA, pp. 1–9.

  88. Snow, Rion, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng (2008). Cheap and Fast--But Is It Good?: Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08). Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 254–263.

  89. Stallings, Robert A. and E.L. Quarantelli (1985). Emergent Citizen Groups and Emergency Management. Public Administration Review, vol. 45, pp. 93–100.

    Article  Google Scholar 

  90. Star, Susan Leigh and James R. Griesemer (1989). Institutional ecology, ‘translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, vol. 19, no. 3, pp. 387–420.

    Article  Google Scholar 

  91. Starbird, Kate (2012). Crowdwork, Crisis and Convergence: How the Connected Crowd Organizes Information During Mass Disruption Events. PhD dissertation. University of Colorado at Boulder, Colorado: Alliance for Technology, Learning, and Society (ATLAS) Institute.

  92. Starbird, Kate (2013). Delivering Patients to Sacré Coeur: Collective Intelligence in Digital Volunteer Communities. In Proceedings of the ACM 2012 Conference on Human Factors in Computing Systems (CHI 2013), ACM, New York, NY, USA, pp. 801–810.

    Google Scholar 

  93. Starbird, Kate and Leysia Palen (2013). Working and Sustaining the Virtual Disaster Desk. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, ACM, New York, NY, USA, pp. 491–502.

  94. Starbird, Kate and Leysia Palen. (2011). “Voluntweeters”: Self-Organizing by Digital Volunteers in Times of Crisis. In Proceedings of the 2011 ACM Human Factors in Computing Systems (CHI 2011), ACM, New York, NY, USA, pp. 1071–1080.

    Google Scholar 

  95. Starbird, Kate, Leysia Palen, Amanda Hughes, and Sarah Vieweg (2010). Chatter on The Red: What Hazards Threat Reveals about the Social Life of Microblogged Information. In Proceedings of the ACM 2010 Conference on Computer Supported Cooperative Work (CSCW 2010), ACM, New York, NY, USA, pp. 241–250.

    Google Scholar 

  96. Steven J. Jackson, Tarleton Gillespie, and Sandy Payette (2014). The policy knot: Re-integrating policy, practice and design in CSCW studies of social computing. Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14). ACM, New York, NY, USA, pp. 588–602.

  97. Stockdon, Hilary F., Kara J. Doran, David M. Thompson, Kristin L. Sopkin, and Nathaniel G. Plant (2013). National assessment of hurricane-induced coastal erosion hazards: Southeast Atlantic Coast. U.S. Geological Survey Open–File Report 2013-1130.

  98. Stockdon, Hilary F., Kara J. Doran, David M. Thompson, Kristin L. Sopkin, Nathaniel G. Plant, and Asbury H. Sallenger (2012). National assessment of hurricane-induced coastal erosion hazards—Gulf of Mexico. U.S. Geological Survey Open-File Report 2012–1084.

  99. Strauss, Anselm (1985). Work and the division of labor. The Sociological Quarterly, JAI Press, Inc., vol. 26, no. 1, pp. 1–19.

  100. Strauss, Anselm (1988). The Articulation of Project Work: An Organizational Process. The Sociological Quarterly, vol. 29, no. 2, pp. 163–178.

    Article  Google Scholar 

  101. Strauss, Anselm (1993). Continual Permutations of Actions. Aldine de Gruyter, New York.

    Google Scholar 

  102. Suchman, Lucy (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press.

    Google Scholar 

  103. Teraguchi, Masayoshi, Shin Saito, Tessa Lau, Masaki Ohno, Julian A Cerruti, and Hironobu Takagi (2012). Crowdsourcing in Crisis Informatics for Disaster Relief. In Proceedings of the Computer-Supported Cooperative Workshop on Collaboration and Crisis Informatics. Available in the International Reports on Socio-Informatics, vol. 9, no. 2, pp. 61–66.

  104. Tierney, Kathleen J., Michael K. Lindell, and Ronald W. Perry (2001). Facing the Unexpected: Disaster Preparedness and Response in the United States. Washington, DC: Joseph Henry Press/National Academy Press.

    Google Scholar 

  105. Trumbull, Deborah J., Rick Bonney, Derek Bascom, and Anna Cabral (2000). Thinking scientifically during participation in a citizen-science project. Science Education, vol. 84, no. 2, pp. 265–275.

    Article  Google Scholar 

  106. Turner, Andrew (2006). An Introduction to Neogeography. Sebastapol, CA: O’Reilly Media.

    Google Scholar 

  107. Van der Windt, Peter (2012). From Crowdsourcing to Crowdseeding: The Cutting Edge of Empowerment? Accessed 13 January 2014.

  108. Vander Wal, Thomas (2004). Folksonomy. Accessed 15 January 2014.

  109. Vieweg, Sarah, Amanda L. Hughes, Kate Starbird, and Leysia Palen (2010). Microblogging during two natural hazards events: what twitter may contribute to situational awareness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), ACM, New York, NY, USA, pp. 1079–1088.

  110. Vivacqua, Adriana S. and Marcos R.S. Borges (2012). Taking Advantage of Collective Knowledge in Emergency Response Systems. Journal of Network and Computer Applications, vol. 35, no. 1, pp. 189–198.

    Article  Google Scholar 

  111. Wachtendorf, Tricia and James M. Kendra (2005). Improvising Disaster in the City of Jazz: Organizational Response to Hurricane Katrina. Understanding Katrina: Perspectives from the Social Sciences. Accessed 29 February 2011.

  112. Wald, David J., Paul S. Earle, Kuo-Wan Lin, Vincent Quitoriano, and Bruce Worden (2006). Challenges in Rapid Ground Motion Estimation for the Prompt Assessment of Global Urban Earthquakes. Bulletin of the Earthquake Research Institute, vol. 81, pp. 273–281.

    Google Scholar 

  113. Wald, David J., Vincent Quitoriano, Bruce Worden, Margaret Hopper, and James W. Dewey (2011). USGS “Did You Feel It?” Internet-based Macroseismic Intensity Maps. Annals of Geophysics Special Section on Citizen Empowered Seismology, vol. 54, no. 6, pp. 688–707.

    Google Scholar 

  114. Wald, David J., Vincent Quitoriano, Lori A. Dengler, and James W. Dewey. (1999a). Utilization of the Internet for rapid community intensity maps. Seismological Research Letters vol. 70, no. 6, pp. 680–697.

    Article  Google Scholar 

  115. Wald, David J., Vincent Quitoriano, Thomas H. Heaton, Hiroo Kanamori, Craig W. Scrivner, and C. Bruce Worden (1999b). TriNet “ShakeMaps” – Rapid generation of peak groundmotion and intensity maps for earthquakes in southern California. Earthquake Spectra, vol. 15, no. 3, pp. 537–555.

    Article  Google Scholar 

  116. Warren, Jeffrey Yoo (2010). Grassroots mapping: Tools for participatory and activist cartography. PhD dissertation. Massachusetts Institute of Technology.

  117. Yates, Dave and Scott Paquette (2011). Emergency Knowledge Management and Social Media Technologies: A Case Study of the 2010 Haitian Earthquake. International Journal of Information Management, vol. 31, no. 1, pp. 6–13.

    Article  Google Scholar 

  118. Young, Jason C., David J. Wald, Paul S. Earle, and Lea A. Shanley (2013). Transforming Earthquake Detection and Science Through Citizen Seismology. Washington, D.C.: Woodrow Wilson International Center for Scholars.

    Google Scholar 

  119. Ziemke, Jen (2012). Crisis Mapping: The Construction of a New Interdisciplinary Field? Journal of Map & Geography Libraries, vol. 8, no. 2, pp. 101–117

    Article  Google Scholar 

  120. Zook, Matthew, Mark Graham, Taylor Shelton, and Sean Gorman (2010). Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake. World Medical & Health Policy, vol. 2, no. 2, pp. 7–33.

    Article  Google Scholar 

Download references


The research reported here is supported through a U.S. Geological Survey Mendenhall Postdoctoral Fellowship awarded to Sophia B. Liu. The author would like to thank Paul Earle and Barbara Poore for their early involvement and guidance on this manuscript, the research participants for their insights and feedback, Betsy Boynton for her contributions to the Crisis Crowdsourcing Framework graphic, and the anonymous reviewers for helping to improve the structure and concepts of the paper. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.

Author information



Corresponding author

Correspondence to Sophia B. Liu.

Additional information

Liu et al. (2014) Crisis Crowdsourcing Framework: Designing Strategic Configurations of Crowdsourcing for the Emergency Management Domain. Computer-Supported Cooperative Work (CSCW) special issue on Crisis Informatics and Collaboration.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, S.B. Crisis Crowdsourcing Framework: Designing Strategic Configurations of Crowdsourcing for the Emergency Management Domain. Comput Supported Coop Work 23, 389–443 (2014).

Download citation


  • Articulation work
  • Conceptual framework
  • Crisis informatics
  • Crisis mapping
  • Crowdsourcing
  • Crowdwork
  • Digital volunteers
  • Disasters
  • Emergency management
  • Human computation
  • Information management
  • Social media