Information Systems Frontiers

, Volume 16, Issue 3, pp 417–434 | Cite as

Evaluation on crowdsourcing research: Current status and future direction

  • Yuxiang ZhaoEmail author
  • Qinghua Zhu


Crowdsourcing is one of the emerging Web 2.0 based phenomenon and has attracted great attention from both practitioners and scholars over the years. It can facilitate the connectivity and collaboration of people, organizations, and societies. We believe that Information Systems scholars are in a unique position to make significant contributions to this emerging research area and consider it as a new research frontier. However, so far, few studies have elaborated what have been achieved and what should be done. This paper seeks to present a critical examination of the substrate of crowdsourcing research by surveying the landscape of existing studies, including theoretical foundations, research methods, and research foci, and identifies several important research directions for IS scholars from three perspectives—the participant, organization, and system—and which warrant further study. This research contributes to the IS literature and provides insights for researchers, designers, policy-makers, and managers to better understand various issues in crowdsourcing systems and projects.


Crowdsourcing Web 2.0 Socio-technical systems Collective intelligence Mass collaboration Problem-solving Research progress 



This work is jointly supported by the National Social Science Foundation in China (Grand No.10ATQ004), and the Ministry of Education, Humanities and Social Sciences Council in China (Grand No. 09YJA870014). Firstly, the authors sincerely appreciate two anonymous reviewers’ thoughtful comments and suggestions. Secondly, the authors would like to thank Prof. Ping Zhang from Syracuse University for her insightful and constructive suggestions on this paper. Thirdly, the authors thank Xu Meng from Syracuse University for his contribution during the coding stage. Finally, the authors thank the editors and publishers for their support to this article.


  1. Agarwal, R., & Henry Lucas, C. J. (2005). The information systems identify crisis: Focusing on high-visibility and high-impact research. MIS Quarterly, 29(3), 381–398.Google Scholar
  2. Alavi, M., & Carlson, P. (1992). A review of MIS research and disciplinary development. Journal of Management Information Systems, 8(4), 45–62.Google Scholar
  3. Albors, J., Ramos, J. C., & Hervas, J. L. (2008). New learning network paradigms: communities of objectives, crowdsourcing, wikis and open source. International Journal of Information Management, 28(3), 194–202.CrossRefGoogle Scholar
  4. Alonso, O., Rose, D., & Stewart, B. (2008). Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2), 9–15.CrossRefGoogle Scholar
  5. Ambati, V., Vogel, S., & Carbonell, J. (2010). Active learning and crowd-sourcing for machine translation. Proceedings of the LREC, 2010, 2169–2174.Google Scholar
  6. Archak, N. (2010). Money, glory and cheap talk: Analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on in Proceedings of the 19th international conference on World Wide Web ACM, pp. 21–30.Google Scholar
  7. Archak, N., & Sundararajan, A. (2009). “Optimal Design of Crowdsourcing Contest,” in Proceedings Thirtieth International Conference on Information Systems (ICIS 2009), Phoenix, USA.Google Scholar
  8. Ba, S., Stallaert, J., & Whinston, A. B. (2001). Introducing a third dimension in decentralized information systems design: the case for incentive alignment. Information Systems Research, 12(3), 225–239.CrossRefGoogle Scholar
  9. Bagozzi, R. P., & Dholakia, U. M. (2006). Open source software user communities: a study of participation in Linux user groups. Management Science, 52(7), 1099–1115.Google Scholar
  10. Bayus, B. L. (2010). Crowdsourcing and individual creativity over time: the detrimental effects of past success. SSRN Quantitative Marketing eJournal, Retrieved from
  11. Benbasat, I., & Zmud, R. W. (2003). The identify crisis within the IS discipline: defining and communicating the discipline’s core properties. MIS Quarterly, 27(2), 183–194.Google Scholar
  12. Bigliardi, B., Dormio, A. I., & Galati, F. (2012). The adoption of open innovation within the telecommunication industry. European Journal of Innovation Management, 15(1), 27–54.CrossRefGoogle Scholar
  13. Bonabeau, E. (2009). Decisions 2.0: the power of collective intelligence. MIT Sloan Management Review, 50(2), 45–52.Google Scholar
  14. Boudreau, M., Gefen, D., & Straub, D. (2001). Validation in IS research: a state-of-the-art assessment. MIS Quarterly, 25(1), 1–24.CrossRefGoogle Scholar
  15. Boudreau, K. J., Nicola, L., & Karim, R. L. (2011). Incentives and problem uncertainty in innovation contests: an empirical Analysis. Management Science, 57(5), 843–863.CrossRefGoogle Scholar
  16. Brabham, D. C. (2008a). Crowdsourcing as a model for problem solving: an introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90.Google Scholar
  17. Brabham, D. C. (2008b). “Moving the crowd at iStockphoto: the composition of the crowd and motivations for participation in a crowdsourcing application. First Monday, 13(6).Google Scholar
  18. Brabham, D. C. (2009). Crowdsourcing the public participation process for planning projects. Planning Theory, 8(3), 242–262.CrossRefGoogle Scholar
  19. Brabham, D. C. (2010). Moving the crowd at threadless: motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8), 1122–1145.CrossRefGoogle Scholar
  20. Brabham, D. C. (2011) forthcoming. Crowdsourcing: A model for leveraging online communities. In A. Delwiche & J. Henderson (Eds.), The Routledge Handbook of Participatory Culture.Google Scholar
  21. Brew, A., Greene, D., & Cunningham, P. (2010). Using crowdsourcing and active learning to track sentiment in online media. in Proceedings of ECAI 2010 19th European Conference on Artificial Intelligence, pp. 86–96.Google Scholar
  22. Brito, J. (2008). Hack, Mash, & Peer: Crowdsourcing government transparency. The Columbia Science and Technology Law Review, 9, 119–157.Google Scholar
  23. Chanal, V., Caron-Fasan, & M. L. (2008). How to invent a new business model based on crowdsourcing: The CROWDSPIRIT Case. Conférence de l'Association Internationale de Management Stratégique, Nice, May.Google Scholar
  24. Cheon, M. J., Grover, V., & Teng, T. C. (1995). Theoretical perspectives on the outsourcing of information systems. Journal of Information Technology, 10(4), 209–219.CrossRefGoogle Scholar
  25. Chesbrough, H. W. (2003). The era of open innovation. MIT Sloan Management Review, 44(3), 35–41.Google Scholar
  26. Chesbrough, H. (2006a). Open innovation: A new paradigm for understanding industrial innovation. In Chesbrough et al., 2006. “Open Innovation: Researching a New Paradigm (pp 1–12). Oxford: Oxford University Press.Google Scholar
  27. Chesbrough, H. (2006b). Open business models: How to thrive in the new innovation landscape. Cambridge, MA: Harvard Business School Press.Google Scholar
  28. Chiaroni, D., Chiesa, V., & Frattini. (2009). Investigating the adoption of open innovation in the bio-pharmaceutical industry: a framework and an empirical analysis. European Journal of Innovation Management, 12(3), 285–305.CrossRefGoogle Scholar
  29. Choffnes, D. R., Bustamante, F. E., & Ge, Z. H. (2010). Crowdsourcing service-level network event monitoring. in Proceedings of the ACM SIGCOMM 2010 conference on SIGCOMM ACM, pp. 387–398.Google Scholar
  30. Chris, C. B. (2009). “Fast, cheap, and creative: Evaluating translation quality using Amazons mechanical Turk,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Vol. 1. PA, USA: Association for Computational Linguistics StroudsburgGoogle Scholar
  31. Delaney, D. G. (2008). Marine invasive species: validation of citizen science and implications for national monitoring networks. Biological Invasions, 10, 117–128.CrossRefGoogle Scholar
  32. Diana, R. (2010). Crowdsourcing is outsourcing Web 2.0 Style. Retrieved January 12, from
  33. DiPalantino, D., & Vojnovic, M. (2009). Crowdsourcing and all-pay auctions. in Proceedings of the Tenth ACM conference on Electronic commerce, pp. 119–128.Google Scholar
  34. Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86–96.CrossRefGoogle Scholar
  35. Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., & Stuckenschmidt, H. (2010). Crowdsourcing the Assembly of Concept Hierarchies. in Proceedings of the 10th annual joint conference on Digital libraries ACM, pp. 139–148.Google Scholar
  36. Estellés-Arolas, E., & González Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science (in press) Google Scholar
  37. Franklin, M, Kossman, D., Kraska, T, Ramesh, S, & Xin, R. (2011). CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of SIGMOD 2011.Google Scholar
  38. Gallivan, M. J. (2001). Striking a balance between trust anti control in a virtual organization: a content analysis of open source software case studies. Information Systems Journal, 11(4), 277–304.CrossRefGoogle Scholar
  39. Galloway, A. W. E., et al. (2006). The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildlife Society Bulletin, 34, 1425–1429.CrossRefGoogle Scholar
  40. Greengard, S. (2011). Following the Crowd. Communications of the ACM, 54(2), 20–22.CrossRefGoogle Scholar
  41. Gregg, D. G. (2010). Designing for collective intelligence. Communications of the ACM, 53(4), 134–138.CrossRefGoogle Scholar
  42. Gregor, S. (2006). The Nature of Theory in Information Systems. MIS Quarterly, 30(3), 611–642.Google Scholar
  43. Hars, A., & Ou, S. S. (2002). Working for free? Motivations for participating in open-source projects. International Journal of Electronic Commerce, 6(3), 25–39.Google Scholar
  44. Hauser, L. (1988). A conceptual analysis of information science. Library and Information Science Research, 10, 3–34.Google Scholar
  45. Heer, J., & Bostock, M. (2010). Crowdsourcing Graphical Perception. in Proceedings of the 28th international conference on Human factors in computing systems ACM, pp. 286–295.Google Scholar
  46. Hertel, G., Niedner, S., & Herrmann, S. (2003). Motivation of software developers in open source projects: an internet-based survey of contributors to the Linux Kernel. Research Policy, 32(7), 1159–1177.Google Scholar
  47. Hevner, A., March, S., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.Google Scholar
  48. Holley, R. (2009). Crowdsourcing and social engagement: potential, power and freedom for libraries and users. in Proceedings of Pacific Rim Digital Library Alliance (PRDLA) Annual meeting and Conference: Libraries at the End of the World: Digital Content and Knowledge Creation, pp. 1–28.Google Scholar
  49. Horton, J. J., & Chilton, L. B. (2010). The Labor Economics of Paid Crowdsourcing. in Proceedings of the 11th ACM conference on Electronic commerce ACM, pp. 209–218.Google Scholar
  50. Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.Google Scholar
  51. Howe, J. (2008). Crowdsourcing. New York: Crown Publishing Group.Google Scholar
  52. Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. in Proceedings of the NAACL HLT 2009Workshop on Active Learning for Natural Language Processing, pp. 27–35.Google Scholar
  53. Huberman, B. A., Romero, D., & Wu, F. (2009). Crowdsourcing, attention and productivity. Journal of Information Science, 35(6), 758–765.CrossRefGoogle Scholar
  54. Hudson-Smith, A., Batty, M., Crooks, A., & Milton, R. 2009. Mapping for the Masses: Accessing Web 2.0 through Crowdsourcing. Social Science Computer Review, pp. 1066–1084.Google Scholar
  55. Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality Management on Amazon Mechanical Turk. In KDD-HCOMP’10, July 2010, Washington DC, USA.Google Scholar
  56. Jain, R. (2010). Investigation of Governance Mechanisms for Crowdsourcing Initiatives. in Proceedings of Americas Conference on Information Systems, pp. 557–563.Google Scholar
  57. Johnson, J. (2007). Social networks and the wisdom of crowds. Network World, 24, 28–29.Google Scholar
  58. Kazman, R., & Chen, H. (2009). The metropolis model a new logic for development of crowdsourced systems. Communications of the ACM, 52(7), 76–84.CrossRefGoogle Scholar
  59. Ke, W., & Zhang, P. (2009). Motivations in open source software communities: the mediating role of effort intensity and goal commitment. International Journal of Electronic Commerce, 13(4), 39–66.CrossRefGoogle Scholar
  60. Ke, W., & Zhang, P. (2010). The effects of extrinsic motivations and satisfaction in open source software development. Journal of the Association for Information Systems, 11(12), 784–808.Google Scholar
  61. Keen, A. (2007). The cult of the amateur: How today’s internet is killing our culture and assaulting our economy. London: Nicholas Brealey.Google Scholar
  62. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing User Studies with Mechanical Turk. in Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pp. 453–456.Google Scholar
  63. Kleeman, F., Voss, G. G., & Rieder, K. (2008). Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing. Science Technology & Innovation Studies, 4(1), 5–26.Google Scholar
  64. Lakhani, K. R., & Wolf R. G. (2005). Why hackers do what they do: understanding motivation and effort in free/open source software projects. In J. Feller, B. Fitzgerald, S. Hissam, & K. R. Lakhani (Eds.). Perspectives on free and open source software. Cambridge, MA: MIT Press.Google Scholar
  65. Lakhani, K. R., Jeppesen, L. B., Lohse, P. A., & Panetta, J. A. (2007). The value of openness in scientific problem solving. Harvard Business School Working Paper, (7: 50), Retrieved October 15, from–050.pdf
  66. Lane, S. (2010). Collective intelligence for competitive advantage: Crowd sourcing and open innovation. Continuing Education, 1–80.Google Scholar
  67. Lee, J.-N., & Kim, Y.-G. (1999). Effect of partnership quality on is outsourcing success: conceptual framework and empirical validation. Journal of Management Information Systems, 15(4), 29–61.CrossRefGoogle Scholar
  68. Leimeister, J. M. (2010). Collective Intelligence. Business & Information Systems Engineering, April, pp. 245–248.Google Scholar
  69. Leimeister, J. M., Huber, M., Bretschneider, U., & Krcmar, H. (2009). Leveraging crowdsourcing -activation-supporting components for IT-based idea competitions. Journal of Management Information Systems, 26(1), 197–224.CrossRefGoogle Scholar
  70. Lévy, P. (1995). Collective intelligence: Mankind’s emerging world in cyberspace (R. Bononno, Trans.). New York: Plenum.Google Scholar
  71. Maiolini, R., & Naggi, R. (2010). Crowdsourcing and SMEs: Opportunities and challenges. in Proceedings of ITAIS, 2010.Google Scholar
  72. Malone, T. W., Laubacher, R., & Dellarocas, C. (2010). Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan School Working Paper, no.4732-09.Google Scholar
  73. Marcus, A., Wu, E., & Madden, S. (2011). Crowdsourcing Databases: Query Processing with People. In Proceedings of CRDR 2011. Google Scholar
  74. Mason, W., & Watts, D. J. (2010). Financial Incentives and the performance of crowds. SIGKDD Explorations Newsletter, 11(2), 100. ACM.CrossRefGoogle Scholar
  75. Mazzola, D., & Distefano, A. (2010). Crowdsourcing and the participation process for problem solving: The Case of BP. in Proceedings of ItAIS 2010 VII Conference of the Italian Chapter of AIS, pp. 42–49.Google Scholar
  76. Meier, P. (2010). How to run a successful crowdsourcing project. Retrieved December 22, from
  77. Munro, R., Steven B., Victor K., Robin M., Christopher P., Tyler S., et al. (2010). Crowdsourcing and language studies: The new generation of linguistic data. in Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Mechanical Turk, pp. 122–130.Google Scholar
  78. Nov, O., Arazy, O., & Anderson, D. (2011). Technology-mediated citizen science participation: a motivational model. Proceedings of the AAAI International Conference on Weblogs and Social Media (ICWSM 2011), Barcelona, Spain.Google Scholar
  79. Orlikowski, W. J., & Iacono, C. S. (2001). Desperately seeking the ‘IT’ in IS research - a call to theorizing the IT artifact. Information Systems Research, 12(2), 121–134.CrossRefGoogle Scholar
  80. Pettigrew, K. E., & McKechnie, L. M. (2001). The use of theory in information science research. Journal of the American Society for Information Science & Technology, 52(1), 62–73.CrossRefGoogle Scholar
  81. Poetz, M. K., & Schreier, M. (2009). The value of crowdsourcing: can users really compete with professionals in generating new product ideas? Proceedings of the DRUID Summer Conference, 2009, 110–128.Google Scholar
  82. Raddick, M., et al. (2010). Galaxy zoo: exploring the motivations of citizen science volunteers. Astronomy Education Review, 9(1).Google Scholar
  83. Reeve, J. (2005). Understanding motivation and emotion. New York: John Wiley & Sons.Google Scholar
  84. Riedl, C., Blohm, I., Leimeister, J. M., & Krcmar, H. (2010). Rating scales for collective intelligence in innovation communities: Why quick and easy decision making does not get it right. In Proceedings of 2010 International Conference on Information Systems, St. Louis, Mi, USA: AIS.Google Scholar
  85. Roberts, J. A., Hann, I. H., & Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: a longitudinal study of the apache projects. Management Science, 52(7), 984–999.Google Scholar
  86. Roman, D. (2009). Crowdsourcing and the question of expertise. Communications of the ACM, 52(12), 12.CrossRefGoogle Scholar
  87. Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. ACIS 2010 Proceedings, Paper 76.
  88. Schenk, E., & Guittard, C. (2009). Crowdsourcing: What can be outsourced to the crowd, and why? Paper provided by HAL in its series of working papers with numbers halshs-00439256_v1 Google Scholar
  89. Shah, N., Dhanesha, A., & Seetharam, D. (2009). Crowdsourcing for e-governance: Case study. in Proceedings of the 3rd International Conference on Theory and Practice of Electronic Governance (New York, NY, USA, 2009), ACM, pp. 253–258.Google Scholar
  90. Sharma, A. (2010). Crowdsourcing critical success factor model: Strategies to harness the collective intelligence of the crowd. Retrieved January 27, from
  91. Sims, J., & Crossland, C. (2010). “Partners or Pariahs? Firm engagement with open innovation communities. in: Academy of Management Conference. Montreal, Ca.Google Scholar
  92. Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 50–53.Google Scholar
  93. Stewart, O., Lubensky, D., & Huerta, J. M. (2010). Crowdsourcing participation inequality: A SCOUT model for the enterprise domain. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 30–33.Google Scholar
  94. Stolee, K. T., & Elbaum, S. (2010). Exploring the use of crowdsourcing to support empirical studies in software engineering. in Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement ACM, pp. 35–38.Google Scholar
  95. Strauss, A. (1987). Qualitative research for social scientists. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  96. Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. New York. Google Scholar
  97. Tapscott, D., & Williams, A. D. (2007). Wikinomics: How mass collaboration changes everything. London, UK: Portfolio, Penguin.Google Scholar
  98. Trompette, P. (2008). Crowdsourcing as a way to access external knowledge for innovation: Control, incentive and coordination in hybrid forms of innovation. in Proceedings of 24th EGOS Colloquium, pp. 1–29.Google Scholar
  99. von Hippel, E. (2005). Democratizing innovation. Cambridge, Mass: MIT Press.Google Scholar
  100. Vukovic, M. (2009). Crowdsourcing for enterprises. in Proceedings of Congress on Services, pp. 686–692.Google Scholar
  101. Vukovic, M., Kumara, S., & Greenshpan, O. (2010). Ubiquitous crowdsourcing. in Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing ACM, pp. 523–525.Google Scholar
  102. Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), 13–23.Google Scholar
  103. Whitla, P. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15–28.Google Scholar
  104. Wiggins, A., & Crowston, K. (2011). From conservation to crowdsourcing: A typology of citizen science. in Proceedings of the Forty-fourth Hawai'i International Conference on System Science (HICSS-44).Google Scholar
  105. Wilcox, R. T. (2000). Experts and amateurs: the role of experience in internet auctions. Marketing Letters, 11, 363–374.CrossRefGoogle Scholar
  106. Yang, J., Adamic, L. A., & Ackerman, M. S. (2008). Crowdsourcing and knowledge Sharing: Strategic user behavior on taskcn. in Proceedings of ACM Electronic Commerce’08, pp. 246–255.Google Scholar
  107. Zhang, P. (2008). Motivational affordances: Reasons for ICT design and USE. Communications of the ACM, 61(11), 145–147.CrossRefGoogle Scholar
  108. Zhang, P., & Benjamin, R. I. (2007). Understanding information related fields: a conceptual framework. Journal of the American Society for Information Science and Technology, 58(13), 1934–1947.CrossRefGoogle Scholar
  109. Zhang, P., & Li, N. (2005). The intellectual development of human–computer interaction research: a critical assessment of the MIS literature (1990–2002). Journal of Association for Information Systems, 6(11), 227–292.Google Scholar
  110. Zhang, P., Scialdone, M., & Ku, M. (2011). IT artifacts and the state of IS research. forthcoming in the Proceedings of ICIS 2011, Shanghai, China, December.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.School of Information ManagementNanjing UniversityNanjingChina

Personalised recommendations