Crowdsourcing Controls: A Review and Research Agenda for Crowdsourcing Controls Used for Macro-tasks

  • Lionel P. RobertJr.Email author
Part of the Human–Computer Interaction Series book series (HCIS)


Crowdsourcing—the employment of ad hoc online labor to perform various tasks—has become a popular outsourcing vehicle. Our current approach to crowdsourcing—focusing on micro-tasks—fails to leverage the potential of crowds to tackle more complex problems. To leverage crowds to tackle more complex macro-tasks requires a better comprehension of crowdsourcing controls. Crowdsourcing controls are mechanisms used to align crowd workers’ actions with predefined standards to achieve a set of goals and objectives. Unfortunately, we know very little about the topic of crowdsourcing controls directed at accomplishing complex macro-tasks. To address issues associated with crowdsourcing controls for macro-tasks, this chapter has several objectives. First, it presents and discusses the literature on control theory. Second, this chapter presents a scoping literature review of crowdsourcing controls. Finally, the chapter identifies gaps and puts forth a research agenda to address these shortcomings. The research agenda focuses on understanding how to employ the controls needed to perform macro-tasking in crowds and the implications for crowdsourcing system designers.



This book chapter was supported in part by the National Science Foundation [grant CHS-1617820].


  1. Aker, A., El-Haj, M., Albakour, M. D., & Kruschwitz, U. (2012). Assessing crowdsourcing quality through objective tasks. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (pp. 1456–1461).Google Scholar
  2. Ashikawa, M., Kawamura, T., & Ohsuga, A. (2015). Deployment of private crowdsourcing system with quality control methods. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) (Vol. 1, pp. 9–16). IEEE.Google Scholar
  3. Baba, Y., & Kashima, H. (2013, August). Statistical quality estimation for general crowdsourcing tasks. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 554–562). ACM.Google Scholar
  4. Baba, Y., Kashima, H., Kinoshita, K., Yamaguchi, G., & Akiyoshi, Y. (2013, June). Leveraging crowdsourcing to detect improper tasks in crowdsourcing marketplaces. In Twenty-fifth Innovative Applications of Artificial Intelligence Conference (pp. 1487–1492).Google Scholar
  5. Baba, Y., Kashima, H., Kinoshita, K., Yamaguchi, G., & Akiyoshi, Y. (2014). Leveraging non-expert crowdsourcing workers for improper task detection in crowdsourcing marketplaces. Expert Systems with Applications, 41(6), 2678–2687.Google Scholar
  6. Bell, S., & Bala, K. (2015). Learning visual similarity for product design with convolutional neural networks. ACM Transactions on Graphics (TOG), 34(4), 98.Google Scholar
  7. Bontcheva, K., Roberts, I., Derczynski, L., & Rout, D. (2014). The GATE crowdsourcing plugin: Crowdsourcing annotated corpora made easy. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics (pp. 97–100).Google Scholar
  8. Bozzon, A., Brambilla, M., Ceri, S., & Mauri, A. (2013, May). Reactive crowdsourcing. In Proceedings of the 22nd International Conference on World Wide Web (pp. 153–164). ACM.Google Scholar
  9. Bozzon, A., Brambilla, M., Ceri, S., Mauri, A., & Volonterio, R. (2014, July). Pattern-based specification of crowdsourcing applications. In International Conference on Web Engineering (pp. 218–235). Cham: Springer.Google Scholar
  10. Bragg, J., & Weld, D. S. (2013, November). Crowdsourcing multi-label classification for taxonomy creation. In First AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
  11. Cardinal, L. B., Kreutzer, M., & Miller, C. C. (2017). An aspirational view of organizational control research: Re-invigorating empirical work to better meet the challenges of 21st century organizations. Academy of Management Annals, 11(2), 559–592.Google Scholar
  12. Cardinal, L. B., Sitkin, S. B., & Long, C. P. (2004). Balancing and rebalancing in the creation and evolution of organizational control. Organization Science, 15, 411–431.Google Scholar
  13. Cardinal, L. B., Sitkin, S. B., & Long, C. P. (2010). A configurational theory of control. In S. B. Sitkin, L. B. Cardinal, & K. M. Bijlsma-Frankema (Eds.), Organizational control (pp. 51–79). Cambridge, UK: Cambridge University Press.Google Scholar
  14. Carpenter, M. A., Bauer, T., Erdogan, B., & Short, J. (2010). Principles of management. Flatworld Knowledge.Google Scholar
  15. Causer, T., Tonra, J., & Wallace, V. (2012). Transcription maximized; expense minimized? Crowdsourcing and editing the collected works of Jeremy Bentham. Literary and Linguistic Computing, 27(2), 119–137.Google Scholar
  16. Chang, D., Chen, C. H., & Lee, K. M. (2014). A crowdsourcing development approach based on a neuro-fuzzy network for creating innovative product concepts. Neurocomputing, 142, 60–72.Google Scholar
  17. Chatman, J. A. (2010). Norms in mixed sex and mixed race work groups. Academy of Management Annals, 4(1), 447–484.Google Scholar
  18. Chen, Z., Fu, R., Zhao, Z., Liu, Z., Xia, L., Chen, L., et al. (2014). gMission: A general spatial crowdsourcing platform. Proceedings of the VLDB Endowment, 7(13), 1629–1632.Google Scholar
  19. Cheng, J., Teevan, J., Iqbal, S. T., & Bernstein, M. S. (2015, April). Break it down: A comparison of macro-and microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4061–4064). ACM.Google Scholar
  20. Chiu, C. M., Liang, T. P., & Turban, E. (2014). What can crowdsourcing do for decision support? Decision Support Systems, 65, 40–49.Google Scholar
  21. Choudhury, V., & Sabherwal, R. (2003). Portfolios of control in outsourced software development projects. Information Systems Research, 14(3), 291–314.Google Scholar
  22. Chung, M. J. Y., Forbes, M., Cakmak, M., & Rao, R. P. (2014, May). Accelerating imitation learning through crowdsourcing. In ICRA (pp. 4777–4784).Google Scholar
  23. Dai, P., Lin, C. H., & Weld, D. S. (2013). POMDP-based control of workflows for crowdsourcing. Artificial Intelligence, 202, 52–85.MathSciNetzbMATHGoogle Scholar
  24. Dai, P., Rzeszotarski, J. M., Paritosh, P., & Chi, E. H. (2015, February). And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 628–638). ACM.Google Scholar
  25. Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018). Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys (CSUR), 51(1), 7.Google Scholar
  26. de Herrera, A. G. S., Foncubierta-Rodrıguez, A., Markonis, D., Schaer, R., & Müller, H. (2014, September). Crowdsourcing for medical image classification. In Annual Congress SGMI (Vol. 2014).Google Scholar
  27. Deng, J., Krause, J., & Fei-Fei, L. (2013). Fine-grained crowdsourcing for fine-grained recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 580–587).Google Scholar
  28. Dennis, A. R., Robert, L. P., Kowalczyck, S. T., Curtis, A., & Hasty, B. K. (2012). Trust is in the eye of the beholder: A vignette study of postevent behavioral controls’ effects on individual trust in virtual teams. Information Systems Research, 23(2), 546–558.Google Scholar
  29. Difallah, D. E., Demartini, G., & Cudré-Mauroux, P. (2012, April). Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch 2010 Workshop at WWW 2012 (pp. 26–30).Google Scholar
  30. Duan, L., Oyama, S., Sato, H., & Kurihara, M. (2014). Separate or joint? Estimation of multiple labels from crowdsourced annotations. Expert Systems with Applications, 41(13), 5723–5732.Google Scholar
  31. Eickhoff, C., & de Vries, A. (2011, February). How crowdsourcable is your task? In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the Fourth ACM International Conference on Web Search and Data Mining (WSDM) (pp. 11–14).Google Scholar
  32. Eickhoff, C., & de Vries, A. P. (2013). Increasing cheat robustness of crowdsourcing tasks. Information Retrieval, 16(2), 121–137.Google Scholar
  33. Eisenhardt, K. M. (1985). Control: Organizational and economic approaches. Management Science, 31, 134–149.Google Scholar
  34. Fan, J., Li, G., Ooi, B. C., Tan, K. L., & Feng, J. (2015, May). iCrowd: An adaptive crowdsourcing framework. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (pp. 1015–1030). ACM.Google Scholar
  35. Fang, Y., Sun, H., Li, G., Zhang, R., & Huai, J. (2016, April). Effective result inference for context-sensitive tasks in crowdsourcing. In International Conference on Database Systems for Advanced Applications (pp. 33–48). Cham: Springer.Google Scholar
  36. Filatova, E. (2012, May). Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (pp. 392–398).Google Scholar
  37. Finin, T., Murnane, W., Karandikar, A., Keller, N., Martineau, J., & Dredze, M. (2010, June). Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 80–88). Association for Computational Linguistics.Google Scholar
  38. Foncubierta Rodríguez, A., & Müller, H. (2012, October). Ground truth generation in medical imaging: A crowdsourcing-based iterative approach. In Proceedings of the ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia (pp. 9–14). ACM.Google Scholar
  39. Franklin, M. J., Kossmann, D., Kraska, T., Ramesh, S., & Xin, R. (2011, June). CrowdDB: Answering queries with crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (pp. 61–72). ACM.Google Scholar
  40. Fu, W. T., & Liao, V. (2011, March). Crowdsourcing quality control of online information: A quality-based cascade model. In International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction (pp. 147–154). Berlin, Heidelberg: Springer.Google Scholar
  41. Gadiraju, U., Kawase, R., Dietze, S., & Demartini, G. (2015). Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1631–1640). ACM.Google Scholar
  42. Gao, Y., Chen, Y., & Liu, K. R. (2015). On cost-effective incentive mechanisms in microtask crowdsourcing. IEEE Transactions in Computational Intelligence and AI in Games, 7(1), 3–15.Google Scholar
  43. Gould, S. J., Cox, A. L., & Brumby, D. P. (2016). Diminished control in crowdsourcing: An investigation of crowdworker multitasking behavior. ACM Transactions on Computer-Human Interaction (TOCHI), 23(3), 19.Google Scholar
  44. Haas, D., Ansel, J., Gu, L., & Marcus, A. (2015). Argonaut: Macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12), 1642–1653.Google Scholar
  45. Han, S., Dai, P., Paritosh, P., & Huynh, D. (2016). Crowdsourcing human annotation on web page structure: Infrastructure design and behavior-based quality control. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4), 56.Google Scholar
  46. Hansen, D. L., Schone, P. J., Corey, D., Reid, M., & Gehring, J. (2013, February). Quality control mechanisms for crowdsourcing: Peer review, arbitration, & expertise at family search indexing. In Proceedings of the 2013 Conference on Computer-Supported Cooperative Work (pp. 649–660). ACM.Google Scholar
  47. Hara, K., Le, V., & Froehlich, J. (2013, April). Combining crowdsourcing and Google Street View to identify street-level accessibility problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 631–640). ACM.Google Scholar
  48. Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2010). Cheat-detection mechanisms for crowdsourcing. Research report series, report No. 474.Google Scholar
  49. Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2011, June). Cost-optimal validation mechanisms and cheat-detection for crowdsourcing platforms. In 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS) (pp. 316–321). IEEE.Google Scholar
  50. Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57(11–12), 2918–2932.Google Scholar
  51. Hoßfeld, T., & Keimel, C. (2014). Crowdsourcing in QoE evaluation. In Quality of experience (pp. 315–327). Cham: Springer.Google Scholar
  52. Homan, A. C., van Knippenberg, D., Van Kleef, G. A., & De Dreu, C. K. W. (2007). Bridging faultlines by valuing diversity: The effects of diversity beliefs on information elaboration and performance in diverse work groups. Journal of Applied Psychology, 92, 1189–1199.Google Scholar
  53. Hosio, S., Goncalves, J., Lehdonvirta, V., Ferreira, D., & Kostakos, V. (2014, October). Situated crowdsourcing using a market model. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (pp. 55–64). ACM.Google Scholar
  54. Hutton, A., Liu, A., & Martin, C. E. (2012, March). Crowdsourcing evaluations of classifier interpretability. In AAAI Spring Symposium: Wisdom of the Crowd.Google Scholar
  55. Jaworski, B. J., & Kohli, A. K. (1993). Market orientation: Antecedents and consequences. Journal of Marketing, 57, 53–70.Google Scholar
  56. Jo, J., Stevens, A., & Tan, C. (2013). A quality control model for trustworthy crowdsourcing in collaborative learning. In Robot intelligence technology and applications 2012 (pp. 85–90). Berlin, Heidelberg: Springer.Google Scholar
  57. Kajino, H., Arai, H., & Kashima, H. (2014). Preserving worker privacy in crowdsourcing. Data Mining and Knowledge Discovery, 28(5–6), 1314–1335.MathSciNetzbMATHGoogle Scholar
  58. Kamar, E. (2016, July). Directions in hybrid intelligence: Complementing AI systems with human intelligence. In IJCAI (pp. 4070–4073).Google Scholar
  59. Kamar, E., Kapoor, A., Horvitz, E., & Redmond, W. A. (2013, August). Lifelong learning for acquiring the wisdom of the crowd. In IJCAI (Vol. 13, pp. 2313–2320).Google Scholar
  60. Kannangara, S. N., & Uguccioni, P. (2013). Risk management in crowdsourcing-based business ecosystems. Technology Innovation Management Review, 3(12).Google Scholar
  61. Kazai, G. (2011, April). In search of quality in crowdsourcing for search engine evaluation. In European Conference on Information Retrieval (pp. 165–176). Berlin, Heidelberg: Springer.Google Scholar
  62. Kazai, G., Kamps, J., Koolen, M., & Milic-Frayling, N. (2011, July). Crowdsourcing for book search evaluation: Impact of hit design on comparative system ranking. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 205–214). ACM.Google Scholar
  63. Kazai, G., Kamps, J., & Milic-Frayling, N. (2012, October). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (pp. 2583–2586). ACM.Google Scholar
  64. Kazai, G., Koolen, M., Kamps, J., Doucet, A., & Landoni, M. (2010, December). Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing. In International Workshop of the Initiative for the Evaluation of XML Retrieval (pp. 98–117). Berlin, Heidelberg: Springer.Google Scholar
  65. Kazai, G., & Zitouni, I. (2016, February). Quality management in crowdsourcing using gold judges behavior. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (pp. 267–276). ACM.Google Scholar
  66. Khapra, M. M., Ramanathan, A., Kunchukuttan, A., Visweswariah, K., & Bhattacharyya, P. (2014). When transliteration met crowdsourcing: An empirical study of transliteration via crowdsourcing using efficient, non-redundant and fair quality control. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014) (pp. 196–202).Google Scholar
  67. Khazankin, R., Psaier, H., Schall, D., & Dustdar, S. (2011, December). Qos-based task scheduling in crowdsourcing environments. In International Conference on Service-oriented Computing (pp. 297–311). Berlin, Heidelberg: Springer.Google Scholar
  68. Kim, S., Marquis, E., Alahmad, R., Pierce, C., & Robert, L. P. (2018). The impacts of platform quality on gig workers’ autonomy and satisfaction. In Proceedings of the 21th ACM Conference on Computer-supported Cooperative Work and Social Computing Companion. Jersey City, NJ, USA.Google Scholar
  69. Kirsch, L. J. (1997). Portfolios of control modes and IS project management. Information Systems Research, 8(3), 215–239.Google Scholar
  70. Kirsch, L. J., Ko, D. G., & Haney, M. H. (2010). Investigating the antecedents of team-based clan control: Adding social capital as a predictor. Organization Science, 21(2), 469–489.Google Scholar
  71. Lange, R., & Lange, X. (2012, March). Quality control in crowdsourcing: An objective measurement approach to identifying and correcting rater effects in the social evaluation of products and services. In AAAI Spring Symposium: Wisdom of the Crowd (Vol. 12, p. 6).Google Scholar
  72. Lasecki, W. S., & Bigham, J. P. (2012, October). Online quality control for real-time crowd captioning. In Proceedings of the 14th international ACM SIGACCESS Conference on Computers and Accessibility (pp. 143–150). ACM.Google Scholar
  73. Lasecki, W. S., Miller, C. D., & Bigham, J. P. (2013, April). Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2033–2036). ACM.Google Scholar
  74. Lasecki, W. S., Murray, K. I., White, S., Miller, R. C., & Bigham, J. P. (2011, October). Real-time crowd control of existing interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (pp. 23–32). ACM.Google Scholar
  75. Lasecki, W. S., Teevan, J., & Kamar, E. (2014, February). Information extraction and manipulation threats in crowd-powered systems. In Proceedings of the 17th ACM Conference on Computer-supported Cooperative Work & Social Computing (pp. 248–256). ACM.Google Scholar
  76. Le, J., Edmonds, A., Hester, V., & Biewald, L. (2010, July). Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (Vol. 2126).Google Scholar
  77. Lee, C. Y., & Glass, J. (2011). A transcription task for crowdsourcing with automatic quality control. Paper Presented at the Twelfth Annual Conference of the International Speech Communication Association.Google Scholar
  78. Li, H., Zhao, B., & Fuxman, A. (2014, April). The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web (pp. 165–176). ACM.Google Scholar
  79. Li, Q., Vempaty, A., Varshney, L. R., & Varshney, P. K. (2017). Multi-object classification via crowdsourcing with a reject option. IEEE Transactions on Signal Processing, 65(4), 1068–1081.MathSciNetzbMATHGoogle Scholar
  80. Lin, C. H., & Weld, D. (2012). In N. de Freitas & K. Murphy (Eds.), Proceedings of the Twenty-eighth Conference on Uncertainty in Artificial Intelligence (UAI’12) (pp. 491–500). Arlington, VA: AUAI Press.Google Scholar
  81. Liu, Q., Ihler, A. T., & Steyvers, M. (2013). Scoring workers in crowdsourcing: How many control questions are enough? In Advances in neural information processing systems (pp. 1914–1922).Google Scholar
  82. Liu, S. (2015). Effects of control on the performance of information systems projects: The moderating role of complexity risk. Journal of Operations Management, 36, 46–62.Google Scholar
  83. Liu, Z., Shabani, S., Balet, N. G., Sokhn, M., & Cretton, F. (2018, January). How to motivate participation and improve quality of crowdsourcing when building accessibility maps. In 2018 15th IEEE Annual Consumer Communications & Networking Conference (CCNC) (pp. 1–6). IEEE.Google Scholar
  84. Loni, B., Menendez, M., Georgescu, M., Galli, L., Massari, C., Altingovde, I. S., … & Larson, M. (2013, February). Fashion-focused creative commons social dataset. In Proceedings of the 4th ACM Multimedia Systems Conference (pp. 72–77). ACM.Google Scholar
  85. Malhotra, A., & Majchrzak, A. (2014). Managing crowds in innovation challenges. California Management Review, 56(4), 103–123.Google Scholar
  86. Maruping, L. M., Venkatesh, V., & Agarwal, R. (2009). A control theory perspective on agile methodology use and changing user requirements. Information Systems Research, 20(3), 377–399.Google Scholar
  87. Massung, E., Coyle, D., Cater, K. F., Jay, M., & Preist, C. (2013, April). Using crowdsourcing to support pro-environmental community activism. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 371–380). ACM.Google Scholar
  88. Mays, N., Roberts, E., & Popay, J. (2001). Synthesising research evidence. In N. Fulop, P. Allen, A. Clarke, & N. Black (Eds.), Studying the organisation and delivery of health services: Research methods (pp. 188–219). London: Routledge.Google Scholar
  89. McGraw, I., & Polifroni, J. (2013). How to control and utilize crowd-collected speech. In M. Eskenazi, G. Levow, H. Meng, G. Parent, & D. Suendermann (Eds.), Crowdsourcing for speech processing: Applications to data collection, transcription and assessment (pp. 106–136). Chichester, UK: Wiley.Google Scholar
  90. Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., et al. (2016). Crowdsourcing quality control for Dark Energy Survey images. Astronomy and Computing, 16, 99–108.Google Scholar
  91. Munro, R., Bethard, S., Kuperman, V., Lai, V. T., Melnick, R., Potts, C., … & Tily, H. (2010, June). Crowdsourcing and language studies: The new generation of linguistic data. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 122–130). Association for Computational Linguistics.Google Scholar
  92. Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D., & Marchetti, A. (2011, July). Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 670–679). Association for Computational Linguistics.Google Scholar
  93. Oleson, D., Sorokin, A., Laughlin, G. P., Hester, V., Le, J., & Biewald, L. (2011). Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human Computation, 11(11).Google Scholar
  94. Otani, N., Baba, Y., & Kashima, H. (2016). Quality control of crowdsourced classification using hierarchical class structures. Expert Systems with Applications, 58, 155–163.Google Scholar
  95. Ouchi, W. G. (1979). A conceptual framework for the design of organizational control mechanisms. Management Science, 25(9), 833–848.Google Scholar
  96. Ouchi, W. G. (1980). Markets, bureaucracies, and clans. Administrative Science Quarterly, 25(1), 129–141.Google Scholar
  97. Ouchi, W. G., & Price, R. L. (1978). Hierarchies, clans, and theory Z: A new perspective on organization development. Organizational Dynamics, 7(2), 25–44.Google Scholar
  98. Oyama, S., Baba, Y., Ohmukai, I., Dokoshi, H., & Kashima, H. (2015). From one star to three stars: Upgrading legacy open data using crowdsourcing. In IEEE International Conference on Data Science and Advanced Analytics (pp. 1–9). IEEE.Google Scholar
  99. Oyama, S., Baba, Y., Sakurai, Y., & Kashima, H. (2013, August). Accurate integration of crowdsourced labels using workers’ self-reported confidence scores. In Twenty-third International Joint Conference on Artificial Intelligence (pp. 2554–2560).Google Scholar
  100. Paul, S. A., Hong, L., & Chi, E. H. (2011). What is a question? Crowdsourcing tweet categorization. Paper Presented at HCOMP Workshop CHI 2011.Google Scholar
  101. Peterson, J., Pearce, P. F., Ferguson, L. A., & Langford, C. A. (2017). Understanding scoping reviews: Definition, purpose, and process. Journal of the American Association of Nurse Practitioners, 29(1), 12–16.Google Scholar
  102. Piccoli, G., & Ives, B. (2003). Trust and the unintended effects of behavior control in virtual teams. MIS Quarterly, 27(3), 365–395.Google Scholar
  103. Post, M., Callison-Burch, C., & Osborne, M. (2012, June). Constructing parallel corpora for six Indian languages via crowdsourcing. In Proceedings of the Seventh Workshop on Statistical Machine Translation (pp. 401–409). Association for Computational Linguistics.Google Scholar
  104. Qiu, C., Squicciarini, A. C., Carminati, B., Caverlee, J., & Khare, D. R. (2016, October). Crowdselect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 539–548). ACM.Google Scholar
  105. Rhyn, M., & Blohm, I. (2017) A machine learning approach for classifying textual data in crowdsourcing. In J. M. Leimeister & W. Brenner, W. (Eds.), Proceedings der 13. Internationalen Tagung Wirtschaftsinformatik (WI 2017) (pp. 1171–1185).Google Scholar
  106. Riccardi, G., Ghosh, A., Chowdhury, S. A., & Bayer, A. O. (2013, August). Motivational feedback in crowdsourcing: A case study in speech transcription. In INTERSPEECH (pp. 1111–1115).Google Scholar
  107. Riegler, M., Gaddam, V. R., Larson, M., Eg, R., Halvorsen, P., & Griwodz, C. (2016, June). Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks. In 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI) (pp. 1–6). IEEE.Google Scholar
  108. Robert, L. P. (2016). Monitoring and trust in virtual teams. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2016). ACM.Google Scholar
  109. Robert, L. P., Jr., Dennis, A. R., & Ahuja, M. K. (2008). Social capital and knowledge integration in digitally enabled teams. Information Systems Research, 19(3), 314–334.Google Scholar
  110. Salehi, N., McCabe, A., Valentine, M., & Bernstein, M. (2017). Huddler: Convening stable and familiar crowd teams despite unpredictable availability. In Proceedings of the 2017 ACM Conference on Computer-Supported Cooperative Work and Social Computing (pp. 1700–1713). ACM.Google Scholar
  111. Salk, C. F., Sturn, T., See, L., Fritz, S., & Perger, C. (2016). Assessing quality of volunteer crowdsourcing contributions: Lessons from the Cropland Capture game. International Journal of Digital Earth, 9(4), 410–426.Google Scholar
  112. Satzger, B., Psaier, H., Schall, D., & Dustdar, S. (2013). Auction-based crowdsourcing supporting skill management. Information Systems, 38(4), 547–560.Google Scholar
  113. Schmitz, H., & Lykourentzou, I. (2018). Online sequencing of non-decomposable macrotasks in expert crowdsourcing. ACM Transactions on Social Computing, 1(1), 1.Google Scholar
  114. See, L., Schepaschenko, D., Lesiv, M., McCallum, I., Fritz, S., Comber, A., et al. (2015). Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS Journal of Photogrammetry and Remote Sensing, 103, 48–56.Google Scholar
  115. Sitkin, S. B., & George, E. (2005). Managerial trust-building through the use of legitimating formal and informal control mechanisms. International Sociology, 20(3), 307–338.Google Scholar
  116. Sorokin, A., Berenson, D., Srinivasa, S. S., & Hebert, M. (2010, October). People helping robots helping people: Crowdsourcing for grasping novel objects. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 2117–2122). IEEE.Google Scholar
  117. Sprugnoli, R., Moretti, G., Fuoli, M., Giuliani, D., Bentivogli, L., Pianta, E., … & Brugnara, F. (2013, May). Comparing two methods for crowdsourcing speech transcription. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8116–8120). IEEE.Google Scholar
  118. Star, S., & Griesemer, J. (1989). Institutional ecology, ‘translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387–420.Google Scholar
  119. Stolee, K. T., & Elbaum, S. (2010, September). Exploring the use of crowdsourcing to support empirical studies in software engineering. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (p. 35). ACM.Google Scholar
  120. Su, H., Deng, J., & Fei-Fei, L. (2012, July). Crowdsourcing annotations for visual object detection. In Workshops at the Twenty-sixth AAAI Conference on Artificial Intelligence (Vol. 1, No. 2).Google Scholar
  121. Tai, L., Chuang, Z., Tao, X., Ming, W., & Jingjing, X. (2011). Quality control of crowdsourcing through workers [sic] experience. In Proceedings of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval.Google Scholar
  122. Tang, W., & Lease, M. (2011, July). Semi-supervised consensus labeling for crowdsourcing. In SIGIR 2011 Workshop on Crowdsourcing for Information Retrieval (CIR) (pp. 1–6).Google Scholar
  123. Tiwana, A. (2010). Systems development ambidexterity: Explaining the complementary and substitutive roles of formal and informal controls. Journal of Management Information Systems, 27(2), 87–126.MathSciNetGoogle Scholar
  124. Tran-Thanh, L., Huynh, T. D., Rosenfeld, A., Ramchurn, S. D., & Jennings, N. R. (2014, May). BudgetFix: Budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (pp. 477–484). International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
  125. Trompette, P., Chanal, V., & Pelissier, C. (2008, July). Crowdsourcing as a way to access external knowledge for innovation. In 24th EGOS Colloquium.Google Scholar
  126. Turner, K. L., & Makhija, M. V. (2006). The role of organizational controls in managing knowledge. Academy of Management Review, 31(1), 197–217.Google Scholar
  127. Ul Hassan, U., Zaveri, A., Marx, E., Curry, E., & Lehmann, J. (2016, November). ACRyLIQ: Leveraging DBpedia for adaptive crowdsourcing in linked data quality assessment. In European Knowledge Acquisition Workshop (pp. 681–696). Cham: Springer.Google Scholar
  128. Vempaty, A., Varshney, L. R., & Varshney, P. K. (2014). Reliable crowdsourcing for multi-class labeling using coding theory. IEEE Journal of Selected Topics in Signal Processing, 8(4), 667–679.Google Scholar
  129. Venetis, P., & Garcia-Molina, H. (2012, August). Quality control for comparison microtasks. In Proceedings of the First International Workshop on Crowdsourcing and Data Mining (pp. 15–21). ACM.Google Scholar
  130. Vliegendhart, R., Larson, M., Kofler, C., Eickhoff, C., & Pouwelse, J. (2011, February). Investigating factors influencing crowdsourcing tasks with high imaginative load. In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the Fourth ACM International Conference on Web Search and Data Mining (pp. 27–30). ACM.Google Scholar
  131. Wais, P., Lingamneni, S., Cook, D., Fennell, J., Goldenberg, B., Lubarov, D., … & Simons, H. (2010). Towards building a high-quality workforce with Mechanical Turk. In Proceedings of Computational Social Science and the Wisdom of Crowds (NIPS) (pp. 1–5).Google Scholar
  132. Wang, S., Huang, C. R., Yao, Y., & Chan, A. (2014). Exploring mental lexicon in an efficient and economic way: Crowdsourcing method for linguistic experiments. In Proceedings of the 4th Workshop on Cognitive Aspects of the Lexicon (CogALex) (pp. 105–113).Google Scholar
  133. Weibel, A., Den Hartog, D. N., Gillespie, N., Searle, R., Six, F., & Skinner, D. (2016). How do controls impact employee trust in the employer? Human Resource Management, 55(3), 437–462.Google Scholar
  134. Windeler, J. B., Maruping, L. M., Robert, L. P., & Riemenschneider, C. K. (2015). E-profiles, conflict, and shared understanding in distributed teams. Journal of the Association for Information Systems, 16(7), 608.Google Scholar
  135. Wu, C. C., Chen, K. T., Chang, Y. C., & Lei, C. L. (2013). Crowdsourcing multimedia QoE evaluation: A trusted framework. IEEE Transactions on Multimedia, 15(5), 1121–1137.Google Scholar
  136. Xia, T., Zhang, C., Xie, J., & Li, T. (2012, September). Real-time quality control for crowdsourcing relevance evaluation. In 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) (pp. 535–539). IEEE.Google Scholar
  137. Ye, T., You, S., & Robert, L. P. (2017). When does more money work? Examining the role of perceived fairness in pay on the performance quality of crowdworkers. In Proceedings of the 11th International AAAI Conference on Web and Social Media.Google Scholar
  138. You, S., Robert Jr, L. P., & Rieh, S. Y. (2015, April). The appropriation paradox: Benefits and burdens of appropriating collaboration technologies. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1741–1746). ACM.Google Scholar
  139. Yung, D., Li, M. L., & Chang, S. (2014). Evolutionary approach for crowdsourcing quality control. Journal of Visual Languages & Computing, 25(6), 879–890.Google Scholar
  140. Zaidan, O. F., & Callison-Burch, C. (2011, June). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 1220–1229). Association for Computational Linguistics.Google Scholar
  141. Zhai, H., Lingren, T., Deleger, L., Li, Q., Kaiser, M., Stoutenborough, L., & Solti, I. (2013). Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of Medical Internet Research, 15(4).Google Scholar
  142. Zhang, G., & Chen, H. (2013, October). Quality control for crowdsourcing with spatial and temporal distribution. In International Conference on Internet and Distributed Computing Systems (pp. 169–182). Berlin, Heidelberg: Springer.Google Scholar
  143. Zhang, G., & Chen, H. (2013, December). Quality control of massive data for crowdsourcing in location-based services. In International Conference on Algorithms and Architectures for Parallel Processing (pp. 112–121). Cham: Springer.Google Scholar
  144. Zogaj, S., & Bretschneider, U. (2014). Analyzing governance mechanisms for crowdsourcing information systems: A multiple case analysis. In Proceedings of the European Conference on Information Systems 2014.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of Michigan School of InformationAnn ArborUSA

Personalised recommendations