Skip to main content
SpringerLink
Log in

Macrotask Crowdsourcing: An Integrated Definition

  • Chapter
  • First Online:
Book cover Macrotask Crowdsourcing

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

The conceptual distinction between microtasks and macrotasks has been made relatively early on in the crowdsourcing literature. However, only recently a handful of research works has explored it explicitly. These works, for the most part, have focused on simply discussing macrotasks within the confines of their own work (e.g., in terms of creativity), without taking into account the multiple facets that working with such tasks involves. This has resulted in the term “macrotask” to be severely convoluted and largely meaning different things to different individuals. More importantly, it has resulted in disregarding macrotask crowdsourcing as a new labor model of its own right. To address this scholarly gap, in this paper we discuss macrotask crowdsourcing from a multitude of dimensions, namely the nature of the problem it can solve, the crowdworker skills it involves, and the work management structures it necessitates. In view of our analysis, we provide a first integrated definition of macrotask crowdsourcing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Altshuller, G. (2005). The innovation algorithm: TRIZ, systematic innovation and technical creativity. Worchesrter, MA: Technical Innovation Center.

    Google Scholar 

  • Ambati, V., Vogel, S., & Carbonell, J. (2012). Collaborative workflow for crowdsourcing translation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12) (pp. 1191–1194). ACM, New York, NY, USA.

    Google Scholar 

  • Basu Roy, S., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2015). Task assignment optimization in knowledge-intensive crowdsourcing. The VLDB Journal—The International Journal on Very Large Data Bases, 24(4), 467–491.

    Google Scholar 

  • Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D. & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST ’10) (pp. 313–322). ACM, New York, NY, USA.

    Google Scholar 

  • Chan, J., Dang, S., & Dow, S. P. (2016). Improving crowd innovation with expert facilitation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16) (pp. 1223–1235). ACM, New York, NY, USA.

    Google Scholar 

  • Cheng, J., Teevan, J., Iqbal, S. T., & Bernstein, M. S. (2015, April). Break it down: A comparison of macro-and microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4061–4064). ACM.

    Google Scholar 

  • Chilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013). Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 1999–2008). ACM, New York, NY, USA.

    Google Scholar 

  • Creative and Cultural Skills. (2017). Building a creative nation: Current and future skills need. https://ccskills.org.uk/downloads/Building_a_Creative_Nation_-_Current_and_Future_Skills_Needs.pdf.

  • Edmondson, A. C., Bohmer, R. M., & Pisano, G. P. (2001). Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly, 46(4), 685–716.

    Article  Google Scholar 

  • Goel, G., Nikzad, A., & Singla, A. (2014). Allocating tasks to workers with matching constraints: Truthful mechanisms for crowdsourcing markets. In Proceedings of the Companion Publication of the 23rd International Conference on World Wide Web Companion (WWW Companion ’14) (pp. 279–280). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.

    Google Scholar 

  • Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016, February). The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 134–147). ACM.

    Google Scholar 

  • Grier, D. A. (2013). Crowdsourcing for dummies. Wiley.

    Google Scholar 

  • Haas, D., Ansel, J., Gu, L., & Marcus, A. (2015). Argonaut: Macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12), 1642–1653.

    Article  Google Scholar 

  • Ho, C. J., & Vaughan, J. W. (2012, July). Online task assignment in crowdsourcing markets. In Twenty-sixth AAAI Conference on Artificial Intelligence.

    Google Scholar 

  • Huang, S., & Holden, D. (2016). The R&D boundaries of the firm: A problem solving perspective. International Journal of the Economics of Business, 23(3), 287–317.

    Google Scholar 

  • Ipeirotis, P. G., & Gabrilovich, E. (2014, April). Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd International Conference on World Wide Web (pp. 143–154). ACM.

    Google Scholar 

  • Irani, L. C., & Silberman, M. (2013, April). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 611–620). ACM.

    Google Scholar 

  • Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11.

    Google Scholar 

  • Kim, J., Cheng, J., Bernstein, & M.S. (2014) Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 745–755). CSCW ’14, ACM, New York, NY, USA. https://doi.org/10.1145/2531602.2531638, http://doi.acm.org/10.1145/2531602.2531638.

  • Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08) (pp. 453–456). ACM, New York, NY, USA

    Google Scholar 

  • Lawler, E. E., & Worley, C. G. (2006). Designing organizations that are built to change. MIT Sloan Management Review, 48(1), 19–23.

    Google Scholar 

  • Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016). Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2296–2319.

    Article  Google Scholar 

  • Little, G. Chilton, L. B., Goldman, M., & Miller, R. C. (2010). Exploring Iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10). ACM, New York, NY, USA, 68–76.

    Google Scholar 

  • Machado, L., Pereira, G., Prikladnicki, R., Carmel, E., & de Souza, C. R. (2014, November). Crowdsourcing in the Brazilian IT industry: What we know and what we don’t know. In Proceedings of the 1st International Workshop on Crowd-based Software Development Methods and Technologies (pp. 7–12). ACM.

    Google Scholar 

  • Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014, February). Being a turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 224–235). ACM.

    Google Scholar 

  • Mason, W., & Watts, D. J. (2009, June). Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 77–85). ACM.

    Google Scholar 

  • Mavridis, P., Gross-Amblard, D., & Miklós, Z. (2016, April). Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In Proceedings of the 25th International Conference on World Wide Web (pp. 843–853). International World Wide Web Conferences Steering Committee.

    Google Scholar 

  • Majchrzak, A., & Malhotra, A. (2013). Towards an information systems perspective and research agenda on crowdsourcing for innovation. The Journal of Strategic Information Systems, 22(4), 257–268.

    Google Scholar 

  • Morris, M. R., Bigham, J. P., Brewer, R., Bragg, J., Kulkarni, A., Li, J., & Savage, S. (2017, May). Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1867–1876). ACM.

    Google Scholar 

  • Musthag, M., & Ganesan, D. (2013). Labor dynamics in a mobile micro-task market. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 641–650). ACM, New York, NY, USA.

    Google Scholar 

  • Nickerson, J. A., & Zenger, T. R. (2004). A knowledge-based theory of the firm–the problem-solving perspective. Organization Science, 15(6), 617–632.

    Google Scholar 

  • Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., … & Bernstein, M. S. (2014, October). Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM Symposium on User Interface Software and Technology (pp. 75–85). ACM.

    Google Scholar 

  • Retelny, D., Bernstein, M. S., & Valentine, M. A. (2017). No workflow can ever be enough: How crowdsourcing workflows constrain complex work. In Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 89.

    Google Scholar 

  • Salehi, N., & Bernstein, M. S. (2018). Hive: Collective design through network rotation. In Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 151.

    Google Scholar 

  • Schmitz, H., & Lykourentzou, I. (2018). Online sequencing of non-decomposable macrotasks in expert crowdsourcing. ACM Transactions on Social Computing, 1(1), 1.

    Article  Google Scholar 

  • Sieg, J. H, Wallin, M. W., & Von Krogh, G. (2010). Managerial challenges in open innovation: a study of innovation intermediation in the chemical industry. R&D Management, 40(3), 281–291.

    Google Scholar 

  • Teevan, J., Iqbal, S. T., Cai, C. J., Bigham, J. P., Bernstein, M. S., & Gerber, E. M. (2016). Productivity decomposed: Getting big things done with microtasks. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16) (pp. 3500–3507). ACM, New York, NY, USA.

    Google Scholar 

  • Teodoro, R., Ozturk, P., Naaman, M., Mason, W., & Lindqvist, J. (2014, February). The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 236–247). ACM.

    Google Scholar 

  • Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017, May). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 3523–3537). ACM.

    Google Scholar 

  • Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR). In A practical guide, 1st edn. Cham: Springer.

    Google Scholar 

  • Vondrick, C., Patterson, D., & Ramanan, D. (2013). Efficiently scaling up crowdsourced video annotation. International Journal of Computer Vision, 101(1), 184–204.

    Article  Google Scholar 

  • Wagner, T. (2014). The global achievement gap: Why even our best schools don’t teach the new survival skills our children need—and what we can do about it. Basic Books.

    Google Scholar 

  • Wood, A. J., Graham, M., Lehdonvirta, V., & Hjorth, I. (2019). Networked but commodified: The (Dis) embeddedness of digital labour in the gig economy. Sociology, 0038038519828906.

    Google Scholar 

  • Xie, H., & Lui, J. C. (2018). Incentive mechanism and rating system design for crowdsourcing systems: Analysis, tradeoffs and inference. IEEE Transactions on Services Computing, 11(1), 90–102.

    Article  Google Scholar 

  • Zheng, H., Li, D., & Hou, W. (2011). Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce, 15(4), 57–88.

    Article  Google Scholar 

  • Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12) (pp. 217–226). ACM, New York, NY, USA.

    Google Scholar 

  • Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (HLT ’11) (pp. 1220–1229). Association for Computational Linguistics, Stroudsburg, PA, USA.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information and Computing Sciences, Utrecht University, Princetonplein 5, De Uithof, 3584 CC, Utrecht, The Netherlands

    Ioanna Lykourentzou

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Vassillis-Javed Khan & Panos Markopoulos

  3. Xi’an Jiaotong-Liverpool University, Suzhou, China

    Konstantinos Papangelis

Authors
  1. Ioanna Lykourentzou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vassillis-Javed Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Konstantinos Papangelis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Panos Markopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioanna Lykourentzou .

Editor information

Editors and Affiliations

  1. Eindhoven University of Technology, Eindhoven, The Netherlands

    Vassillis-Javed Khan

  2. Xi’an Jiaotong-Liverpool University, Suzhou, China

    Konstantinos Papangelis

  3. Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

    Ioanna Lykourentzou

  4. Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands

    Panos Markopoulos

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Lykourentzou, I., Khan, VJ., Papangelis, K., Markopoulos, P. (2019). Macrotask Crowdsourcing: An Integrated Definition. In: Khan, VJ., Papangelis, K., Lykourentzou, I., Markopoulos, P. (eds) Macrotask Crowdsourcing. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-12334-5_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-12334-5_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-12333-8

  • Online ISBN: 978-3-030-12334-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation