Advertisement

Macrotask Crowdsourcing: An Integrated Definition

  • Ioanna LykourentzouEmail author
  • Vassillis-Javed Khan
  • Konstantinos Papangelis
  • Panos Markopoulos
Chapter
Part of the Human–Computer Interaction Series book series (HCIS)

Abstract

The conceptual distinction between microtasks and macrotasks has been made relatively early on in the crowdsourcing literature. However, only recently a handful of research works has explored it explicitly. These works, for the most part, have focused on simply discussing macrotasks within the confines of their own work (e.g., in terms of creativity), without taking into account the multiple facets that working with such tasks involves. This has resulted in the term “macrotask” to be severely convoluted and largely meaning different things to different individuals. More importantly, it has resulted in disregarding macrotask crowdsourcing as a new labor model of its own right. To address this scholarly gap, in this paper we discuss macrotask crowdsourcing from a multitude of dimensions, namely the nature of the problem it can solve, the crowdworker skills it involves, and the work management structures it necessitates. In view of our analysis, we provide a first integrated definition of macrotask crowdsourcing.

References

  1. Altshuller, G. (2005). The innovation algorithm: TRIZ, systematic innovation and technical creativity. Worchesrter, MA: Technical Innovation Center.Google Scholar
  2. Ambati, V., Vogel, S., & Carbonell, J. (2012). Collaborative workflow for crowdsourcing translation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12) (pp. 1191–1194). ACM, New York, NY, USA.Google Scholar
  3. Basu Roy, S., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2015). Task assignment optimization in knowledge-intensive crowdsourcing. The VLDB Journal—The International Journal on Very Large Data Bases, 24(4), 467–491.Google Scholar
  4. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D. & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST ’10) (pp. 313–322). ACM, New York, NY, USA.Google Scholar
  5. Chan, J., Dang, S., & Dow, S. P. (2016). Improving crowd innovation with expert facilitation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16) (pp. 1223–1235). ACM, New York, NY, USA.Google Scholar
  6. Cheng, J., Teevan, J., Iqbal, S. T., & Bernstein, M. S. (2015, April). Break it down: A comparison of macro-and microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4061–4064). ACM.Google Scholar
  7. Chilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013). Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 1999–2008). ACM, New York, NY, USA.Google Scholar
  8. Creative and Cultural Skills. (2017). Building a creative nation: Current and future skills need. https://ccskills.org.uk/downloads/Building_a_Creative_Nation_-_Current_and_Future_Skills_Needs.pdf.
  9. Edmondson, A. C., Bohmer, R. M., & Pisano, G. P. (2001). Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly, 46(4), 685–716.CrossRefGoogle Scholar
  10. Goel, G., Nikzad, A., & Singla, A. (2014). Allocating tasks to workers with matching constraints: Truthful mechanisms for crowdsourcing markets. In Proceedings of the Companion Publication of the 23rd International Conference on World Wide Web Companion (WWW Companion ’14) (pp. 279–280). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.Google Scholar
  11. Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016, February). The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 134–147). ACM.Google Scholar
  12. Grier, D. A. (2013). Crowdsourcing for dummies. Wiley.Google Scholar
  13. Haas, D., Ansel, J., Gu, L., & Marcus, A. (2015). Argonaut: Macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12), 1642–1653.CrossRefGoogle Scholar
  14. Ho, C. J., & Vaughan, J. W. (2012, July). Online task assignment in crowdsourcing markets. In Twenty-sixth AAAI Conference on Artificial Intelligence.Google Scholar
  15. Huang, S., & Holden, D. (2016). The R&D boundaries of the firm: A problem solving perspective. International Journal of the Economics of Business, 23(3), 287–317.Google Scholar
  16. Ipeirotis, P. G., & Gabrilovich, E. (2014, April). Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd International Conference on World Wide Web (pp. 143–154). ACM.Google Scholar
  17. Irani, L. C., & Silberman, M. (2013, April). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 611–620). ACM.Google Scholar
  18. Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11.Google Scholar
  19. Kim, J., Cheng, J., Bernstein, & M.S. (2014) Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 745–755). CSCW ’14, ACM, New York, NY, USA.  https://doi.org/10.1145/2531602.2531638, http://doi.acm.org/10.1145/2531602.2531638.
  20. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08) (pp. 453–456). ACM, New York, NY, USAGoogle Scholar
  21. Lawler, E. E., & Worley, C. G. (2006). Designing organizations that are built to change. MIT Sloan Management Review, 48(1), 19–23.Google Scholar
  22. Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016). Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2296–2319.CrossRefGoogle Scholar
  23. Little, G. Chilton, L. B., Goldman, M., & Miller, R. C. (2010). Exploring Iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10). ACM, New York, NY, USA, 68–76.Google Scholar
  24. Machado, L., Pereira, G., Prikladnicki, R., Carmel, E., & de Souza, C. R. (2014, November). Crowdsourcing in the Brazilian IT industry: What we know and what we don’t know. In Proceedings of the 1st International Workshop on Crowd-based Software Development Methods and Technologies (pp. 7–12). ACM.Google Scholar
  25. Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014, February). Being a turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 224–235). ACM.Google Scholar
  26. Mason, W., & Watts, D. J. (2009, June). Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 77–85). ACM.Google Scholar
  27. Mavridis, P., Gross-Amblard, D., & Miklós, Z. (2016, April). Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In Proceedings of the 25th International Conference on World Wide Web (pp. 843–853). International World Wide Web Conferences Steering Committee.Google Scholar
  28. Majchrzak, A., & Malhotra, A. (2013). Towards an information systems perspective and research agenda on crowdsourcing for innovation. The Journal of Strategic Information Systems, 22(4), 257–268.Google Scholar
  29. Morris, M. R., Bigham, J. P., Brewer, R., Bragg, J., Kulkarni, A., Li, J., & Savage, S. (2017, May). Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1867–1876). ACM.Google Scholar
  30. Musthag, M., & Ganesan, D. (2013). Labor dynamics in a mobile micro-task market. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 641–650). ACM, New York, NY, USA.Google Scholar
  31. Nickerson, J. A., & Zenger, T. R. (2004). A knowledge-based theory of the firm–the problem-solving perspective. Organization Science, 15(6), 617–632.Google Scholar
  32. Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., … & Bernstein, M. S. (2014, October). Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM Symposium on User Interface Software and Technology (pp. 75–85). ACM.Google Scholar
  33. Retelny, D., Bernstein, M. S., & Valentine, M. A. (2017). No workflow can ever be enough: How crowdsourcing workflows constrain complex work. In Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 89.Google Scholar
  34. Salehi, N., & Bernstein, M. S. (2018). Hive: Collective design through network rotation. In Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 151.Google Scholar
  35. Schmitz, H., & Lykourentzou, I. (2018). Online sequencing of non-decomposable macrotasks in expert crowdsourcing. ACM Transactions on Social Computing, 1(1), 1.CrossRefGoogle Scholar
  36. Sieg, J. H, Wallin, M. W., & Von Krogh, G. (2010). Managerial challenges in open innovation: a study of innovation intermediation in the chemical industry. R&D Management, 40(3), 281–291.Google Scholar
  37. Teevan, J., Iqbal, S. T., Cai, C. J., Bigham, J. P., Bernstein, M. S., & Gerber, E. M. (2016). Productivity decomposed: Getting big things done with microtasks. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16) (pp. 3500–3507). ACM, New York, NY, USA.Google Scholar
  38. Teodoro, R., Ozturk, P., Naaman, M., Mason, W., & Lindqvist, J. (2014, February). The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 236–247). ACM.Google Scholar
  39. Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017, May). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 3523–3537). ACM.Google Scholar
  40. Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR). In A practical guide, 1st edn. Cham: Springer.Google Scholar
  41. Vondrick, C., Patterson, D., & Ramanan, D. (2013). Efficiently scaling up crowdsourced video annotation. International Journal of Computer Vision, 101(1), 184–204.CrossRefGoogle Scholar
  42. Wagner, T. (2014). The global achievement gap: Why even our best schools don’t teach the new survival skills our children need—and what we can do about it. Basic Books.Google Scholar
  43. Wood, A. J., Graham, M., Lehdonvirta, V., & Hjorth, I. (2019). Networked but commodified: The (Dis) embeddedness of digital labour in the gig economy. Sociology, 0038038519828906.Google Scholar
  44. Xie, H., & Lui, J. C. (2018). Incentive mechanism and rating system design for crowdsourcing systems: Analysis, tradeoffs and inference. IEEE Transactions on Services Computing, 11(1), 90–102.CrossRefGoogle Scholar
  45. Zheng, H., Li, D., & Hou, W. (2011). Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce, 15(4), 57–88.CrossRefGoogle Scholar
  46. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12) (pp. 217–226). ACM, New York, NY, USA.Google Scholar
  47. Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (HLT ’11) (pp. 1220–1229). Association for Computational Linguistics, Stroudsburg, PA, USA.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ioanna Lykourentzou
    • 1
    Email author
  • Vassillis-Javed Khan
    • 2
  • Konstantinos Papangelis
    • 3
  • Panos Markopoulos
    • 2
  1. 1.Department of Information and Computing SciencesUtrecht UniversityUtrechtThe Netherlands
  2. 2.Eindhoven University of TechnologyEindhovenThe Netherlands
  3. 3.Xi’an Jiaotong-Liverpool UniversitySuzhouChina

Personalised recommendations