SmartCrowd: A Workflow Framework for Complex Crowdsourcing Tasks

  • Tianhong Xiong
  • Yang YuEmail author
  • Maolin Pan
  • Jing Yang
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 342)


Over the past decade, a number of frameworks have been introduced to support different crowdsourcing tasks. However, complex creative tasks have remained out of reach for workflow modeling. Unlike typical tasks, creative tasks are often interdependent, requiring human cognitive ability and team collaboration. The crowd workers are required not only to perform typical tasks, but also to participate in the analysis and manipulation of complex tasks, hence the number and execution order of tasks are unknown until runtime. Thus, it is difficult to model this kind of complex tasks by using existing workflow approaches. Therefore, we propose a workflow modeling approach based on state machine to design crowdsourcing model that can be translated into SCXML code and executed by an open source engine. This approach and engine are embodied in SmartCrowd. Through two evaluations, we found that SmartCrowd can provide support for complex crowdsourcing tasks, especially on creative tasks. Moreover, we introduce a set of basic design patterns, and by employing them to compose complex patterns, our framework can support more crowdsourcing research.


Crowdsourcing State machine Workflow Complex tasks Creative tasks Design patterns 



This work was supported by the National Key Research and Development Program of China under Grant No. 2017YFB0202200; the National Natural Science Foundation of China under Grant No. 61572539; the Research Foundation of Science and Technology Major Project in Guangdong Province under Grant Nos. 2015B010106007, 2016B010110003; the Research Foundation of Science and Technology Plan Project in Guangdong Province under Grant No. 2016B050502006.


  1. 1.
    Saxton, G.D., Oh, O., Kishore, R.: Rules of crowdsourcing: models, issues, and systems of control. Inf. Syst. Manag. 30(1), 2–20 (2013)CrossRefGoogle Scholar
  2. 2.
    Amazon Mechanical Turk. Accessed 10 May 2018
  3. 3.
    Kim, J., Sterman, S., Cohen, A.A.B., Bernstein, M.S.: Mechanical novel: crowdsourcing complex work through reflection and revision. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 233–245. ACM (2017)Google Scholar
  4. 4.
    Orlowska, M.E.: Challenges for workflows technology to support crowdsourcing activities. In: International Conference on Business Intelligence and Technology, pp. 88–92. Bustech (2015)Google Scholar
  5. 5.
    Kim, J., Cheng, J., Bernstein, M.S.: Ensemble: exploring complementary strengths of leaders and crowds in creative collaboration. In: Proceedings of CSCW, pp. 745–755. ACM, New York (2014)Google Scholar
  6. 6.
    Malone, T.W., Laubacher, R., Dellarocas, C.: The collective intelligence genome. MIT Sloan Manag. Rev. 51(3), 21 (2010)Google Scholar
  7. 7.
    Harel, D.: Statecharts: a visual formalism for complex systems. Sci. Comput. Program. 8(3), 231–274 (1987)MathSciNetCrossRefGoogle Scholar
  8. 8.
    SCXML. Accessed 10 June 2018
  9. 9.
    Apache Commons SCXML. Accessed 15 June 2018
  10. 10.
    Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: tools for iterative tasks on mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 29–30. ACM (2009)Google Scholar
  11. 11.
    Barowy, D.W., Curtsinger, C., Berger, E.D., McGregor, A.: Automan: a platform for integrating human-based and digital computation. ACM Sigplan Not. 47(10), 639–654 (2012)CrossRefGoogle Scholar
  12. 12.
    Franklin, M.J., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: CrowdDB: answering queries with crowdsourcing. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, pp. 61–72. ACM (2011)Google Scholar
  13. 13.
    Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM (2011)Google Scholar
  14. 14.
    Ahmad, S., Battle, A., Malkani, Z., Kamvar, S.: The Jabberwocky programming environment for structured social computing. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 53–64. ACM (2011)Google Scholar
  15. 15.
    Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of CSCW, CSCW 2012, pp. 1003–1012. ACM, New York (2012)Google Scholar
  16. 16.
    Chittilappilly, A.I., Chen, L., Amer-Yahia, S.: A survey of general-purpose crowdsourcing techniques. IEEE Trans. Knowl. Data Eng. 28(9), 2246–2266 (2016)CrossRefGoogle Scholar
  17. 17.
    Minder, P., Bernstein, A.: CrowdLang: a programming language for the systematic exploration of human computation systems. In: Aberer, K., Flache, A., Jager, W., Liu, L., Tang, J., Guéret, C. (eds.) SocInfo 2012. LNCS, vol. 7710, pp. 124–137. Springer, Heidelberg (2012). Scholar
  18. 18.
    Bozzon, A., Brambilla, M., Ceri, S.: Answering search queries with CrowdSearcher. In: Proceedings of the 21st International Conference on World Wide Web, pp. 1009–1018. ACM (2012)Google Scholar
  19. 19.
    Bozzon, A., Brambilla, M., Ceri, S., Mauri, A., Volonterio, R.: Designing complex crowdsourcing applications covering multiple platforms and tasks. J. Web Eng. 14(5–6), 443–473 (2015)Google Scholar
  20. 20.
    Tranquillini, S., Daniel, F., Kucherbaev, P., Casati, F.: Modeling, enacting, and integrating custom crowdsourcing processes. ACM Trans. Web (TWEB) 9(2), 7 (2015)Google Scholar
  21. 21.
    Object Management Group: Business Process Model and Notation (BPMN) Version 2.0 (2011). Accessed 15 Mar 2018
  22. 22.
  23. 23.
    Chen, J.J., Menezes, N.J., Bradley, A.D., North, T.: Opportunities for CrowdSourcing research on amazon mechanical turk. Interfaces 5(3), (2011)Google Scholar
  24. 24.
    Lofi, C., El Maarry, K.: Design patterns for hybrid algorithmic-crowdsourcing workflows. In: 2014 IEEE 16th Conference on Business Informatics (CBI), vol. 1, pp. 1–8. IEEE (2014)Google Scholar
  25. 25.
    Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1403–1412. DBLP (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tianhong Xiong
    • 1
  • Yang Yu
    • 1
    Email author
  • Maolin Pan
    • 1
  • Jing Yang
    • 1
  1. 1.Sun Yat-sen UniversityGuangzhouChina

Personalised recommendations