Group Decision and Negotiation

, Volume 27, Issue 2, pp 285–312 | Cite as

A Decision Tool for Business Process Crowdsourcing: Ontology, Design, and Evaluation

  • Nguyen Hoang ThuanEmail author
  • Pedro Antunes
  • David Johnstone


As the crowdsourcing strategy becomes better known, the managerial decisions necessary to establish it as a viable business process are becoming increasingly important. However, a divide and conquer approach, currently dominant in the field, leads to scattered decision support for the crowdsourcing processes. We propose an ontology-based decision tool that supports the whole business process crowdsourcing. The advantage of the ontology approach is that it collects and consolidates knowledge from the existing literature to provide a solid knowledge base for the tool construction. Operationalising the ontology, the tool helps make the decision to crowdsource or not, and choose appropriate design alternatives for the crowdsourcing process. We evaluated the tool through a controlled experiment with 190 participants. The obtained results show that the tool is useful by significantly increasing: (1) the performance in making the decision to crowdsource or not, and (2) the design of crowdsourcing processes.


Business process crowdsourcing Crowdsourcing Decision support system Experiment Ontology 


Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Afuah A, Tucci CL (2012) Crowdsourcing as a solution to distant search. Acad Manag Rev 37:355–375CrossRefGoogle Scholar
  2. Allahbakhsh M, Benatallah B, Ignjatovic A, Motahari-Nezhad HR, Bertino E, Dustdar S (2013) Quality control in crowdsourcing systems: issues and directions. IEEE Intern Comput 17:76–81CrossRefGoogle Scholar
  3. Amailef K, Lu J (2013) Ontology-supported case-based reasoning approach for intelligent m-Government emergency response services. Decis Support Syst 55:79–97CrossRefGoogle Scholar
  4. Amrollahi A (2015) A process model for crowdsourcing: insights from the literature on implementation. In: Proceedings of the Australasian conference on information systems 2015 Paper 18Google Scholar
  5. Anderson DR, Sweeney DJ, Williams TA (2011) Statistics for business and economics. Cengage Learning, South-WesterbGoogle Scholar
  6. Brank J, Grobelnik M, Mladenić D (2005) A survey of ontology evaluation techniques. In: Proceedings of the conference on data mining and data warehouses (SiKDD), Ljubljana, SloveniaGoogle Scholar
  7. Corcho O, López MF, Gómez-Pérez A (2003) Methodologies, tools and languages for building ontologies. Where is their meeting point? Data Knowl Eng 46:41–64CrossRefGoogle Scholar
  8. Delir Haghighi P, Burstein F, Zaslavsky A, Arbon P (2013) Development and evaluation of ontology for intelligent decision support in medical emergency management for mass gatherings. Decis Support Syst 54:1192–1204CrossRefGoogle Scholar
  9. Dennis AR, Valacich JS (2001) Conducting experimental research in information systems. Commun Assoc Inf Syst 7:5Google Scholar
  10. Djelassi S, Decoopman I (2013) Customers’ participation in product development through crowdsourcing: issues and implications. Ind Mark Manag 42:683–692CrossRefGoogle Scholar
  11. Geiger D (2016) Personalized task recommendation in crowdsourcing systems. Progress in IS, Springer, BerlinCrossRefGoogle Scholar
  12. Geiger D, Schader M (2014) Personalized task recommendation in crowdsourcing information systems–current state of the art. Decis Support Syst 65:3–16CrossRefGoogle Scholar
  13. Geiger D, Seedorf S, Schulze T, Nickerson RC, Schader M (2011) Managing the crowd: towards a taxonomy of crowdsourcing processes. In: Proceedings of the seventeenth Americas conference on information systems: Paper 430Google Scholar
  14. Gregor S, Hevner AR (2013) Positioning and presenting design science research for maximum impact. MIS Q 37:337–355CrossRefGoogle Scholar
  15. Guarino N, Oberle D, Staab S (2009) What is an ontology? In: Staab S, Studer R (eds) Handbook on ontologies. Springer, Berlin, pp 1–17Google Scholar
  16. Hetmank L (2013) Components and functions of crowdsourcing systems—A systematic literature review. In: Paper presented at the 11th international conference on Wirtschaftsinformatik, Leipzig, GermanyGoogle Scholar
  17. Hetmank L (2014) Developing an ontology for enterprise crowdsourcing. In: Paper presented at the Multikonferenz Wirtschaftsinformatik, PaderbornGoogle Scholar
  18. Hevner A, Chatterjee S (2010) Design research in information systems: theory and practice. integrated series in information systems, vol 22. Springer, BerlinGoogle Scholar
  19. Hevner A, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28:75–105CrossRefGoogle Scholar
  20. Hossfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran-Gia P (2014) Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans Multimed 16:541–558CrossRefGoogle Scholar
  21. Howe J (2006) The rise of crowdsourcing, vol 14. Dorsey Press, MadisonGoogle Scholar
  22. Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing—a study on mechanical turk. In: Proceedings of the seventeenth Americas conference on information systems, Detroit, MI, Paper 340Google Scholar
  23. Khazankin R, Satzger B, Dustdar S (2012) Optimized execution of business processes on crowdsourcing platforms. In: IEEE 8th international conference on collaborative computing: networking, applications and worksharing, pp. 443–451Google Scholar
  24. Kitchenham B (2007) Guidelines for performing systematic literature reviews in software engineering. Ver 23 EBSE Technical ReportGoogle Scholar
  25. Kittur A, Nickerson J, Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J (2013) The future of crowd work. In: Proceedings of the 2013 conference on computer supported cooperative workGoogle Scholar
  26. Kucherbaev P, Daniel F, Tranquillini S, Marchese M (2016) Crowdsourcing processes: a survey of approaches and opportunities. IEEE Intern Comput 2:50–56CrossRefGoogle Scholar
  27. La Vecchia G, Cisternino A (2010) Collaborative workforce, business process crowdsourcing as an alternative of BPO. In: Daniel F, Facca FM (eds) ICWE2010. LNCS, vol 6385. LNCS. Springer, Berlin, pp 425–430Google Scholar
  28. Little G, Chilton LB, Goldman M, Miller RC (2010) Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp 57–66Google Scholar
  29. Liu S, Zaraté P (2014) Knowledge based decision support systems: a survey on technologies and application domains. Joint international conference on group decision and negotiation. Springer, Berlin, pp 62–72Google Scholar
  30. Lüttgens D, Pollok P, Antons D, Piller F (2014) Wisdom of the crowd and capabilities of a few: internal success factors of crowdsourcing for innovation. J Bus Econ 84:339–374CrossRefGoogle Scholar
  31. Luz N, Silva N, Novais P (2015) A survey of task-oriented crowdsourcing. Artif Intell Rev 44:187–213CrossRefGoogle Scholar
  32. March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15:251–266CrossRefGoogle Scholar
  33. Mendling J, Strembeck M, Recker J (2012) Factors of process model comprehension-findings from a series of experiments. Decis Support Syst 53:195–206CrossRefGoogle Scholar
  34. Miah SJ, Kerr D, von Hellens L (2014) A collective artefact design of decision support systems: design science research perspective. Inf Technol People 27:259–279CrossRefGoogle Scholar
  35. Mo J, Sarkar S, Menon S (2015) Making task recommendations in crowdsourcing contests. In: PACIS 2015 proceedings AISeL Paper 132Google Scholar
  36. Montgomery CD (2012) Design and analysis of experiments, 8th edn. Willey, New YorkGoogle Scholar
  37. Moral C, De Antonio A, Ferre X (2017) A visual UML-based conceptual model of information-seeking by computer science researchers. Inf Process Manag 53:963–988CrossRefGoogle Scholar
  38. Morente-Molinera JA, Wikström R, Herrera-Viedma E, Carlsson C (2016) A linguistic mobile decision support system based on fuzzy ontology to facilitate knowledge mobilization. Decis Support Syst 81:66–75CrossRefGoogle Scholar
  39. Muhdi L, Daiber M, Friesike S, Boutellier R (2011) The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. Int J Entrep Innov Manag 14:315–332Google Scholar
  40. Okoli C (2015) A guide to conducting a standalone systematic literature review. Communications of the Association for Information Systems 37, Article 43Google Scholar
  41. Paré G, Trudel M-C, Jaana M, Kitsiou S (2015) Synthesizing information systems knowledge: a typology of literature reviews. Inf Manag 52:183–199CrossRefGoogle Scholar
  42. Pedersen J, Kocsis D, Tripathi A, Tarrell A, Weerakoon A, Tahmasbi N, Jie X, Wei D, Onook O, De Vreede GJ (2013) Conceptual foundations of crowdsourcing: a review of IS research. In: 46th Hawaii international conference on system sciences (HICSS), pp 579–588Google Scholar
  43. Prokesch T, Wohlenberg H (2014) Results from A group wisdom supporting system. In: Proceedings of the European conference on information systems (ECIS) 2014: Paper 7Google Scholar
  44. Rouse AC (2010) A preliminary taxonomy of crowdsourcing. In: Proceedings of the 21st Australasian conference on information systems, Paper 76Google Scholar
  45. Rowe M, Poblet M, Thomson JD (2015) Creating value through crowdsourcing: the antecedent conditions. International conference on group decision and negotiation. Springer, Berlin, pp 345–355Google Scholar
  46. Sánchez D, Moreno A (2008) Learning non-taxonomic relationships from web documents for domain ontology construction. Data Knowl Eng 64:600–623CrossRefGoogle Scholar
  47. Satzger B, Psaier H, Schall D, Dustdar S (2011) Stimulating skill evolution in market-based crowdsourcing. In: Rinderle-Ma S, Toumani F, Wolf K (eds) BPM 2011. LNCS, vol 6896. Lecture notes in computer science. Springer, Berlin, pp 66–82Google Scholar
  48. Schenk E, Guittard C (2009) Crowdsourcing: what can be outsourced to the crowd, and why? In: Workshop on open source innovation, Strasbourg, FranceGoogle Scholar
  49. Schenk E, Guittard C (2011) Towards a characterization of crowdsourcing practices. J Innov Econ 7:93–107CrossRefGoogle Scholar
  50. Schnitzer S, Neitzel S, Schmidt S, Rensing C (2016) Perceived Task Similarities for Task Recommendation in Crowdsourcing Systems. In: Proceedings of the 25th international conference companion on world wide web. International World Wide Web Conferences Steering Committee, pp 585–590Google Scholar
  51. Seeber I, Merz A, De Vreede G-J, Maier R, Weber B (2017) Convergence on self-generated vs. crowdsourced ideas in crisis response: comparing social exchange processes and satisfaction with process. In: Proceedings of the 50th Hawaii international conference on system sciences, pp 687–696Google Scholar
  52. Shanks G, Tansley E, Weber R (2003) Using ontology to validate conceptual models. Commun ACM 46:85–89CrossRefGoogle Scholar
  53. Sjøberg DI, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg N-K, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31:733–753CrossRefGoogle Scholar
  54. Soh C, Markus ML, Goh KH (2006) Electronic marketplaces and price transparency: strategy, information technology, and success. MIS Q 705–723Google Scholar
  55. Thuan NH, Antunes P, Johnstone D, Ha XS (2015) Building an enterprise ontology of business process crowdsourcing: a design science approach. In: PACIS 2015 proceedings AISeL Paper 112Google Scholar
  56. Thuan NH, Antunes P, Johnstone D (2016) Factors influencing the decision to crowdsource: a systematic literature review. Inf Syst Front 18:47–68CrossRefGoogle Scholar
  57. Thuan NH, Antunes P,  Johnstone D (2017) A process model for establishing business process crowdsourcing. Australas J Inf Syst 21:1–21Google Scholar
  58. Tokarchuk O, Cuel R, Zamarian M (2012) Analyzing crowd labor and designing incentives for humans in the loop. IEEE Intern Comput Mag 16:45–51CrossRefGoogle Scholar
  59. Tranquillini S, Daniel F, Kucherbaev P, Casati F (2015) Modeling, enacting, and integrating custom crowdsourcing processes. ACM Trans Web (TWEB) 9:7Google Scholar
  60. Uschold M, King M (1995) Towards a methodology for building ontologies. In: IJCAI95 workshop on basic ontological issues in knowledge sharing, MontrealGoogle Scholar
  61. Vukovic M (2009) Crowdsourcing for enterprises. 2009 World conference on services-I. Los Angeles, CA. IEEE, pp 686–692Google Scholar
  62. Vukovic M, Laredo J, Rajagopal S (2010) Challenges and experiences in deploying enterprise crowdsourcing service. In: Benatallah B, Casati F, Kappel G, Rossi G (eds) ICWE 2010. LNCS, vol 6189. LNCS. Springer, Berlin, pp 460–467Google Scholar
  63. Yuen M-C, King I, Leung K-S (2015) Taskrec: a task recommendation framework in crowdsourcing systems. Neural Process Lett 41:223–238CrossRefGoogle Scholar
  64. Zhao Y, Zhu Q (2014) Evaluation on crowdsourcing research: current status and future direction. Inf Syst Front 16:417–434CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  • Nguyen Hoang Thuan
    • 1
    Email author
  • Pedro Antunes
    • 2
  • David Johnstone
    • 2
  1. 1.Faculty of Information TechnologyCan Tho University of TechnologyCan Tho CityVietnam
  2. 2.School of Information ManagementVictoria University of WellingtonWellingtonNew Zealand

Personalised recommendations