Skip to main content
SpringerLink
Log in

A Decision Tool for Business Process Crowdsourcing: Ontology, Design, and Evaluation

  • Published:
Group Decision and Negotiation Aims and scope Submit manuscript

Abstract

As the crowdsourcing strategy becomes better known, the managerial decisions necessary to establish it as a viable business process are becoming increasingly important. However, a divide and conquer approach, currently dominant in the field, leads to scattered decision support for the crowdsourcing processes. We propose an ontology-based decision tool that supports the whole business process crowdsourcing. The advantage of the ontology approach is that it collects and consolidates knowledge from the existing literature to provide a solid knowledge base for the tool construction. Operationalising the ontology, the tool helps make the decision to crowdsource or not, and choose appropriate design alternatives for the crowdsourcing process. We evaluated the tool through a controlled experiment with 190 participants. The obtained results show that the tool is useful by significantly increasing: (1) the performance in making the decision to crowdsource or not, and (2) the design of crowdsourcing processes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Afuah A, Tucci CL (2012) Crowdsourcing as a solution to distant search. Acad Manag Rev 37:355–375

    Article  Google Scholar 

  • Allahbakhsh M, Benatallah B, Ignjatovic A, Motahari-Nezhad HR, Bertino E, Dustdar S (2013) Quality control in crowdsourcing systems: issues and directions. IEEE Intern Comput 17:76–81

    Article  Google Scholar 

  • Amailef K, Lu J (2013) Ontology-supported case-based reasoning approach for intelligent m-Government emergency response services. Decis Support Syst 55:79–97

    Article  Google Scholar 

  • Amrollahi A (2015) A process model for crowdsourcing: insights from the literature on implementation. In: Proceedings of the Australasian conference on information systems 2015 Paper 18

  • Anderson DR, Sweeney DJ, Williams TA (2011) Statistics for business and economics. Cengage Learning, South-Westerb

    Google Scholar 

  • Brank J, Grobelnik M, Mladenić D (2005) A survey of ontology evaluation techniques. In: Proceedings of the conference on data mining and data warehouses (SiKDD), Ljubljana, Slovenia

  • Corcho O, López MF, Gómez-Pérez A (2003) Methodologies, tools and languages for building ontologies. Where is their meeting point? Data Knowl Eng 46:41–64

    Article  Google Scholar 

  • Delir Haghighi P, Burstein F, Zaslavsky A, Arbon P (2013) Development and evaluation of ontology for intelligent decision support in medical emergency management for mass gatherings. Decis Support Syst 54:1192–1204

    Article  Google Scholar 

  • Dennis AR, Valacich JS (2001) Conducting experimental research in information systems. Commun Assoc Inf Syst 7:5

    Google Scholar 

  • Djelassi S, Decoopman I (2013) Customers’ participation in product development through crowdsourcing: issues and implications. Ind Mark Manag 42:683–692

    Article  Google Scholar 

  • Geiger D (2016) Personalized task recommendation in crowdsourcing systems. Progress in IS, Springer, Berlin

    Book  Google Scholar 

  • Geiger D, Schader M (2014) Personalized task recommendation in crowdsourcing information systems–current state of the art. Decis Support Syst 65:3–16

    Article  Google Scholar 

  • Geiger D, Seedorf S, Schulze T, Nickerson RC, Schader M (2011) Managing the crowd: towards a taxonomy of crowdsourcing processes. In: Proceedings of the seventeenth Americas conference on information systems: Paper 430

  • Gregor S, Hevner AR (2013) Positioning and presenting design science research for maximum impact. MIS Q 37:337–355

    Article  Google Scholar 

  • Guarino N, Oberle D, Staab S (2009) What is an ontology? In: Staab S, Studer R (eds) Handbook on ontologies. Springer, Berlin, pp 1–17

    Google Scholar 

  • Hetmank L (2013) Components and functions of crowdsourcing systems—A systematic literature review. In: Paper presented at the 11th international conference on Wirtschaftsinformatik, Leipzig, Germany

  • Hetmank L (2014) Developing an ontology for enterprise crowdsourcing. In: Paper presented at the Multikonferenz Wirtschaftsinformatik, Paderborn

  • Hevner A, Chatterjee S (2010) Design research in information systems: theory and practice. integrated series in information systems, vol 22. Springer, Berlin

  • Hevner A, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28:75–105

    Article  Google Scholar 

  • Hossfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran-Gia P (2014) Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans Multimed 16:541–558

    Article  Google Scholar 

  • Howe J (2006) The rise of crowdsourcing, vol 14. Dorsey Press, Madison

    Google Scholar 

  • Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing—a study on mechanical turk. In: Proceedings of the seventeenth Americas conference on information systems, Detroit, MI, Paper 340

  • Khazankin R, Satzger B, Dustdar S (2012) Optimized execution of business processes on crowdsourcing platforms. In: IEEE 8th international conference on collaborative computing: networking, applications and worksharing, pp. 443–451

  • Kitchenham B (2007) Guidelines for performing systematic literature reviews in software engineering. Ver 23 EBSE Technical Report

  • Kittur A, Nickerson J, Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J (2013) The future of crowd work. In: Proceedings of the 2013 conference on computer supported cooperative work

  • Kucherbaev P, Daniel F, Tranquillini S, Marchese M (2016) Crowdsourcing processes: a survey of approaches and opportunities. IEEE Intern Comput 2:50–56

    Article  Google Scholar 

  • La Vecchia G, Cisternino A (2010) Collaborative workforce, business process crowdsourcing as an alternative of BPO. In: Daniel F, Facca FM (eds) ICWE2010. LNCS, vol 6385. LNCS. Springer, Berlin, pp 425–430

  • Little G, Chilton LB, Goldman M, Miller RC (2010) Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp 57–66

  • Liu S, Zaraté P (2014) Knowledge based decision support systems: a survey on technologies and application domains. Joint international conference on group decision and negotiation. Springer, Berlin, pp 62–72

    Google Scholar 

  • Lüttgens D, Pollok P, Antons D, Piller F (2014) Wisdom of the crowd and capabilities of a few: internal success factors of crowdsourcing for innovation. J Bus Econ 84:339–374

    Article  Google Scholar 

  • Luz N, Silva N, Novais P (2015) A survey of task-oriented crowdsourcing. Artif Intell Rev 44:187–213

    Article  Google Scholar 

  • March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15:251–266

    Article  Google Scholar 

  • Mendling J, Strembeck M, Recker J (2012) Factors of process model comprehension-findings from a series of experiments. Decis Support Syst 53:195–206

    Article  Google Scholar 

  • Miah SJ, Kerr D, von Hellens L (2014) A collective artefact design of decision support systems: design science research perspective. Inf Technol People 27:259–279

    Article  Google Scholar 

  • Mo J, Sarkar S, Menon S (2015) Making task recommendations in crowdsourcing contests. In: PACIS 2015 proceedings AISeL Paper 132

  • Montgomery CD (2012) Design and analysis of experiments, 8th edn. Willey, New York

    Google Scholar 

  • Moral C, De Antonio A, Ferre X (2017) A visual UML-based conceptual model of information-seeking by computer science researchers. Inf Process Manag 53:963–988

    Article  Google Scholar 

  • Morente-Molinera JA, Wikström R, Herrera-Viedma E, Carlsson C (2016) A linguistic mobile decision support system based on fuzzy ontology to facilitate knowledge mobilization. Decis Support Syst 81:66–75

    Article  Google Scholar 

  • Muhdi L, Daiber M, Friesike S, Boutellier R (2011) The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. Int J Entrep Innov Manag 14:315–332

    Google Scholar 

  • Okoli C (2015) A guide to conducting a standalone systematic literature review. Communications of the Association for Information Systems 37, Article 43

  • Paré G, Trudel M-C, Jaana M, Kitsiou S (2015) Synthesizing information systems knowledge: a typology of literature reviews. Inf Manag 52:183–199

    Article  Google Scholar 

  • Pedersen J, Kocsis D, Tripathi A, Tarrell A, Weerakoon A, Tahmasbi N, Jie X, Wei D, Onook O, De Vreede GJ (2013) Conceptual foundations of crowdsourcing: a review of IS research. In: 46th Hawaii international conference on system sciences (HICSS), pp 579–588

  • Prokesch T, Wohlenberg H (2014) Results from A group wisdom supporting system. In: Proceedings of the European conference on information systems (ECIS) 2014: Paper 7

  • Rouse AC (2010) A preliminary taxonomy of crowdsourcing. In: Proceedings of the 21st Australasian conference on information systems, Paper 76

  • Rowe M, Poblet M, Thomson JD (2015) Creating value through crowdsourcing: the antecedent conditions. International conference on group decision and negotiation. Springer, Berlin, pp 345–355

    Google Scholar 

  • Sánchez D, Moreno A (2008) Learning non-taxonomic relationships from web documents for domain ontology construction. Data Knowl Eng 64:600–623

    Article  Google Scholar 

  • Satzger B, Psaier H, Schall D, Dustdar S (2011) Stimulating skill evolution in market-based crowdsourcing. In: Rinderle-Ma S, Toumani F, Wolf K (eds) BPM 2011. LNCS, vol 6896. Lecture notes in computer science. Springer, Berlin, pp 66–82

  • Schenk E, Guittard C (2009) Crowdsourcing: what can be outsourced to the crowd, and why? In: Workshop on open source innovation, Strasbourg, France

  • Schenk E, Guittard C (2011) Towards a characterization of crowdsourcing practices. J Innov Econ 7:93–107

    Article  Google Scholar 

  • Schnitzer S, Neitzel S, Schmidt S, Rensing C (2016) Perceived Task Similarities for Task Recommendation in Crowdsourcing Systems. In: Proceedings of the 25th international conference companion on world wide web. International World Wide Web Conferences Steering Committee, pp 585–590

  • Seeber I, Merz A, De Vreede G-J, Maier R, Weber B (2017) Convergence on self-generated vs. crowdsourced ideas in crisis response: comparing social exchange processes and satisfaction with process. In: Proceedings of the 50th Hawaii international conference on system sciences, pp 687–696

  • Shanks G, Tansley E, Weber R (2003) Using ontology to validate conceptual models. Commun ACM 46:85–89

    Article  Google Scholar 

  • Sjøberg DI, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg N-K, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31:733–753

    Article  Google Scholar 

  • Soh C, Markus ML, Goh KH (2006) Electronic marketplaces and price transparency: strategy, information technology, and success. MIS Q 705–723

  • Thuan NH, Antunes P, Johnstone D, Ha XS (2015) Building an enterprise ontology of business process crowdsourcing: a design science approach. In: PACIS 2015 proceedings AISeL Paper 112

  • Thuan NH, Antunes P, Johnstone D (2016) Factors influencing the decision to crowdsource: a systematic literature review. Inf Syst Front 18:47–68

    Article  Google Scholar 

  • Thuan NH, Antunes P,  Johnstone D (2017) A process model for establishing business process crowdsourcing. Australas J Inf Syst 21:1–21

  • Tokarchuk O, Cuel R, Zamarian M (2012) Analyzing crowd labor and designing incentives for humans in the loop. IEEE Intern Comput Mag 16:45–51

    Article  Google Scholar 

  • Tranquillini S, Daniel F, Kucherbaev P, Casati F (2015) Modeling, enacting, and integrating custom crowdsourcing processes. ACM Trans Web (TWEB) 9:7

    Google Scholar 

  • Uschold M, King M (1995) Towards a methodology for building ontologies. In: IJCAI95 workshop on basic ontological issues in knowledge sharing, Montreal

  • Vukovic M (2009) Crowdsourcing for enterprises. 2009 World conference on services-I. Los Angeles, CA. IEEE, pp 686–692

    Google Scholar 

  • Vukovic M, Laredo J, Rajagopal S (2010) Challenges and experiences in deploying enterprise crowdsourcing service. In: Benatallah B, Casati F, Kappel G, Rossi G (eds) ICWE 2010. LNCS, vol 6189. LNCS. Springer, Berlin, pp 460–467

  • Yuen M-C, King I, Leung K-S (2015) Taskrec: a task recommendation framework in crowdsourcing systems. Neural Process Lett 41:223–238

    Article  Google Scholar 

  • Zhao Y, Zhu Q (2014) Evaluation on crowdsourcing research: current status and future direction. Inf Syst Front 16:417–434

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Can Tho University of Technology, 256 Nguyen Van Cu Street, Can Tho City, Vietnam

    Nguyen Hoang Thuan

  2. School of Information Management, Victoria University of Wellington, PO Box 600, Wellington, 6140, New Zealand

    Pedro Antunes & David Johnstone

Authors
  1. Nguyen Hoang Thuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro Antunes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Johnstone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nguyen Hoang Thuan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendix A

Appendix A

figure a
figure b
figure c

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thuan, N.H., Antunes, P. & Johnstone, D. A Decision Tool for Business Process Crowdsourcing: Ontology, Design, and Evaluation. Group Decis Negot 27, 285–312 (2018). https://doi.org/10.1007/s10726-018-9557-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10726-018-9557-y

Keywords

Search

Navigation