Skip to main content
SpringerLink
Log in

Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary

  • Original Paper
  • Published:
Journal of Business Economics Aims and scope Submit manuscript

Abstract

Crowdsourcing has gained much attention in practice over the last years. Numerous companies have drawn on this concept for performing different tasks and value creation activities. Nevertheless, despite its popularity, there is still comparatively little well-founded knowledge on crowdsourcing, particularly with regard to crowdsourcing intermediaries. Crowdsourcing intermediaries play a key role in crowdsourcing initiatives as they assure the connection between the crowdsourcing companies and the crowd. However, the issue of how crowdsourcing intermediaries manage crowdsourcing initiatives and the associated challenges has not been addresses by research yet. We address these issues by conducting a case study with a German start-up crowdsourcing intermediary called testCloud that offers software testing services for companies intending to partly or fully outsource their testing activities to a certain crowd. The case study shows that testCloud faces three main challenges, these are: managing the process, managing the crowd and managing the technology. For each dimension, we outline mechanisms that testCloud applies for facing the challenges associated with crowdsourcing projects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Afuah A, Tucci CL (2012) Crowdsourcing as a solution to distant search. Acad of Manag Rev 37(3):355–375. doi:10.5465/amr.2010.0146

    Article  Google Scholar 

  • Allison TH, Townsend DM (2012) Wisdom of the Crowd? Reputational cascades and emotional contagion in microlender crowdfunding. Paper presented at the 72nd academy of management annual meeting, Boston

  • Bacon DF, Chen Y, Parkes D, Rao M (2009) A market-based approach to software evolution. Paper presented at the proceedings of the 24th ACM SIGPLAN conference

  • Bederson B, Quinn A (2010) Web workers, unite! addressing challenges of online laborers. In: Proceedings of CHI '11 extended abstracts on human factors in computing systems, pp 97–106

  • Bittner E, Leimeister JM (2011) Towards CSR 2.0—potentials and challenges of Web 2.0 for corporate social responsibility communication. In: Proceedings of the 11th European academy of management annual meeting, Tallinn

  • Bjelland OM, Wood RC (2008) An inside view of IBM’s innovation jam. MIT Sloan Manag Rev 50(1):31–40. doi:10.1225/SMR291

    Google Scholar 

  • Blohm I, Köroglu O, Leimeister JM, Krcmar H (2012) Absorptive capacity for open innovation communities—learnings from theory and practice. Paper presented at the academy of management annual meeting

  • Brabham DC (2010) Moving the crowd at threadless: motivations for participation in a crowdsourcing application. Inform Commun Soc 13(8):1122–1145. doi:10.1080/13691181003624090

    Article  Google Scholar 

  • Brabham DC (2012) Crowdsourcing: a model for leveraging online communities. In: Delwiche A, Henderson J (eds) The Routledge handbook of participatory culture. Routledge, London, pp 1–25

  • Brandel M (2008) Crowdsourcing: are you ready to ask the world for answers? Computerworld 42:24–26

    Google Scholar 

  • Bretschneider U (2012) Die Ideen-Community zur Integration von Kunden in den Innovationsprozess: Empirische Analysen und Implikationen. Dissertation. Gabler, Wiesbaden. doi:10.1007/978-3-8349-7173-9

  • Bretschneider U, Leimeister JM (2011) Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen? In: Meißner K, Engelien M (eds) Virtual enterprises, communities and social networks, Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11). Dresden, pp 1–13

  • Bullinger AC, Neyer AK, Rass M, Möslein KM (2010) Community-based innovation contests: where competition meets cooperation. Creativ Innov Manag 19(3):290–303. doi:10.1111/j.1467-8691.2010.00565.x

    Article  Google Scholar 

  • Burger-Helmchen T, Penin J (2010) The limits of crowdsourcing inventive activities: What do transaction cost theory and the evolutionary theories of the firm teach us? Paper presented at the workshop on open source innovation, Strasbourg

  • Burt RS (2005) Brokerage and closure: An introduction to social capital. Oxford University Press, New York. doi:10.1093/esr/jcm030

  • Caraway B (2010) Online labour markets: an inquiry into oDesk providers. Work Organ Labour Global 4(2):111–125

    Google Scholar 

  • Chanal AV, Caron-Fasan M (2010) The difficulties involved in developing business models open to innovation communities: the case of a crowdsourcing platform. Manag 13(4):318–340. doi:10.3917/mana.134.0318

    Google Scholar 

  • Chesbrough H, Crowther A (2006) Beyond high tech: early adopters of open innovation in other industries. R&D Manag 36(3):229–236. doi:10.1111/j.1467-9310.2006.00428.x

    Article  Google Scholar 

  • Corney JR, Torres-Sanchez C, Jagadeesan AP, Regli WC (2009) Outsourcing labour to the cloud. Int J Innov Sustain Develop 4(4):294–313. doi:10.1504/IJISD.2009.033083

    Article  Google Scholar 

  • Darke P, Shanks G, Broadbent M (1998) Successfully completing case study research: combining rigour, relevance and pragmatism. Inform Syst J 8(4):273–289. doi:10.1046/j.1365-2575.1998.00040.x

    Article  Google Scholar 

  • Doan A, Ramakrishnan R, Halevy AY (2011) Crowdsourcing systems on the World-Wide Web. Commun ACM 54:86–96. doi:10.1145/1924421.1924442

    Article  Google Scholar 

  • Eisenhardt KM (1989) Building theories from case study research. Acad Manag Rev 14(4):532–550. doi:10.5465/AMR.1989.4308385

    Google Scholar 

  • Ernst H (2002) Success factors of new product development: a review of the empirical literature. Intern J Manag Rev 4(1):1–40. doi:10.1111/1468-2370.00075

    Article  Google Scholar 

  • Estellés-Arolas E, González-Ladrón-de-Guevara F (2012) Towards an integrated crowdsourcing definition. J Inform Sci 38(2):189–200. doi:10.1177/0165551512437638

    Article  Google Scholar 

  • Faste H (2011) Opening “open” innovation. In: DPPI ‘11 proceedings of the 2011 conference on design pleasurable products and interfaces

  • Franke N, Piller F (2004) Value creation by toolkits for user innovation and design: the case of the watch market. J Prod Innov Manag 21:401–415. doi:10.1111/j.0737-6782.2004.00094.x

    Article  Google Scholar 

  • Füller J, Matzler K (2007) Virtual product experience and customer participation—a chance for customer-centred, really new products. Technovation 27(6–7):378–387. doi:10.1016/j.technovation.2006.09.005

    Article  Google Scholar 

  • Füller J, Hutter K, Faullant R (2011) Why co-creation experience matters? Creative experience and its impact on the quantity and quality of creative contributions. R&D Manag 41(3):259–273. doi:10.1111/j.1467-9310.2011.00640.x

    Article  Google Scholar 

  • Gassmann O, Enkel E (2004) Towards a theory of open innovation: three core process archetypes. Paper presented at the R&D management conference (RADMA), Lissabon

  • Geiger D, Seedorf S, Schulze T, Nickerson R, Schader M (2011) Managing the crowd: towards a taxonomy of crowdsourcing processes. In: Proceedings of the 7th American conference on information system, Detroit

  • Hartley SE (2010) Kiva.org: crowd-sourced microfinance and cooperation in group lending working paper, 25 March 2010 Available at SSRN: http://ssrn.com/abstract=1572182

  • Hoßfeld T, Hirth M, Tran-Gia P (2012) Aktuelles Schlagwort: crowdsourcing. Inf Spek 35(3):204–208. doi:10.1007/s11576-012-0321-7

    Article  Google Scholar 

  • Howe J (2006a) Crowdsourcing: a definition. http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html. Accessed 18 May 2012

  • Howe J (2006b) The rise of crowdsourcing. Wired Mag 14(6):1–4

    Google Scholar 

  • Howe J (2008) Crowdsourcing. Why the power of the crowd is driving the future of business. Crown Business Publishing, New York

  • Howells J (2006) Intermediation and the role of intermediaries in innovation. Res Policy 35:715–728. doi:10.1016/j.respol.2006.03.005

    Article  Google Scholar 

  • Jain R (2010) Investigation of governance mechanisms for crowdsourcing initiatives. Paper presented at the AMCIS 2010 proceedings

  • Jayakanthan R, Sundararajan D (2011) Enterprise crowdsourcing solutions for software development and ideation. Paper presented at the proceedings 2nd international workshop on ubiquitous crowdsouring

  • Jeppesen LB, Lakhani KR (2010) Marginality and problem-solving effectiveness in broadcast search. Organ Sci 21(5):1016–1033. doi:10.1287/orsc.1090.0491

    Article  Google Scholar 

  • Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing—a study on mechanical turk. In: Proceedings of the 7th AMCIS, Detroit

  • Kirkels Y, Duysters G (2010) Brokerage in SME networks. Res Policy 39:375–385. doi:10.1016/j.respol.2010.01.005

    Article  Google Scholar 

  • Kleeman F, Voss GG, Rieder K (2008) Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing. Sci Tech Innov Stud 4(1):5–26

    Google Scholar 

  • Klerkx L, Leeuwis C (2009) Establishment and embedding of innovation brokers at different innovation system levels: insights from the Dutch agricultural sector. Tech Forecast Soc Change 76:849–860. doi:10.1016/j.techfore.2008.10.001

    Article  Google Scholar 

  • Lakhani KR, Wolf B (2005) Why hackers do what they do. Understanding motivation and effort in free/open source software projects. In: Feller J, Fitzgerald B, Hissam S, Lakhani KR (eds) Perspectives on free and open source software. The MIT Press, Cambridge. doi:10.2139/ssrn.443040

  • Lakhani KR, Jeppesen LB, Lohse PA, Panetta JA (2007) The value of openness in scientific problem solving. Harvard Business School Working Paper No 07-050 [Online] http://www.hbs.edu/research/pdf/07-050pdf

  • Leimeister JM (2010) Collective intelligence. Bus Inf Syst Eng 4(2):245–248. doi:10.1007/s12599-010-0114-8

    Article  Google Scholar 

  • Leimeister JM, Huber M, Bretschneider U, Krcmar H (2009) Leveraging crowdsourcing: activation-supporting components for IT-based ideas competition. J Manag Inf Syst 26(1):197–224. doi:10.2753/MIS0742-1222260108

    Article  Google Scholar 

  • Malone TW, Laubacher R, Dellarocas C (2010) The collective intelligence genome. MIT Sloan Manag Rev 51(3):20–31

    Google Scholar 

  • Malone T, Laubacher R, Johns T (2011) The big idea: the age of hyperspecialization. Harvard Bus Rev, July–August, pp 1–11. http://www.topcoder.com/wp-content/uploads/2011/09/Hyperspecialization.pdf. Accessed 13 June 2012

  • Mao K, Yang Y, Li M, Harman M (2013) Pricing Crowdsourcing-Based Software Development Tasks. Paper presented at the 26th international conf on software engineering, San Francisco

  • Meredith J (1998) Building operations management theory through case and field research. J Operat Manag 16(4):441–454. doi:10.1016/S0272-6963(98)00023-0

    Article  Google Scholar 

  • Muhdi L, Boutellier R (2011a) The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. Int J Entrepren Innov Manag 14(4):315–332. doi:10.1504/IJEIM.2011.043052

    Google Scholar 

  • Muhdi L, Boutellier R (2011b) Motivational factors affecting participation and collaboration of members in two different Swiss Innovation communities. Int J Innov Manag 15(3):543–562. doi:10.1142/S1363919611003477

    Article  Google Scholar 

  • Myers GJ, Sandler C, Badgett T (2011) The art of software testing. New Jersey

  • Oliveira F, Ramos I, Santos L (2010) Definition of a crowdsourcing innovation service for the European SMEs. In: Daniel F, Facca FM (eds) Current trends in web engineering. Springer, Heidelberg

  • Paulini M, Murty P, Maher ML (2012) Understanding collective design communication in open innovation communities. Working Paper. University of Sydney, Australia

  • Pénin J (2008) More open than open innovation? Rethinking the concept of openness in innovation studies. Paper presented at the DT BETA, Strasbourg

  • Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. Paper presented at the CHI 2011—proceed of the SIGCHI conference, Vancouver, May 7–12

  • Rayna T, Striukova L (2010) Large-scale open innovation: open source vs. patent pools. Int J Techn Manag 52(3, 4):477–496

    Article  Google Scholar 

  • Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: exploring the practice. IEEE Softw 29(2):46–51. doi:10.1109/MS.2011.132

    Article  Google Scholar 

  • Rouse AC (2010) A Preliminary taxonomy of crowdsourcing. In: Australian conference on information system (ACIS), Brisbane, 1–3 Dec 2010

  • Schenk E, Guittard C (2009) Crowdsourcing: what can be outsourced to the crowd, and why? In: HAL: sciences de l’Homme et de la Société

  • Schenk E, Guittard C (2011) Towards a characterization of crowdsourcing practices. J Innov Econ 7(1):93–107. doi:10.3917/jie.007.0093

    Article  Google Scholar 

  • Steinfield C, Markus ML, Wigand RT (2011) Through a glass clearly: standards, architecture, and process transparency in global supply chains. J Manag Inform Sys 28(2):75–108. doi:10.2753/MIS0742-1222280204

    Article  Google Scholar 

  • Stewart J, Hyysalo S (2008) Intermediaries, users and social learning in technological innovation. Int J Innov Manag 12(3):295–325. doi:10.1142/S1363919608002035

    Article  Google Scholar 

  • Surowiecki J (2004) The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations, 1st edn. Doubleday Books, New York

    Google Scholar 

  • Tapscott D, Williams AD (2007) Wikinomics: how mass collaboration changes everything. New York. doi:10.1111/j.1468-0270.2008.864_2.x

    Google Scholar 

  • Tung Y-H, Tseng S-S (2013) A novel approach to collaborative testing in a crowdsourcing environment. J Syst Softw 86(8):2143–2153. doi:10.1016/j.jss.2013.03.079

    Article  Google Scholar 

  • Verner JM, Abdullah LM (2012) Exploratory case study research: outsourced project failure. Inform Softw Technol 54(8):866–886. doi:10.1016/j.infsof.2011.11.001

    Article  Google Scholar 

  • von Hippel E (1986) Lead users: a source of novel product concepts. Manag Sci 32(7):791–805. doi:10.1287/mnsc.32.7.791

    Article  Google Scholar 

  • Vukovic M (2009) Crowdsourcing for Enterprises. Paper presented at the SERVICES ‘09 proceedings of the 2009 congress on services—I, Los Angeles

  • Vukovic M, Bartolini C (2010) Towards a research agenda for enterprise crowdsourcing. In: Margaria T, Steffen B (eds) Leveraging Applications of formal methods, verification, and validation, Lecture notes in computer science, vol 6415. Springer, Berlin Heidelberg, pp 425-434. doi:10.1007/978-3-642-16558-0_36

  • Walmsley A (2009) The art of delegation. Marketing, 22 July 2009, pp 12–12 (1 page)

  • Ward C, Ramachandra V (2010) Crowdfunding the next hit: microfunding online experience goods. Workshop on computational social science and the wisdom of crowds (NIPS 2010). http://people.cs.umass.edu/~wallach/workshops/nips2010css/papers/ward.pdf. Accessed 20 July 2012

  • West J, Lakhani K (2008) Getting clear about communities in open innovation. Ind Innov 15(2):223–231

    Article  Google Scholar 

  • Whitla P (2009) Crowdsourcing and its application in marketing activities. Contem Manag Res 5(1):15–28

    Google Scholar 

  • Whittaker JA (2000) What is software testing? And why is it so hard? IEEE Softw 17(1):70–79. doi:10.1109/52.819971

    Google Scholar 

  • Winch GM, Courtney R (2007) The organization of innovation brokers: an international review. Technol Analy Strategy Manag 19:747–763. doi:10.1080/09537320701711223

    Article  Google Scholar 

  • Yin RK (2003) Case study research: design and methods, vol 5. Applied social research methods series, 3rd edn. Sage Publications, Thousand Oaks. doi:10.1111/j.1540-4781.2011.01212_17.x

  • Yuen M-C, King I, Leung K-S (2011) A survey of crowdsourcing systems. Paper presented at the 2011 IEEE international conference on privacy, security, risk, and trust, Boston

  • Zhao Y, Zhu Q (2012) Evaluation on crowdsourcing research: current status and future direction. Inf Syst Front. doi:10.1007/s10796-012-9350-4

  • Zwass V (2010) Co-creation: toward a taxonomy and an integrated research perspective. Int J Electr Com 15(1):11–48. doi:10.2753/JEC1086-4415150101

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Fachgebiet Wirtschaftsinformatik, Universität Kassel, Pfannkuchstr. 1, 34121, Kassel, Germany

    Shkodran Zogaj, Ulrich Bretschneider & Jan Marco Leimeister

Authors
  1. Shkodran Zogaj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ulrich Bretschneider
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Marco Leimeister
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shkodran Zogaj.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zogaj, S., Bretschneider, U. & Leimeister, J.M. Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary. J Bus Econ 84, 375–405 (2014). https://doi.org/10.1007/s11573-014-0721-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11573-014-0721-9

Keywords

JEL Classification

Search

Navigation