Key Crowdsourcing Technologies for Product Design and Development
Traditionally, small and medium enterprises (SMEs) in manufacturing rely heavily on a skilled, technical and professional workforce to increase productivity and remain globally competitive. Crowdsourcing offers an opportunity for SMEs to get access to online communities who may provide requested services such as generating design ideas or problem solutions. However, there are some barriers preventing them from adopting crowdsourcing into their product design and development (PDD) practice. In this paper, we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools, crowdsourcing frameworks, and techniques in terms of open call generation, rewarding, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process. We also explore the necessary techniques and tools to support the crowdsourcing PDD process. Finally, we propose some key guidelines for coping with the aforementioned challenges in the crowdsourcing PDD process.
KeywordsCrowdsourcing technologies product design and development (PDD) communication information sharing design evaluation feedback
This work was supported by the China Scholarship Council and State Key Laboratory of Traction Power at Southwest Jiaotong University (No. TPL1501). We thank anonymous reviewers for their helpful comments which helped to improve the paper.
- E. Simperl. How to use crowdsourcing effectively: Guidelines and examples. LIBER Quarterly, vol. 25, no. 1, pp. 18–39, 2015. DOI: 10.18352/lq.9948.Google Scholar
- Open Innovation Defintion, [Online], Available: https://doi.org/www.innoget.com/open-innovation-definition, September 20, 2017.
- A. Wichman. Challenges, Crowdsourcing, Citizen Science: What’s the Dfi? [Online], Available: https://doi.org/digital.gov/2015/12/16/challenges-crowdsourcing-citizen-sciencewhats-the-dif/, September 20, 2017.
- Citizen Science: Definition, [Online], Available: https://doi.org/www.citizensciencecenter.com/citizen-science-definition/, September 15, 2017.
- J. Howe. The rise of crowdsourcing. Wired magazine, vol. 14, no. 6, pp. 1–4, 2006.Google Scholar
- A. Kittur, J. V. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, J. Horton. The future of crowd work. In Proceedings of the Conference on Computer Supported Cooperative Work, ACM, San Antonio, USA, pp. 1301–1318, 2013. DOI: 10.1145/2441776. 2441923.Google Scholar
- X. J. Niu, S. F. Qin. A review of crowdsourcing technology for product design and development. In Proceedings of the 23rd International Conference on Automation and Computing, IEEE, Huddersfield, UK, 2017. DOI: 10.23919/IConAC.2017.8081981.Google Scholar
- Open Innovation & Crowdsourcing Examples, [Online], Available: https://doi.org/www.boardofinnovation.com/list-openinnovation-crowdsourcing-examples/, September 25, 2017.
- Cocreate with Your Fans. [Online], Available: https://doi.org/www.jovoto.com/cocreate-with-your-fans/, September 26, 2017.
- J. Jin, P. Ji, Y. Liu. Prioritising engineering characteristics based on customer online reviews for quality function deployment. Journal of Engineering Design vol. 25, no. 7–9, pp. 303–324, 2014. DOI: 10.1080/09544828.2014. 984665.Google Scholar
- S. N. Pedersen, M. E. Christensen, T. J. Howard. Robust design requirements specification: A quantitative method for requirements development using quality loss functions. Journal of Engineering Design, vol. 27, no. 8, pp. 544–567, 2016. DOI: 10.1080/09544828.2016.1183163.Google Scholar
- Open IDEO, [Online], Available: https://doi.org/challenges.openideo.com, September 26, 2017.
- Herox. Where Innovators Come to Compete and Businesses Come for Solutions, [Online], Available: https://doi.org/herox.com/ September 28, 2017.
- Challenge.gov., [Online], Available: https://doi.org/www.challenge.gov/challenge, September 28, 2017.
- E.A.T. School Lunch UX Challenge, [Online], Available: https://doi.org/lunchux.devpost.com/ September 28, 2017.
- H. To. Task assignment in spatial crowdsourcing: challenges and approaches. In Proceedings of the 3rd ACM Sigspatial Ph.D. Symposium, ACM, Burlingame, USA, pp. 1–4, 2016. DOI: 10.1145/3003819.3003820.Google Scholar
- R. R. Morris, D. McDuff. Crowdsourcing techniques for affective computing. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, A. Kappas, Eds., Oxford, UK: Oxford University Press, pp. 384–394, 2014.Google Scholar
- J. Thebault-Spieker, L. G. Terveen, B. Hecht. Avoiding the south side and the suburbs: The geography of mobile crowdsourcing markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, Canada, pp. 265–275, 2015. DOI: 10.1145/2675133.2675278.Google Scholar
- H. To, R. Geraldes, C. Shahabi, S. H. Kim, H. Prendinger. An empirical study of workers’ behavior in spatial crowdsourcing. In Proceedings of the 3rd International ACM SIGMOD Workshop on Managing and Mining Enriched Geo-spatial Data, San Francisco, USA, 2016. DOI: 10.1145/2948649.2948657.Google Scholar
- S. Djelassi, I. Decoopman. Customers’ participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management, vol. 42, no. 5, pp. 683–692, 2013. DOI: 10.1016/j.indmarman.2013.05.006.Google Scholar
- H. Carpenter. Motivating the crowd to participate in your innovation initiative. A Guide to Open Innovation and Crowdsourcing, pp. 76–84, 2011.Google Scholar
- H. Simula, T. Ahola. A network perspective on idea and innovation crowdsourcing in industrial firms. Industrial Marketing Management, vol. 43, no. 3, pp. 400–408, 2014. DOI: 10.1016/j.indmarman.2013.12.008.Google Scholar
- W. Mason, D. J. Watts. Financial incentives and the “performance of crowds”. ACM Special Interest Group on Knowledge Discovery and Data Mining Explorations Newsletter, vol. 11, no. 2, pp. 100–108, 2009. DOI: 10.1145/1809400.1809422.Google Scholar
- J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the 5th International AAAI Conference on Web and Social Media, Barcelona, Spain, pp. 17–21, 2011.Google Scholar
- J. Howe. Crowdsousrcing: Why the Power of the Crowd is Driving the Future of Business. New York, USA: Crown Business, 2009.Google Scholar
- L. Von Ahn, L. Dabbish. Designing games with a purpose. Communications of the ACM, vol. 51, no. 8, pp. 58–67, 2008. DOI: 10.1145/1378704.1378719.Google Scholar
- B. Jonathan Mausam, D. S. Weld. Optimal testing for crowd workers. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Singapore, pp. 966–974, 2016.Google Scholar
- D. N. Chang, C. H. Chen. Product concept evaluation and selection using data mining and domain ontology in a crowdsourcing environment. Advanced Engineering Informatics, vol. 29, no. 4, pp. 759–774, 2015. DOI: 10.1016/j.aei.2015.06.003.Google Scholar
- P. G. Ipeirotis, F. Provost, J. Wang. Quality management on amazon mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, pp. 64–67, 2010. DOI: 10.1145/1837885.1837906.Google Scholar
- S. Sedhain, S. Sanner, D. Braziunas, L. X. Xie, J. Christensen. Social collaborative filtering for cold-start recommendations. In Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, USA, pp. 345–348, 2014. DOI: 10.1145/2645710.2645772.Google Scholar
- N. Holladay. Achieving quality through teamwork, [Online], Available: https://doi.org/www.unice.fr/crookallcours/teams/docs/teams%20Achieving%20Quality%20Through%20Teamwork.pdf, August 15, 2017.
- J. Pedersen, D. Kocsis, A. Tripathi, A. Tarrell, A. Weerakoon, N. Tahmasbi, J. Xiong, W. Deng, O. Oh, G. J. de Vreede. Conceptual foundations of crowdsourcing: A review of IS research. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 579–588, 2013. DOI: 10.1109/HICSS.2013.143.Google Scholar
- Quirky, [Online], Available: http://quirky.com/faq/#toggle-id-1, September 26, 2017.Google Scholar
- 99designs, [Online], Available: https://doi.org/99designs.co.uk/how-it-works, September 26, 2017.
- Jovoto, [Online], Available: https://doi.org/www.jovoto.com/create-outstanding-products/ September 26, 2017.
- T. P. Walter, A. Back. A text mining approach to evaluate submissions to crowdsourcing contests. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 3109–3118, 2013. DOI: 10.1109/HICSS.2013.64.Google Scholar
- T. P. Walter, A. Back. Towards Measuring Crowdsourcing Success: An Empirical Study on Effects of External Factors in Online Idea Contest, [Online], Available: https://doi.org/aisel.aisnet.org/cgi/viewcontent.cgi?article=1064&contextmcis2011, September 26, 2017.
- J. Bjork, M. Magnusson. Where do good innovation ideas come from? Exploring the influence of network connectivity on innovation idea quality Journal of Product Innovation Management, vol. 26, no. 6, pp. 662–670, 2009. DOI: 10.1111/j.1540-5885.2009.00691.x.Google Scholar
- G. Wang, X. M. Liu, W. G. Fan. A knowledge adoption model based framework for finding helpful user-generated contents in online communities. In Proceedings of the 32nd International Conference on Information Systems, Shanghai, China, pp. 1–11, 2011.Google Scholar
- K. Aniket, E. H. Chi, B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the Special Interest Group on Computer-Human Interaction Conference on Human Factors in Computing Systems, ACM, Florence, Italy, pp. 453–456, 2008. DOI: 10.1145/1357054. 1357127.Google Scholar
- A. D. Shaw, J. J. Horton, D. L. Chen. Designing incentives for inexpert human raters. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, ACM, Hangzhou, China, pp. 275–284, 2011. DOI: 10.1145/1958824.1958865.Google Scholar
- A. B. Xu, H. M. Rao, S. P. Dow, B. P. Bailey. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 1637–1648, 2015. DOI: 10.1145/2675133.2675140.Google Scholar
- P. Dai, J. M. Rzeszotarski, P. Paritosh, E. H. Chi. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 628–638, 2015. DOI: 10.1145/2675133. 2675260.Google Scholar
- P. Dai, Mausam, D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, AAAI, Atlanta, USA, pp. 1168–1174, 2010.Google Scholar
- A. Mohammad, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, S. Dustdar. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing, vol. 17, no. 2, pp. 76–81, 2013. DOI: 10.1109/MIC.2013.20.Google Scholar
- M. S. Bernstein, G. Little, R. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, D. Crowell. Soylent: A word processor with a crowd inside. Communications of the ACM, vol. 58, no. 8, pp. 85–94, 2015. DOI: 10. 1145/2791285.Google Scholar
- D. Ofer, O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In Proceedings of the 22nd Annual Conference on Learning Theory, Montreal, Canada, 2009.Google Scholar
- What Are the Best Tools for Crowdsourcing Ideas? [Online], Available: https://doi.org/www.quora.com/What-are-thebest-tools-for-crowdsourcing-idea, February 15, 2018.
- Crowdsourcing Tool, [Online], Available: https://doi.org/ideascale.com/service/crowdsourcing-tool/, February 15, 2018.
- S. F. Qin, D. van der Velde, E. Chatzakis, T. McStea, N. Smith. Exploring barriers and opportunities in adopting crowdsourcing based new product development in manufacturing SMEs. Chinese Journal of Mechanical Engineering, vol. 29, no. 6, pp. 1052–1066, 2016. DOI: 10.3901/CJME.2016.0808.089.Google Scholar
- C. Y. Xu, S. F. Qin, Z. P. Xiao. Crowdsourcing based product innovation design service model for small-sized and medium-sized enterprises. In Proceedings of 18th International Conference on Automation and Computing, IEEE, Leicestershire, UK, pp. 1–5, 2012.Google Scholar
- Product Design, [Online], Available: https://doi.org/en.wikipedia.org/wiki/Product_design, October 20, 2017.
- K. Ulrich, S. Eppinger. Product Design and Development. 6th ed. New York, USA: McGraw-Hill Higher Education, 2015.Google Scholar
- J. L. Nevins, D. E. Whitney. Concurrent Design of Products and Processes: A Strategy for the Next Generation in Manufacturing. New York, USA: McGraw-Hill Companies, 1989.Google Scholar
- Y. L. Tu, S. Q. Xie, R. Y. K. Fung. Product development cost estimation in mass customization. IEEE Transactions on Engineering Management, vol. 54, no. 1, pp. 29–40, 2007. DOI: 10.1109/TEM.2006.889065.Google Scholar
- H. P. Shang, Z. X. Zhao. Integration of manufacturing services into virtual environments over the Internet. International Journal of Automation and Computing, vol. 1, no. 1, pp. 89–106, 2004. DOI: 10.1007/s11633-004-0089-3.Google Scholar
- M. L. Gray, S. Suri, S. S. Ali, D. Kulkarni. The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-supported Cooperative Work & Social Computing, ACM, San Francisco, USA, pp. 134–147, 2016. DOI: 10.1115/DETC2002/DTM-34020.Google Scholar
- W. Y. Song, X. G. Ming, Z. Y. Wu. An integrated rough number-based approach to design concept evaluation under subjective environments. Journal of Engineering Design, vol. 24, no. 5, pp. 320–341, 2013. DOI: 10.1080/ 09544828.2012.732994.Google Scholar
- Z. Ayag, R. G. Ozdem. An analytic network process-based approach to concept evaluation in a new product development environment. Journal of Engineering Design, vol. 18, no. 3, pp. 209–226, 2007. DOI: 10.1080/09544820600752740.Google Scholar
- E. Alex, N. Rahmati, N. Zhu. Structured handoffs in expert crowdsourcing improve communication and work output. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology, ACM, Honolulu, USA, pp. 99–100, 2014.Google Scholar
- H. P. Zhang, R. Q. Zhang, Y. P. Zhao, B. J. Ma. Big data modeling and analysis of microblog ecosystem. International Journal of Automation and Computing, vol. 11, no. 2, pp. 119–127, 2014. DOI: 10.1007/s11633-014-0774-9.Google Scholar
- L. Y. Zhai, L. P. Khoo, Z. W. Zhong. Design concept evaluation in product development using rough sets and grey relation analysis. Expert Systems with Applications, vol. 36, no. 3, pp. 7072–7079, 2009. DOI: 10.1016/j.eswa. 2008.08.068.Google Scholar
- V. Girotto. Collective creativity through a micro-tasks crowdsourcing approach. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, ACM, San Francisco, USA, pp. 143–146, 2016. DOI: 10.1145/2818052.2874356.Google Scholar
- Z. Ayag. An integrated approach to concept evaluation in a new product development. Journal of Intelligent Manufacturing, vol. 27, no. 5, pp. 991–1005, 2016. DOI: 10.1007/ s10845-014-0930-7.Google Scholar
- V. Tiwari, P. K. Jain, P. Tandon. Product design concept evaluation using rough sets and VIKOR method. Advanced Engineering Informatics, vol. 30, no. 1, pp. 16–25, 2016. DOI: 10.1016/j.aei.2015.11.005.Google Scholar
- Z. Ayag. An integrated approach to evaluating conceptual design alternatives in a new product development environment. International Journal of Production Research, vol. 43, no. 4, pp. 687–713, 2005. DOI: 10.1080/00207540512 331311831.Google Scholar
- T. L. Saaty. Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, USA: RWS publications, 1996.Google Scholar
- D. N. Chang, C. H. Chen. Exploration of a concept screening method in a crowdsourcing environment. Moving Integrated Product Development to Service Clouds in the Global Economy, J. Z. Cha, S. Y. Chou, J. Stjepandic, R. Curran, W. S. Xu, Eds, Netherlands: IOS Press, pp. 861–870, 2014. DOI: 10.3233/978-1-61499-440-4-861.Google Scholar
- W. Qi, Z. W. Ren, Z. F. Guo. XML-based data processing in network supported collaborative design. International Journal of Automation and Computing, vol. 7, no. 3, pp. 330–335, 2010. DOI: 10.1007/s11633-010-0511-y.Google Scholar
- K. Gautam. Advanced Methods of Teaching “Feedback”. Pondicherry University, [Online], Available: https://doi.org/www.slideshare.net/gautamkrverma/feedback-33256451, November 5, 2017.
- P. L. Jackson. Getting Design Right: A Systems Approach. Boca Raton, USA: CRC Press, 2009.Google Scholar
- Z. Y. Ma, A. X. Sun, Q. Yuan, G. Cong. Topic-driven reader comments summarization. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, Maui, USA, pp. 265–274, 2012. DOI: 10.1145/2396761.2396798.Google Scholar
- J. Hui, A. Glenn, R. Jue, E. Gerber, S. Dow. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, AAAI, San Diego, USA, pp. 72–82. 2015.Google Scholar
- A. B. Xu, B. Bailey. What do you think? A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, Seattle, USA, pp. 295–304, 2012. DOI: 10.1145/2145204.2145252.Google Scholar
- S. Dow, E. Gerber, A. Wong. A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Paris, France, pp. 227–236, 2013. DOI: 10.1145/2470654. 2470686.Google Scholar
- A. B. Xu, S. W. Huang, B. Bailey. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Baltimore, USA, pp. 1433–1444, 2014. DOI: 10.1145/2531602.2531604.Google Scholar
- K. Luther, A. Pavel, W. Wu, J. L. Tolentino, M. Agrawala, B. Hartmann, S. P. Dow. CrowdCrit: Crowdsourcing and aggregating visual design critique. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, Baltimore, USA, pp. 21–24, 2014. DOI: 10.1145/2556420.2556788.Google Scholar
- S. Salimun, N. Janom, N. H. Arshad. Quality factors of crowdsourcing system: Paper review. In Proceedings of the 6th IEEE Control and System Graduate Research Colloquium, Shah Alam, Malaysia, pp. 82–86, 2015. DOI: 10.1109/ICSGRC.2015.7412469.Google Scholar
- R. Khazankin, D. Schall, S. Dustdar. Predicting QoS in scheduled crowdsourcing. Advanced Information Systems Engineering, J. Ralyte, X. Franch, S. Brinkkemper, S. Wrycza, Eds., Berlin Heidelberg, Germang: Springer, 2012. DOI: 10.1007/978-3-642-31095-9—30.Google Scholar
- R. Alabduljabbar, H. Al-Dossari. A task ontology-based model for quality control in crowdsourcing systems. In Proceedings of the International Conference on Research in Adaptive and Convergent Systems, ACM, Odense, Denmark, pp. 22–28, 2016. DOI: 10.1145/2987386.2987413.Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://doi.org/creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.