Traditionally, small and medium enterprises (SMEs) in manufacturing rely heavily on a skilled, technical and professional workforce to increase productivity and remain globally competitive. Crowdsourcing offers an opportunity for SMEs to get access to online communities who may provide requested services such as generating design ideas or problem solutions. However, there are some barriers preventing them from adopting crowdsourcing into their product design and development (PDD) practice. In this paper, we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools, crowdsourcing frameworks, and techniques in terms of open call generation, rewarding, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process. We also explore the necessary techniques and tools to support the crowdsourcing PDD process. Finally, we propose some key guidelines for coping with the aforementioned challenges in the crowdsourcing PDD process.
E. Simperl. How to use crowdsourcing effectively: Guidelines and examples. LIBER Quarterly, vol. 25, no. 1, pp. 18–39, 2015. DOI: 10.18352/lq.9948.
Open Innovation Defintion, [Online], Available: https://doi.org/www.innoget.com/open-innovation-definition, September 20, 2017.
A. Wichman. Challenges, Crowdsourcing, Citizen Science: What’s the Dfi? [Online], Available: https://doi.org/digital.gov/2015/12/16/challenges-crowdsourcing-citizen-sciencewhats-the-dif/, September 20, 2017.
Citizen Science: Definition, [Online], Available: https://doi.org/www.citizensciencecenter.com/citizen-science-definition/, September 15, 2017.
J. Howe. The rise of crowdsourcing. Wired magazine, vol. 14, no. 6, pp. 1–4, 2006.
A. Kittur, J. V. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, J. Horton. The future of crowd work. In Proceedings of the Conference on Computer Supported Cooperative Work, ACM, San Antonio, USA, pp. 1301–1318, 2013. DOI: 10.1145/2441776. 2441923.
X. J. Niu, S. F. Qin. A review of crowdsourcing technology for product design and development. In Proceedings of the 23rd International Conference on Automation and Computing, IEEE, Huddersfield, UK, 2017. DOI: 10.23919/IConAC.2017.8081981.
Open Innovation & Crowdsourcing Examples, [Online], Available: https://doi.org/www.boardofinnovation.com/list-openinnovation-crowdsourcing-examples/, September 25, 2017.
Cocreate with Your Fans. [Online], Available: https://doi.org/www.jovoto.com/cocreate-with-your-fans/, September 26, 2017.
J. Jin, P. Ji, Y. Liu. Prioritising engineering characteristics based on customer online reviews for quality function deployment. Journal of Engineering Design vol. 25, no. 7–9, pp. 303–324, 2014. DOI: 10.1080/09544828.2014. 984665.
S. N. Pedersen, M. E. Christensen, T. J. Howard. Robust design requirements specification: A quantitative method for requirements development using quality loss functions. Journal of Engineering Design, vol. 27, no. 8, pp. 544–567, 2016. DOI: 10.1080/09544828.2016.1183163.
Open IDEO, [Online], Available: https://doi.org/challenges.openideo.com, September 26, 2017.
Herox. Where Innovators Come to Compete and Businesses Come for Solutions, [Online], Available: https://doi.org/herox.com/ September 28, 2017.
Challenge.gov., [Online], Available: https://doi.org/www.challenge.gov/challenge, September 28, 2017.
E.A.T. School Lunch UX Challenge, [Online], Available: https://doi.org/lunchux.devpost.com/ September 28, 2017.
H. To. Task assignment in spatial crowdsourcing: challenges and approaches. In Proceedings of the 3rd ACM Sigspatial Ph.D. Symposium, ACM, Burlingame, USA, pp. 1–4, 2016. DOI: 10.1145/3003819.3003820.
R. R. Morris, D. McDuff. Crowdsourcing techniques for affective computing. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, A. Kappas, Eds., Oxford, UK: Oxford University Press, pp. 384–394, 2014.
J. Thebault-Spieker, L. G. Terveen, B. Hecht. Avoiding the south side and the suburbs: The geography of mobile crowdsourcing markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, Canada, pp. 265–275, 2015. DOI: 10.1145/2675133.2675278.
H. To, R. Geraldes, C. Shahabi, S. H. Kim, H. Prendinger. An empirical study of workers’ behavior in spatial crowdsourcing. In Proceedings of the 3rd International ACM SIGMOD Workshop on Managing and Mining Enriched Geo-spatial Data, San Francisco, USA, 2016. DOI: 10.1145/2948649.2948657.
S. Djelassi, I. Decoopman. Customers’ participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management, vol. 42, no. 5, pp. 683–692, 2013. DOI: 10.1016/j.indmarman.2013.05.006.
H. Carpenter. Motivating the crowd to participate in your innovation initiative. A Guide to Open Innovation and Crowdsourcing, pp. 76–84, 2011.
H. Simula, T. Ahola. A network perspective on idea and innovation crowdsourcing in industrial firms. Industrial Marketing Management, vol. 43, no. 3, pp. 400–408, 2014. DOI: 10.1016/j.indmarman.2013.12.008.
W. Mason, D. J. Watts. Financial incentives and the “performance of crowds”. ACM Special Interest Group on Knowledge Discovery and Data Mining Explorations Newsletter, vol. 11, no. 2, pp. 100–108, 2009. DOI: 10.1145/1809400.1809422.
J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the 5th International AAAI Conference on Web and Social Media, Barcelona, Spain, pp. 17–21, 2011.
J. Howe. Crowdsousrcing: Why the Power of the Crowd is Driving the Future of Business. New York, USA: Crown Business, 2009.
L. Von Ahn, L. Dabbish. Designing games with a purpose. Communications of the ACM, vol. 51, no. 8, pp. 58–67, 2008. DOI: 10.1145/1378704.1378719.
B. Jonathan Mausam, D. S. Weld. Optimal testing for crowd workers. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Singapore, pp. 966–974, 2016.
D. N. Chang, C. H. Chen. Product concept evaluation and selection using data mining and domain ontology in a crowdsourcing environment. Advanced Engineering Informatics, vol. 29, no. 4, pp. 759–774, 2015. DOI: 10.1016/j.aei.2015.06.003.
P. G. Ipeirotis, F. Provost, J. Wang. Quality management on amazon mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, pp. 64–67, 2010. DOI: 10.1145/1837885.1837906.
S. Sedhain, S. Sanner, D. Braziunas, L. X. Xie, J. Christensen. Social collaborative filtering for cold-start recommendations. In Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, USA, pp. 345–348, 2014. DOI: 10.1145/2645710.2645772.
N. Holladay. Achieving quality through teamwork, [Online], Available: https://doi.org/www.unice.fr/crookallcours/teams/docs/teams%20Achieving%20Quality%20Through%20Teamwork.pdf, August 15, 2017.
J. Pedersen, D. Kocsis, A. Tripathi, A. Tarrell, A. Weerakoon, N. Tahmasbi, J. Xiong, W. Deng, O. Oh, G. J. de Vreede. Conceptual foundations of crowdsourcing: A review of IS research. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 579–588, 2013. DOI: 10.1109/HICSS.2013.143.
Quirky, [Online], Available: http://quirky.com/faq/#toggle-id-1, September 26, 2017.
99designs, [Online], Available: https://doi.org/99designs.co.uk/how-it-works, September 26, 2017.
Jovoto, [Online], Available: https://doi.org/www.jovoto.com/create-outstanding-products/ September 26, 2017.
T. P. Walter, A. Back. A text mining approach to evaluate submissions to crowdsourcing contests. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 3109–3118, 2013. DOI: 10.1109/HICSS.2013.64.
T. P. Walter, A. Back. Towards Measuring Crowdsourcing Success: An Empirical Study on Effects of External Factors in Online Idea Contest, [Online], Available: https://doi.org/aisel.aisnet.org/cgi/viewcontent.cgi?article=1064&contextmcis2011, September 26, 2017.
J. Bjork, M. Magnusson. Where do good innovation ideas come from? Exploring the influence of network connectivity on innovation idea quality Journal of Product Innovation Management, vol. 26, no. 6, pp. 662–670, 2009. DOI: 10.1111/j.1540-5885.2009.00691.x.
G. Wang, X. M. Liu, W. G. Fan. A knowledge adoption model based framework for finding helpful user-generated contents in online communities. In Proceedings of the 32nd International Conference on Information Systems, Shanghai, China, pp. 1–11, 2011.
K. Aniket, E. H. Chi, B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the Special Interest Group on Computer-Human Interaction Conference on Human Factors in Computing Systems, ACM, Florence, Italy, pp. 453–456, 2008. DOI: 10.1145/1357054. 1357127.
A. D. Shaw, J. J. Horton, D. L. Chen. Designing incentives for inexpert human raters. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, ACM, Hangzhou, China, pp. 275–284, 2011. DOI: 10.1145/1958824.1958865.
A. B. Xu, H. M. Rao, S. P. Dow, B. P. Bailey. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 1637–1648, 2015. DOI: 10.1145/2675133.2675140.
P. Dai, J. M. Rzeszotarski, P. Paritosh, E. H. Chi. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 628–638, 2015. DOI: 10.1145/2675133. 2675260.
P. Dai, Mausam, D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, AAAI, Atlanta, USA, pp. 1168–1174, 2010.
A. Mohammad, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, S. Dustdar. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing, vol. 17, no. 2, pp. 76–81, 2013. DOI: 10.1109/MIC.2013.20.
M. S. Bernstein, G. Little, R. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, D. Crowell. Soylent: A word processor with a crowd inside. Communications of the ACM, vol. 58, no. 8, pp. 85–94, 2015. DOI: 10. 1145/2791285.
D. Ofer, O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In Proceedings of the 22nd Annual Conference on Learning Theory, Montreal, Canada, 2009.
What Are the Best Tools for Crowdsourcing Ideas? [Online], Available: https://doi.org/www.quora.com/What-are-thebest-tools-for-crowdsourcing-idea, February 15, 2018.
Crowdsourcing Tool, [Online], Available: https://doi.org/ideascale.com/service/crowdsourcing-tool/, February 15, 2018.
S. F. Qin, D. van der Velde, E. Chatzakis, T. McStea, N. Smith. Exploring barriers and opportunities in adopting crowdsourcing based new product development in manufacturing SMEs. Chinese Journal of Mechanical Engineering, vol. 29, no. 6, pp. 1052–1066, 2016. DOI: 10.3901/CJME.2016.0808.089.
C. Y. Xu, S. F. Qin, Z. P. Xiao. Crowdsourcing based product innovation design service model for small-sized and medium-sized enterprises. In Proceedings of 18th International Conference on Automation and Computing, IEEE, Leicestershire, UK, pp. 1–5, 2012.
Product Design, [Online], Available: https://doi.org/en.wikipedia.org/wiki/Product_design, October 20, 2017.
K. Ulrich, S. Eppinger. Product Design and Development. 6th ed. New York, USA: McGraw-Hill Higher Education, 2015.
J. L. Nevins, D. E. Whitney. Concurrent Design of Products and Processes: A Strategy for the Next Generation in Manufacturing. New York, USA: McGraw-Hill Companies, 1989.
Y. L. Tu, S. Q. Xie, R. Y. K. Fung. Product development cost estimation in mass customization. IEEE Transactions on Engineering Management, vol. 54, no. 1, pp. 29–40, 2007. DOI: 10.1109/TEM.2006.889065.
H. P. Shang, Z. X. Zhao. Integration of manufacturing services into virtual environments over the Internet. International Journal of Automation and Computing, vol. 1, no. 1, pp. 89–106, 2004. DOI: 10.1007/s11633-004-0089-3.
M. L. Gray, S. Suri, S. S. Ali, D. Kulkarni. The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-supported Cooperative Work & Social Computing, ACM, San Francisco, USA, pp. 134–147, 2016. DOI: 10.1115/DETC2002/DTM-34020.
W. Y. Song, X. G. Ming, Z. Y. Wu. An integrated rough number-based approach to design concept evaluation under subjective environments. Journal of Engineering Design, vol. 24, no. 5, pp. 320–341, 2013. DOI: 10.1080/ 09544828.2012.732994.
Z. Ayag, R. G. Ozdem. An analytic network process-based approach to concept evaluation in a new product development environment. Journal of Engineering Design, vol. 18, no. 3, pp. 209–226, 2007. DOI: 10.1080/09544820600752740.
E. Alex, N. Rahmati, N. Zhu. Structured handoffs in expert crowdsourcing improve communication and work output. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology, ACM, Honolulu, USA, pp. 99–100, 2014.
H. P. Zhang, R. Q. Zhang, Y. P. Zhao, B. J. Ma. Big data modeling and analysis of microblog ecosystem. International Journal of Automation and Computing, vol. 11, no. 2, pp. 119–127, 2014. DOI: 10.1007/s11633-014-0774-9.
L. Y. Zhai, L. P. Khoo, Z. W. Zhong. Design concept evaluation in product development using rough sets and grey relation analysis. Expert Systems with Applications, vol. 36, no. 3, pp. 7072–7079, 2009. DOI: 10.1016/j.eswa. 2008.08.068.
V. Girotto. Collective creativity through a micro-tasks crowdsourcing approach. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, ACM, San Francisco, USA, pp. 143–146, 2016. DOI: 10.1145/2818052.2874356.
Z. Ayag. An integrated approach to concept evaluation in a new product development. Journal of Intelligent Manufacturing, vol. 27, no. 5, pp. 991–1005, 2016. DOI: 10.1007/ s10845-014-0930-7.
V. Tiwari, P. K. Jain, P. Tandon. Product design concept evaluation using rough sets and VIKOR method. Advanced Engineering Informatics, vol. 30, no. 1, pp. 16–25, 2016. DOI: 10.1016/j.aei.2015.11.005.
Z. Ayag. An integrated approach to evaluating conceptual design alternatives in a new product development environment. International Journal of Production Research, vol. 43, no. 4, pp. 687–713, 2005. DOI: 10.1080/00207540512 331311831.
H. Z. Huang, R. F. Bo, W. Chen. An integrated computational intelligence approach to product concept generation and evaluation. Mechanism and Machine Theory, vol. 41, no. 5, pp. 567–583, 2006. DOI: 10.1016/j.mechmachtheory. 2005.07.006.
T. L. Saaty. Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, USA: RWS publications, 1996.
D. N. Chang, C. H. Chen. Exploration of a concept screening method in a crowdsourcing environment. Moving Integrated Product Development to Service Clouds in the Global Economy, J. Z. Cha, S. Y. Chou, J. Stjepandic, R. Curran, W. S. Xu, Eds, Netherlands: IOS Press, pp. 861–870, 2014. DOI: 10.3233/978-1-61499-440-4-861.
W. Qi, Z. W. Ren, Z. F. Guo. XML-based data processing in network supported collaborative design. International Journal of Automation and Computing, vol. 7, no. 3, pp. 330–335, 2010. DOI: 10.1007/s11633-010-0511-y.
K. Gautam. Advanced Methods of Teaching “Feedback”. Pondicherry University, [Online], Available: https://doi.org/www.slideshare.net/gautamkrverma/feedback-33256451, November 5, 2017.
P. L. Jackson. Getting Design Right: A Systems Approach. Boca Raton, USA: CRC Press, 2009.
Z. Y. Ma, A. X. Sun, Q. Yuan, G. Cong. Topic-driven reader comments summarization. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, Maui, USA, pp. 265–274, 2012. DOI: 10.1145/2396761.2396798.
J. Hui, A. Glenn, R. Jue, E. Gerber, S. Dow. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, AAAI, San Diego, USA, pp. 72–82. 2015.
A. B. Xu, B. Bailey. What do you think? A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, Seattle, USA, pp. 295–304, 2012. DOI: 10.1145/2145204.2145252.
S. Dow, E. Gerber, A. Wong. A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Paris, France, pp. 227–236, 2013. DOI: 10.1145/2470654. 2470686.
A. B. Xu, S. W. Huang, B. Bailey. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Baltimore, USA, pp. 1433–1444, 2014. DOI: 10.1145/2531602.2531604.
K. Luther, A. Pavel, W. Wu, J. L. Tolentino, M. Agrawala, B. Hartmann, S. P. Dow. CrowdCrit: Crowdsourcing and aggregating visual design critique. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, Baltimore, USA, pp. 21–24, 2014. DOI: 10.1145/2556420.2556788.
S. Salimun, N. Janom, N. H. Arshad. Quality factors of crowdsourcing system: Paper review. In Proceedings of the 6th IEEE Control and System Graduate Research Colloquium, Shah Alam, Malaysia, pp. 82–86, 2015. DOI: 10.1109/ICSGRC.2015.7412469.
R. Khazankin, D. Schall, S. Dustdar. Predicting QoS in scheduled crowdsourcing. Advanced Information Systems Engineering, J. Ralyte, X. Franch, S. Brinkkemper, S. Wrycza, Eds., Berlin Heidelberg, Germang: Springer, 2012. DOI: 10.1007/978-3-642-31095-9—30.
R. Alabduljabbar, H. Al-Dossari. A task ontology-based model for quality control in crowdsourcing systems. In Proceedings of the International Conference on Research in Adaptive and Convergent Systems, ACM, Odense, Denmark, pp. 22–28, 2016. DOI: 10.1145/2987386.2987413.
This work was supported by the China Scholarship Council and State Key Laboratory of Traction Power at Southwest Jiaotong University (No. TPL1501). We thank anonymous reviewers for their helpful comments which helped to improve the paper.
Recommended by Associate Editor Jie Zhang
Xiao-Jing Niu received the B. Eng. and M. Eng. degrees in computer science from Northwest A&F University, China in 2013 and 2016, respectively. She is currently a Ph. D. degree candidate in industrial design of Department of Northumbria School of Design, Northumbria University Newcastle, UK.
Her research interests include human-computer interaction, computer aided design and machine learning.
Sheng-Feng Qin received the Ph. D. degree in product design from University of Wales Institute, UK in 2000. Now, he is a professor of digital design at Northumbria University with an extensive career in design academia. He is a member of IEEE and the Design Society. He has published more than 150 papers in journals and conferences and 2 books.
His research interests include computer-aided conceptual design, sketch-based interface and modeling, interface and interaction design, simulation modelling and virtual manufacturing, smart product and sustainable design, digital design methods and tools.
John Vines received the Ph. D. degree in interaction design for older people from University of Plymouth, UK in 2010. He is a professor at Department of Northumbria School of Design, Northumbria University, UK.
His research interests include human-computer interaction, methods and processes for participatory, collaborative and experience-centered design and research, experience-centered security and privacy.
Rose Wong received the BA degree in product design from Northumbria University, UK in 2000. She has worked as a senior lecturer on the Bachelor (Hons) 3D design and Bachelor (Hons) design for industry courses at Northumbria University, UK, and is currently the programme leader on Bachelor (Hons) design for industry course. Prior to this, she worked as a “Designer in Residence” at Northumbria University, UK, which operates as an in-house design consultancy employing successful students graduating from 3D Design. This venture has helped many of Northumbria University′s Design School graduates become successful freelance designers, demonstrating a nurture of entrepreneurial skills in our alumni.
Her research interests include in-house design and 3D design.
Hui Lu received the B. Sc. degree in business administration from Zhejiang University, China in 2009, and the M. Sc. degree in operations and supply chain management from the University of Liverpool, UK in 2010. Currently, she is an associated lecturer, research assistant and Ph. D. degree candidate in business management, Newcastle Business School, Northumbria University, UK. She once worked in the Integrated Supply Chain Department in IBM China for 4 years.
Her research interests include sustainable manufacturing and supply chain management, process optimal and control, and technology innovation.
About this article
Cite this article
Niu, X., Qin, S., Vines, J. et al. Key Crowdsourcing Technologies for Product Design and Development. Int. J. Autom. Comput. 16, 1–15 (2019). https://doi.org/10.1007/s11633-018-1138-7
- Crowdsourcing technologies
- product design and development (PDD)
- information sharing
- design evaluation