Key Crowdsourcing Technologies for Product Design and Development

  • Xiao-Jing NiuEmail author
  • Sheng-Feng Qin
  • John Vines
  • Rose Wong
  • Hui Lu
Open Access


Traditionally, small and medium enterprises (SMEs) in manufacturing rely heavily on a skilled, technical and professional workforce to increase productivity and remain globally competitive. Crowdsourcing offers an opportunity for SMEs to get access to online communities who may provide requested services such as generating design ideas or problem solutions. However, there are some barriers preventing them from adopting crowdsourcing into their product design and development (PDD) practice. In this paper, we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools, crowdsourcing frameworks, and techniques in terms of open call generation, rewarding, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process. We also explore the necessary techniques and tools to support the crowdsourcing PDD process. Finally, we propose some key guidelines for coping with the aforementioned challenges in the crowdsourcing PDD process.


Crowdsourcing technologies product design and development (PDD) communication information sharing design evaluation feedback 



This work was supported by the China Scholarship Council and State Key Laboratory of Traction Power at Southwest Jiaotong University (No. TPL1501). We thank anonymous reviewers for their helpful comments which helped to improve the paper.


  1. [1]
    E. Simperl. How to use crowdsourcing effectively: Guidelines and examples. LIBER Quarterly, vol. 25, no. 1, pp. 18–39, 2015. DOI: 10.18352/lq.9948.Google Scholar
  2. [2]
    Open Innovation Defintion, [Online], Available:, September 20, 2017.
  3. [3]
    A. Wichman. Challenges, Crowdsourcing, Citizen Science: What’s the Dfi? [Online], Available:, September 20, 2017.
  4. [4]
    Citizen Science: Definition, [Online], Available:, September 15, 2017.
  5. [5]
    J. Howe. The rise of crowdsourcing. Wired magazine, vol. 14, no. 6, pp. 1–4, 2006.Google Scholar
  6. [6]
    A. Kittur, J. V. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, J. Horton. The future of crowd work. In Proceedings of the Conference on Computer Supported Cooperative Work, ACM, San Antonio, USA, pp. 1301–1318, 2013. DOI: 10.1145/2441776. 2441923.Google Scholar
  7. [7]
    X. J. Niu, S. F. Qin. A review of crowdsourcing technology for product design and development. In Proceedings of the 23rd International Conference on Automation and Computing, IEEE, Huddersfield, UK, 2017. DOI: 10.23919/IConAC.2017.8081981.Google Scholar
  8. [8]
    Open Innovation & Crowdsourcing Examples, [Online], Available:, September 25, 2017.
  9. [9]
    Cocreate with Your Fans. [Online], Available:, September 26, 2017.
  10. [10]
    J. Jin, P. Ji, Y. Liu. Prioritising engineering characteristics based on customer online reviews for quality function deployment. Journal of Engineering Design vol. 25, no. 7–9, pp. 303–324, 2014. DOI: 10.1080/09544828.2014. 984665.Google Scholar
  11. [11]
    S. N. Pedersen, M. E. Christensen, T. J. Howard. Robust design requirements specification: A quantitative method for requirements development using quality loss functions. Journal of Engineering Design, vol. 27, no. 8, pp. 544–567, 2016. DOI: 10.1080/09544828.2016.1183163.Google Scholar
  12. [12]
    Open IDEO, [Online], Available:, September 26, 2017.
  13. [13]
    Herox. Where Innovators Come to Compete and Businesses Come for Solutions, [Online], Available: September 28, 2017.
  14. [14], [Online], Available:, September 28, 2017.
  15. [15]
    E.A.T. School Lunch UX Challenge, [Online], Available: September 28, 2017.
  16. [16]
    H. To. Task assignment in spatial crowdsourcing: challenges and approaches. In Proceedings of the 3rd ACM Sigspatial Ph.D. Symposium, ACM, Burlingame, USA, pp. 1–4, 2016. DOI: 10.1145/3003819.3003820.Google Scholar
  17. [17]
    R. R. Morris, D. McDuff. Crowdsourcing techniques for affective computing. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, A. Kappas, Eds., Oxford, UK: Oxford University Press, pp. 384–394, 2014.Google Scholar
  18. [18]
    J. Thebault-Spieker, L. G. Terveen, B. Hecht. Avoiding the south side and the suburbs: The geography of mobile crowdsourcing markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, Canada, pp. 265–275, 2015. DOI: 10.1145/2675133.2675278.Google Scholar
  19. [19]
    H. To, R. Geraldes, C. Shahabi, S. H. Kim, H. Prendinger. An empirical study of workers’ behavior in spatial crowdsourcing. In Proceedings of the 3rd International ACM SIGMOD Workshop on Managing and Mining Enriched Geo-spatial Data, San Francisco, USA, 2016. DOI: 10.1145/2948649.2948657.Google Scholar
  20. [20]
    S. Djelassi, I. Decoopman. Customers’ participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management, vol. 42, no. 5, pp. 683–692, 2013. DOI: 10.1016/j.indmarman.2013.05.006.Google Scholar
  21. [21]
    H. Carpenter. Motivating the crowd to participate in your innovation initiative. A Guide to Open Innovation and Crowdsourcing, pp. 76–84, 2011.Google Scholar
  22. [22]
    H. Simula, T. Ahola. A network perspective on idea and innovation crowdsourcing in industrial firms. Industrial Marketing Management, vol. 43, no. 3, pp. 400–408, 2014. DOI: 10.1016/j.indmarman.2013.12.008.Google Scholar
  23. [23]
    W. Mason, D. J. Watts. Financial incentives and the “performance of crowds”. ACM Special Interest Group on Knowledge Discovery and Data Mining Explorations Newsletter, vol. 11, no. 2, pp. 100–108, 2009. DOI: 10.1145/1809400.1809422.Google Scholar
  24. [24]
    J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the 5th International AAAI Conference on Web and Social Media, Barcelona, Spain, pp. 17–21, 2011.Google Scholar
  25. [25]
    J. Howe. Crowdsousrcing: Why the Power of the Crowd is Driving the Future of Business. New York, USA: Crown Business, 2009.Google Scholar
  26. [26]
    L. Von Ahn, L. Dabbish. Designing games with a purpose. Communications of the ACM, vol. 51, no. 8, pp. 58–67, 2008. DOI: 10.1145/1378704.1378719.Google Scholar
  27. [27]
    B. Jonathan Mausam, D. S. Weld. Optimal testing for crowd workers. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Singapore, pp. 966–974, 2016.Google Scholar
  28. [28]
    D. N. Chang, C. H. Chen. Product concept evaluation and selection using data mining and domain ontology in a crowdsourcing environment. Advanced Engineering Informatics, vol. 29, no. 4, pp. 759–774, 2015. DOI: 10.1016/j.aei.2015.06.003.Google Scholar
  29. [29]
    P. G. Ipeirotis, F. Provost, J. Wang. Quality management on amazon mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, USA, pp. 64–67, 2010. DOI: 10.1145/1837885.1837906.Google Scholar
  30. [30]
    S. Sedhain, S. Sanner, D. Braziunas, L. X. Xie, J. Christensen. Social collaborative filtering for cold-start recommendations. In Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, USA, pp. 345–348, 2014. DOI: 10.1145/2645710.2645772.Google Scholar
  31. [31]
    N. Holladay. Achieving quality through teamwork, [Online], Available:, August 15, 2017.
  32. [32]
    J. Pedersen, D. Kocsis, A. Tripathi, A. Tarrell, A. Weerakoon, N. Tahmasbi, J. Xiong, W. Deng, O. Oh, G. J. de Vreede. Conceptual foundations of crowdsourcing: A review of IS research. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 579–588, 2013. DOI: 10.1109/HICSS.2013.143.Google Scholar
  33. [33]
    Quirky, [Online], Available:, September 26, 2017.Google Scholar
  34. [34]
    99designs, [Online], Available:, September 26, 2017.
  35. [35]
    Jovoto, [Online], Available: September 26, 2017.
  36. [36]
    T. P. Walter, A. Back. A text mining approach to evaluate submissions to crowdsourcing contests. In Proceedings of the 46th Hawaii International Conference on System Sciences, IEEE, Wailea, USA, pp. 3109–3118, 2013. DOI: 10.1109/HICSS.2013.64.Google Scholar
  37. [37]
    T. P. Walter, A. Back. Towards Measuring Crowdsourcing Success: An Empirical Study on Effects of External Factors in Online Idea Contest, [Online], Available:, September 26, 2017.
  38. [38]
    J. Bjork, M. Magnusson. Where do good innovation ideas come from? Exploring the influence of network connectivity on innovation idea quality Journal of Product Innovation Management, vol. 26, no. 6, pp. 662–670, 2009. DOI: 10.1111/j.1540-5885.2009.00691.x.Google Scholar
  39. [39]
    G. Wang, X. M. Liu, W. G. Fan. A knowledge adoption model based framework for finding helpful user-generated contents in online communities. In Proceedings of the 32nd International Conference on Information Systems, Shanghai, China, pp. 1–11, 2011.Google Scholar
  40. [40]
    K. Aniket, E. H. Chi, B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the Special Interest Group on Computer-Human Interaction Conference on Human Factors in Computing Systems, ACM, Florence, Italy, pp. 453–456, 2008. DOI: 10.1145/1357054. 1357127.Google Scholar
  41. [41]
    A. D. Shaw, J. J. Horton, D. L. Chen. Designing incentives for inexpert human raters. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, ACM, Hangzhou, China, pp. 275–284, 2011. DOI: 10.1145/1958824.1958865.Google Scholar
  42. [42]
    A. B. Xu, H. M. Rao, S. P. Dow, B. P. Bailey. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 1637–1648, 2015. DOI: 10.1145/2675133.2675140.Google Scholar
  43. [43]
    P. Dai, J. M. Rzeszotarski, P. Paritosh, E. H. Chi. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Vancouver, Canada, pp. 628–638, 2015. DOI: 10.1145/2675133. 2675260.Google Scholar
  44. [44]
    P. Dai, Mausam, D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, AAAI, Atlanta, USA, pp. 1168–1174, 2010.Google Scholar
  45. [45]
    A. Mohammad, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, S. Dustdar. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing, vol. 17, no. 2, pp. 76–81, 2013. DOI: 10.1109/MIC.2013.20.Google Scholar
  46. [46]
    M. S. Bernstein, G. Little, R. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, D. Crowell. Soylent: A word processor with a crowd inside. Communications of the ACM, vol. 58, no. 8, pp. 85–94, 2015. DOI: 10. 1145/2791285.Google Scholar
  47. [47]
    D. Ofer, O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In Proceedings of the 22nd Annual Conference on Learning Theory, Montreal, Canada, 2009.Google Scholar
  48. [48]
    What Are the Best Tools for Crowdsourcing Ideas? [Online], Available:, February 15, 2018.
  49. [49]
    Crowdsourcing Tool, [Online], Available:, February 15, 2018.
  50. [50]
    S. F. Qin, D. van der Velde, E. Chatzakis, T. McStea, N. Smith. Exploring barriers and opportunities in adopting crowdsourcing based new product development in manufacturing SMEs. Chinese Journal of Mechanical Engineering, vol. 29, no. 6, pp. 1052–1066, 2016. DOI: 10.3901/CJME.2016.0808.089.Google Scholar
  51. [51]
    C. Y. Xu, S. F. Qin, Z. P. Xiao. Crowdsourcing based product innovation design service model for small-sized and medium-sized enterprises. In Proceedings of 18th International Conference on Automation and Computing, IEEE, Leicestershire, UK, pp. 1–5, 2012.Google Scholar
  52. [52]
    Product Design, [Online], Available:, October 20, 2017.
  53. [53]
    K. Ulrich, S. Eppinger. Product Design and Development. 6th ed. New York, USA: McGraw-Hill Higher Education, 2015.Google Scholar
  54. [54]
    J. L. Nevins, D. E. Whitney. Concurrent Design of Products and Processes: A Strategy for the Next Generation in Manufacturing. New York, USA: McGraw-Hill Companies, 1989.Google Scholar
  55. [55]
    Y. L. Tu, S. Q. Xie, R. Y. K. Fung. Product development cost estimation in mass customization. IEEE Transactions on Engineering Management, vol. 54, no. 1, pp. 29–40, 2007. DOI: 10.1109/TEM.2006.889065.Google Scholar
  56. [56]
    H. P. Shang, Z. X. Zhao. Integration of manufacturing services into virtual environments over the Internet. International Journal of Automation and Computing, vol. 1, no. 1, pp. 89–106, 2004. DOI: 10.1007/s11633-004-0089-3.Google Scholar
  57. [57]
    M. L. Gray, S. Suri, S. S. Ali, D. Kulkarni. The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-supported Cooperative Work & Social Computing, ACM, San Francisco, USA, pp. 134–147, 2016. DOI: 10.1115/DETC2002/DTM-34020.Google Scholar
  58. [58]
    W. Y. Song, X. G. Ming, Z. Y. Wu. An integrated rough number-based approach to design concept evaluation under subjective environments. Journal of Engineering Design, vol. 24, no. 5, pp. 320–341, 2013. DOI: 10.1080/ 09544828.2012.732994.Google Scholar
  59. [59]
    Z. Ayag, R. G. Ozdem. An analytic network process-based approach to concept evaluation in a new product development environment. Journal of Engineering Design, vol. 18, no. 3, pp. 209–226, 2007. DOI: 10.1080/09544820600752740.Google Scholar
  60. [60]
    E. Alex, N. Rahmati, N. Zhu. Structured handoffs in expert crowdsourcing improve communication and work output. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology, ACM, Honolulu, USA, pp. 99–100, 2014.Google Scholar
  61. [61]
    H. P. Zhang, R. Q. Zhang, Y. P. Zhao, B. J. Ma. Big data modeling and analysis of microblog ecosystem. International Journal of Automation and Computing, vol. 11, no. 2, pp. 119–127, 2014. DOI: 10.1007/s11633-014-0774-9.Google Scholar
  62. [62]
    L. Y. Zhai, L. P. Khoo, Z. W. Zhong. Design concept evaluation in product development using rough sets and grey relation analysis. Expert Systems with Applications, vol. 36, no. 3, pp. 7072–7079, 2009. DOI: 10.1016/j.eswa. 2008.08.068.Google Scholar
  63. [63]
    V. Girotto. Collective creativity through a micro-tasks crowdsourcing approach. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, ACM, San Francisco, USA, pp. 143–146, 2016. DOI: 10.1145/2818052.2874356.Google Scholar
  64. [64]
    Z. Ayag. An integrated approach to concept evaluation in a new product development. Journal of Intelligent Manufacturing, vol. 27, no. 5, pp. 991–1005, 2016. DOI: 10.1007/ s10845-014-0930-7.Google Scholar
  65. [65]
    V. Tiwari, P. K. Jain, P. Tandon. Product design concept evaluation using rough sets and VIKOR method. Advanced Engineering Informatics, vol. 30, no. 1, pp. 16–25, 2016. DOI: 10.1016/j.aei.2015.11.005.Google Scholar
  66. [66]
    Z. Ayag. An integrated approach to evaluating conceptual design alternatives in a new product development environment. International Journal of Production Research, vol. 43, no. 4, pp. 687–713, 2005. DOI: 10.1080/00207540512 331311831.Google Scholar
  67. [67]
    H. Z. Huang, R. F. Bo, W. Chen. An integrated computational intelligence approach to product concept generation and evaluation. Mechanism and Machine Theory, vol. 41, no. 5, pp. 567–583, 2006. DOI: 10.1016/j.mechmachtheory. 2005.07.006.zbMATHGoogle Scholar
  68. [68]
    T. L. Saaty. Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, USA: RWS publications, 1996.Google Scholar
  69. [69]
    D. N. Chang, C. H. Chen. Exploration of a concept screening method in a crowdsourcing environment. Moving Integrated Product Development to Service Clouds in the Global Economy, J. Z. Cha, S. Y. Chou, J. Stjepandic, R. Curran, W. S. Xu, Eds, Netherlands: IOS Press, pp. 861–870, 2014. DOI: 10.3233/978-1-61499-440-4-861.Google Scholar
  70. [70]
    W. Qi, Z. W. Ren, Z. F. Guo. XML-based data processing in network supported collaborative design. International Journal of Automation and Computing, vol. 7, no. 3, pp. 330–335, 2010. DOI: 10.1007/s11633-010-0511-y.Google Scholar
  71. [71]
    K. Gautam. Advanced Methods of Teaching “Feedback”. Pondicherry University, [Online], Available:, November 5, 2017.
  72. [72]
    P. L. Jackson. Getting Design Right: A Systems Approach. Boca Raton, USA: CRC Press, 2009.Google Scholar
  73. [73]
    Z. Y. Ma, A. X. Sun, Q. Yuan, G. Cong. Topic-driven reader comments summarization. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, Maui, USA, pp. 265–274, 2012. DOI: 10.1145/2396761.2396798.Google Scholar
  74. [74]
    J. Hui, A. Glenn, R. Jue, E. Gerber, S. Dow. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, AAAI, San Diego, USA, pp. 72–82. 2015.Google Scholar
  75. [75]
    A. B. Xu, B. Bailey. What do you think? A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, Seattle, USA, pp. 295–304, 2012. DOI: 10.1145/2145204.2145252.Google Scholar
  76. [76]
    S. Dow, E. Gerber, A. Wong. A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Paris, France, pp. 227–236, 2013. DOI: 10.1145/2470654. 2470686.Google Scholar
  77. [77]
    A. B. Xu, S. W. Huang, B. Bailey. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, Baltimore, USA, pp. 1433–1444, 2014. DOI: 10.1145/2531602.2531604.Google Scholar
  78. [78]
    K. Luther, A. Pavel, W. Wu, J. L. Tolentino, M. Agrawala, B. Hartmann, S. P. Dow. CrowdCrit: Crowdsourcing and aggregating visual design critique. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, Baltimore, USA, pp. 21–24, 2014. DOI: 10.1145/2556420.2556788.Google Scholar
  79. [79]
    S. Salimun, N. Janom, N. H. Arshad. Quality factors of crowdsourcing system: Paper review. In Proceedings of the 6th IEEE Control and System Graduate Research Colloquium, Shah Alam, Malaysia, pp. 82–86, 2015. DOI: 10.1109/ICSGRC.2015.7412469.Google Scholar
  80. [80]
    R. Khazankin, D. Schall, S. Dustdar. Predicting QoS in scheduled crowdsourcing. Advanced Information Systems Engineering, J. Ralyte, X. Franch, S. Brinkkemper, S. Wrycza, Eds., Berlin Heidelberg, Germang: Springer, 2012. DOI: 10.1007/978-3-642-31095-9—30.Google Scholar
  81. [81]
    R. Alabduljabbar, H. Al-Dossari. A task ontology-based model for quality control in crowdsourcing systems. In Proceedings of the International Conference on Research in Adaptive and Convergent Systems, ACM, Odense, Denmark, pp. 22–28, 2016. DOI: 10.1145/2987386.2987413.Google Scholar

Copyright information

© The Author(s) 2018

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.School of DesignNorthumbria UniversityNewcastle Upon TyneUK
  2. 2.Newcastle Business SchoolNorthumbria UniversityNewcastle Upon TyneUK

Personalised recommendations