Key Crowdsourcing Technologies for Product Design and Development

Traditionally, small and medium enterprises (SMEs) in manufacturing rely heavily on a skilled, technical and professional workforce to increase productivity and remain globally competitive. Crowdsourcing offers an opportunity for SMEs to get access to online communities who may provide requested services such as generating design ideas or problem solutions. However, there are some barriers preventing them from adopting crowdsourcing into their product design and development (PDD) practice. In this paper, we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools, crowdsourcing frameworks, and techniques in terms of open call generation, rewarding, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process. We also explore the necessary techniques and tools to support the crowdsourcing PDD process. Finally, we propose some key guidelines for coping with the aforementioned challenges in the crowdsourcing PDD process.


Introduction
Having benefited from technology promoted by Web 2.0 and smart mobile devices, it is convenient for Internet users to get access to the Internet and share information with others.In this context, the Internet users distributed all over the world (the crowds) show great potential in creating amazing contents available online, and it is easier for them to take part in various aspects of our society.For example, Wikipedia is a great success which has benefited from their continuous contributions.Inspired by this, an increasing number of companies intend to take their potential customers into their decision-making process related to product development, services and policies, or to seek help from the crowds in addressing some problems that they cannot solve because of the shortage of skilled employees and sufficient resources or user engagements, especially for small and medium enterprises (SMEs).In this process, the crowds need to interact with each other and the computer.How they interact is a research focus of computer-supported cooperative work (CSCW).There are a number of research terms related to crowd interactions [1] : wisdom of the crowds, open innovation, citizen science, collective intelligence, human computation, social computing, social machines and crowdsourcing.The comparison [2][3][4] of these terms is shown in Table 1.
From the comparison of these terms in Table 1, it is clear that crowdsourcing shares some features with these terms.Crowdsourcing was first coined by Jeff Howe in Wired Magazine as "the act of taking a job traditionally performed by a designated agent (individual, institution, non-profit organization or enterprise) and outsourcing it to an undefined, generally large group of people in the form of an open call" [5] .It focuses on the gathering, representation, processing and use of information.
Kittur et al. [6] proposed a crowdsourcing framework which encompasses the following research topics: task decomposition, incentive mechanisms, organization hierarchy of crowds, task assignments, communication and coordination, collaboration workflow and quality control.In each part, certain technology is adopted to ensure the execution of the crowdsourcing process and the overall quality of work output.In a previous literature review paper [7] , we have summarized the key techniques applied in incentive mechanisms, task assignments and communication.Here, we mainly describe technologies in other parts in more detail and discuss the technologies needed if applying crowdsourcing in product design and development (PDD) process.
Our main contribution is three-fold: 1) Summarize crowdsourcing techniques in terms of open call generation, incentive mechanisms, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality control.
2) Investigate the technology necessary in a crowdsourcing PDD process.
3) Propose key guidelines for coping with the aforementiond challenges in crowdsourcing PDD process.
The remainder of this paper is organized as follows: Section 2 briefly introduces the crowdsourcing process.The technology used in the crowdsourcing process is described in Section 3, including research findings in terms of techniques, framework, platforms and tools.Section 4 presents the technology needed for crowdsourcing PDD process, and Section 5 summarizes the review work and proposes key guidelines for coping with the aforementiond challenges in the crowdsourcing PDD process.

Key elements of a crowdsourcing process
On intermediary crowdsourcing platforms, whatever task it crowdsources, the crowdsourcing process consists of four key elements: the requester, the crowds, the task which needs to be crowdsourced and the crowdsourcing platform.The platform provides the requester a way to get access to large crowds conveniently and involve them into their production process and decision-making process.A simplified crowdsourcing process is shown in Fig. 1. "Interactions 1" mainly means task input and the feedback from the platform, while "Interactions 2" refers to the broadcast of a task to crowds and crowd′s submissions of the performed task (including other communica-tions with the platforms).
In the process, "Requester" refers to an individual or institution seeking help from crowds."Crowds" refer to a large group of people working on an internet-based crowdsourcing platform and they take on tasks that are advertised via an open call.Only when the crowdsourcing task is well-defined, then the proper crowds with specific knowledge and skills will be selected.The crowdsourcing task proposed by the requester needs to be mapped from the high-level goal to specific subtasks to be completed by the crowds.The crowdsourcing form depends on the nature of crowdsourcing tasks.Before crowdsourcing the task, an open call including the specific task and its evaluation criteria need to be defined first.The evaluation criteria can be provided by the requester directly or be collected through crowdsourcing.The related technology in each element will be presented in later parts.

Crowdsourcing forms
There are a lot of ways [3,8] to crowdsource a task, which are shown in Table 2.The crowdsourcing forms could be used for classifying crowdsourcing platforms.
In these crowdsourcing forms, micro tasks and macro tasks are classified by the granularity of tasks.Compared to macro tasks, micro tasks are highly parallelizable and can be divided into smaller pieces, which can be completed in seconds to minutes.Micro tasks are always the tasks that are simple and easy to be accomplished by hu- Fig. 1 A simplified crowdsourcing process mans but are challenging for computers, such as recognizing things in images.Macro tasks are difficult to be decomposed and the resolutions for macro tasks require much sharing of contextual information or dependencies to intermediary results.Other crowdsourcing forms, i.e., challenges, volunteer campaigns and contests, cannot be classified into the same category as neither can they be divided into micro tasks, nor can they depend on the context information and intermediary results.To some extent, product design belongs to macro task, but it is much more complex than that as product design is an iterative process and the final design is an evolutionary result of countless refinement and improvement based on the initial design ideas.

Crowdsourcing framework
The goal of crowdsourcing is to obtain desired highquality product within a given time scale and with as low a cost as possible.Based on the previous analysis, the crowdsourcing framework [6] which we used in our previous work [7] is for micro tasks.Regardless of the crowdsourcing forms, the crowdsourcing framework (Fig. 2) is shown as follows.
In the crowdsourcing process, the general task is usually performed by an individual crowd as the task is simple enough for individuals to tackle.As for complex tasks, the crowds may need to build his/her own team as in conventional setting.When they finish the task, they can submit their results to the platform for later evaluation.The output evaluation here is relatively simple, as the selection of the winning solution is achieved by the voting of crowds.For some tasks, the solution is assessed by a group of experts in a traditional way, e.g., the innovation process of Jovoto [9] .
In the proposed framework, the techniques mentioned are: call generation, incentive mechanisms, crowd qualification for working, the organization structure of crowds, solution evaluation, workflow and quality control.

Crowdsourcing techniques
Based on the crowdsourcing framework in Fig. 2, this part elaborates crowdsourcing techniques related to the mentioned aspects.

Call generation techniques
Call generation is the first step of the crowdsourcing process.Clear instructions provide a larger possibility of receiving high-quality responses from the crowds [6] .However, there is no related research discussing how to generate a good open crowdsourcing call (or challenge brief).In conventional working environments, Jin et al. [10] investigated how to prioritize engineering characteristics for quality function deployment and Pedersen et al. [11] proposed a quantitative method for requirements development using quality loss function.
In order to fill in this gap, we analyzed 14 cases from OpenIDEO [12] , Herox [13] , challenge.gov [14], devpost.com [15]nd 5 cases from Northumbria Design School and summarized 8 key elements of an open call that are shown as follows: 1) Description The text description should explain the task background and the goals of the task.If the task is to design a specific product or component, the text description should explain its features and functionality and how it will affect the task goals.
2) Timeline It lists the duration of each phase and incorporates it with the crowdsourcing process.
3) Submission requirements It tells the crowds which format of submission is acceptable.It may include a prototype or demonstration video, related text description, image and video requirements, language requirements, intellectual property, financial or preferential support, and so forth.
4) Judging and prizes Sometimes, the judges may be announced.Eligible submissions will be evaluated by a panel of selected judges.The judging criteria may not apply to every prize.Each prize has its own judging criteria that is more specific and concrete.

5) Criteria and disqualifications
The description of criteria could be derived from goals for the challenge and the target customer group.criteria may be in terms of the following aspects: a) User experience (UX) and design appeal, including the degree to which the design reinvents the user experience-focusing on utility, usability, intuitiveness, and design appeal.b) Effectiveness and efficiency.c) Functional properties, emotional and experiential qualities.d) Aesthetics, and practicality, and so forth.
6) Crowd qualification requirement It tells the crowds who can participate in the task (participation eligibility) and how to get involved.
7) Rules and regulations It includes goals for the task, resources to spur ideas and other additional information, such as reference information.

8) About the sponsor
In an open call, not all the eight elements are needed.For instance, when the task is open to all crowds, the crowd qualification requirements are not necessary.On most crowdsourcing platforms, the generation of an open call is generally by an interactive way of answering preset questions.

Incentive mechanisms techniques
Incentive mechanisms play an important role in motivating crowds to involve in performing crowd tasks.A lot of research [16][17][18][19][20][21] has been focused on the relationship between incentives and crowd participation.Crowds do not have to participate, since crowdsourcing systems or platforms are typically open to everyone and do not rely on contracts.Thus, certain measures must be adopted to compel crowds to participate.Otherwise, crowd tasks cannot be performed.The ways of attracting crowds could be categorized into two distinct categories: extrinsic (e.g., reward, building of their personal reputation, etc.) and intrinsic (e.g., enjoyment, being part of the common good, etc.) [22] .The most common three motivations are reward, enjoyment and reputation.
1) Reward Reward is the dominant motivation in the crowdsourcing processes.This includes cash bonuses, discount coupons, free use of product, and virtual money.For instance, Mturk and Figure Eight offer small financial rewards to an anonymous crowd engaged with tasks posted by various requesters.Generally, most crowds are moneydriven and higher pay would usually get more crowds to perform more tasks more quickly [23] .In some boring and tedious tasks, such as transcribing countless hand-written documents, monetary compensation must be guaranteed, or few crowds are likely to participate in the task.However, money is not necessary to enable the high quality of completed tasks, which was found by Mason and Watts [23] and Rogstadius et al. [24] .
2) Enjoyment In crowdsourcing processes, crowds usually choose the tasks that interest them.When crowds are really interested in them or love doing them, they would like to devote themselves to it, even if there is no reward.They are self-incentivised because of the feel of achievement brought by finishing challenging tasks and the opportunities to exercise skills and talents that they have no chance to use in their ordinary lives [25] .
Besides, some crowds are encouraged to engage in the task by a game-like environment [26] , where points, leaderboards and other common game elements are included.By playing tasks of this kind, they can relax themselves, as well as get some rewards.
3) Reputation A large proportion of crowds are driven to compete for the recognition by their peers or their personal values.Each time they finish the task, they will be scored or ranked by the server according to their work quality.If one has good reputation, it will be easier for him/her to be chosen to perform other crowd tasks later.
Reward, enjoyment and reputation are just three main incentives.Crowdsourcing platforms can adopt one or several of them according to different task types.

Crowds′ qualification techniques
Different crowdsourced tasks have different skill and qualification requirements for their participants, such as open to all, reputation-based and credential-based.Generally, complex and domain-specific tasks have higher requirements for their participants.
However, crowdsourcing platforms know little about their users′ (crowds′) expertise and skills, they can only rely on the matching between the requirements from the requester and the profiles or participation histories of crowds to judge whether a crowd is suitable or not to participate in a specific task.Since the profile is provided by the crowds themselves, there is a possibility that some crowds fill in information that is not matched to their actual capabilities.Due to the anonymous nature of participation on a crowdsourcing platform, some techniques are used to avoid such cheating behaviors of the crowds.The most often adopted measure is verification questions (gold questions) [27,28] that are inserted to test the performance of crowds.However, it is only useful in micro tasks.In other forms of crowdsourcing tasks, it is useless.
Besides cheating behaviors, some crowds do not perform the task carefully as a result of poor platform control of the submissions.In order to ensure the quality of work submitted by crowds, worker agreement is usually signed before participating in the task.This measure is adopted by most of the crowdsourcing platforms, but it works little on the final output.The core problem here is the identification of high-quality work, which is a research question in quality control as well.In order to assess a worker′s quality, Ipeirotis et al. [29] seamlessly integrated the gold data for learning the quality of the crowds and their algorithm can be applied when the crowds are asked to answer a multiple choice question to complete a task.
A promising approach for selecting qualified crowds is to identify their strong points by analyzing their work ex-perience and behaviors, thus ensuring the reliability of their submissions.Although this method is powerful, the cold-start question [30] that is common in recommendation system cannot be neglected.

Organization structure techniques
In conventional settings, reasonable organization structures benefit cooperation among employees, decision making and quality control [31] .Generally, organizations adopt hierarchical structure as their management strategy, which could enable groups of employees to tackle complex tasks and increase work efficiency.
In the context of crowdsourcing, hierarchical structure is the most popular organizational structure.Since the crowds have various professional skills and experience, they are good in one or some specific domains, but not in others.As a result, they play different roles in different crowdsourcing tasks.In a hierarchical structure with many layers, the position of a crowd depends on his or her capability.The hierarchical structure is more suitable for performing micro tasks.The crowds at the lowest level perform subtasks with the smallest granularity, while those at higher levels integrate the results submitted by the crowds at the lower layer.
As for tasks that are not easy to decompose, the hierarchical structure is useless as all employees work collaboratively targeted at the same goal and their work may have dependency to others.Take software design and development, which usually consists of various functional modules, as an example.On the whole, the task of software design and development is a micro task, but when focusing on the lowest level of decomposition (module), it is a macro task, as each module is still complex and cannot be decomposed anymore, which will be realized by the collaboration of a group of individuals with various specialities.On this occasion, a team structure should be more effective and efficient.Thus, the traditional team structure can be applied in the virtual environment if it can be well organized and controlled.

Solution evaluation techniques
The evaluation of submitted solutions is a necessary step in the selection of better solutions, and it is also an important way to ensure the quality of submissions.It reduces the volume of the alternative solutions for the winning one [32] .However, the solution evaluation on general crowdsourcing platforms, such as quirky [33] , 99Designs [34] and Jovoto [33] , are relatively simple, as the key ways for selection are crowd voting [33,34] and assessment of a group of experts [35] or the combination of these two methods.Crowd voting effectively reduces the number of solutions, and expert assessment rates submissions, selects winners and adjusts rewards [36] .
The existing evaluation approaches include the scale of every submission on a five-star rating [37] , and the evaluation data from external experts [38] .However, these methods are not sufficient to evaluate the submitted solutions and a multi-attribute scaling including ratings from both experts and crowds should be more reliable.Generally, the submissions on a crowdsourcing platform usually increase rapidly in both volume and complexity.In order to reduce evaluation time and cost, a text-mining approach [36,39] was applied in evaluating submissions from crowds.Although the text-mining approach is effective in reducing the volume of submissions, it does not work when the submission is presented in other formats, such as image and animation.

Workflow management techniques
Workflow management plays an important role in achieving high-quality output.It concerns where the data comes from and where it goes, as well as the integration of data steams coming from various sources.It is affected by many factors, such as the organizational structure of crowds, the volume of submitted solutions and task integration mechanisms [32] .In micro tasks, subtasks can be accomplished in parallel and the independent output can be aggregated through voting or majority rule [6] .But macro tasks have dependencies and require multiple types of expertise, sometimes, their requirements change with the progress of the crowdsourcing process.In this circumstance, workflow management is urgently needed.Aniket et al. [40,41] has found that enabling more complex workflows can result in large differences in output quality even with small differences in rewards and task order.
Generally, the structured workflow [42] is usually used to provide interpretative and diverse feedback.Besides, Dai et al. [43] improved crowdsourcing workflows by microdiversion.They provide timely relief to crowds and hope to improve productivity by retaining crowds to work on their tasks longer or to improve the work quality.They also used decision theory to model the iterative workflows and defined equations that govern the various steps of the crowdsourcing process [44] .Kittur et al. [6] indicated that existing workflows should be improved on a large space of parameters, instructions, incentives and decompositions so that they can be able to support the execution of complex tasks.

Quality control techniques
Quality control is an issue throughout the whole crowdsourcing process and it has received the most attention so far [45] .Aniket et al. [40,46] found that 30% or more of submissions on MTurk may be low quality.Since verifying the quality of a large pool of submitted results is hard, some crowds may submit answers with low quality [29] .Thus, quality control of the crowdsourcing process plays an important role in obtaining high-quality output.The quality control approaches could be classified into two categories [45] : design-time and runtime.The design-time approaches include the open call generation and crowds′ qualification, as described previously.There are a lot of runtime quality control approaches.For example, workflow management, expert review, output agreement, ground truth and majority voting [45] .These approaches can be adopted together for better quality as using one approach alone may contribute to cheating behaviours.For instance, when using output agreement, if independent workers give the same response as the received input, their output is accepted as a quality answer.But, Kittur et al. [6] found that some crowds may agree to coordinate answers, which made this method useless.Besides workflow management, four other approaches are used to filter out poor quality work after results have been submitted.In ground truth, the submitted results can be compared to a gold standard, such as known answers or common sense facts.For instance, Figure Eight relies on its gold standard to judge the answers submitted by crowds.However, gold standards may not be possible for subjective or creative tasks (e.g., designing a new product).Three other methods measure the quality of a submission according to how well that crowd agrees with others [29,47] or according to the crowds′ votes.
All these seven aspects work together to achieve higher quality of submissions rather than working separately.

Crowdsourcing platforms
This research only focuses on online crowdsourcing platforms.Crowdsourcing platforms connect requesters with crowds and shape the practical interactions between them [6] .Table 2 shows the types of crowdsourcing platforms.The general phases of these platforms′ work process are as follows: ideas, review, refinement, final review, top ideas, awards and impact.However, these platforms only support relatively simple and independent tasks (e.g., idea generation, challenge solution and so forth) from the start to finish or graphic designs from idea to realization.For complex tasks that need cooperation from a group of individuals, such as product design, these platforms seem to be helpless.
Most of the crowdsourcing platforms are developed for various applications [8] , such as challenge solving, idea generation and graphic designs.Quirky is a crowdsourcing invention platform where great ideas from general people could come into reality [33] and an e-commerce platform as well.Through it, the produced products would be sold.OpenIDEO and Innocentive are examples of crowdsourcing platforms for making ideas grow, while Crowd-Spring, 99Designs and DesignCrowd provide design options for selection from the requester.Other crowdsourcing platforms, like Figure Eight, are used for data processing and analysis.Regardless of application scenarios of the crowdsourcing platforms, the tasks performed on it are simple and independent.In order to support the crowdsourcing process of complex tasks, the platform needs to be improved in some aspects, e.g., solution evaluation, the communication and cooperation among crowds.

Crowdsourcing tools
Crowdsourcing platforms integrate tools to help re-questers realize specific purposes, such as new inventions, innovations and products.Since the crowdsourcing process is different because of various crowdsourcing tasks, certain tools are needed in various crowdsourcing phases.In the previous part, a general crowdsourcing process is given.As the process is iterative, some phases can be realized by the same tool.For example, the review and the final review phase can be realized by the same tool.Referring to various crowdsourcing phases, the classification of crowdsourcing tools is shown in Table 3.
As social media can only realize relatively simple and independent purposes like idea gathering and comments, crowdsourcing platforms that integrate various tools are easier for requesters to use.In addition, crowdsourcing platforms provide better services for managing mass data collected from crowds and the requester can achieve his/her aim with just a few clicks of the mouse.Besides the aforementioned tools, other tools like an assistive tool that help the requester to input their needs and evaluation criteria to generate an open crowdsourcing call and providing rewards to the winning designer are needed as well.
On a crowdsourcing platform, all these tools work in sequence to ensure the execution of crowdsourcing processes rather than working in isolation.These kind of crowdsourcing platforms are useful for general simple tasks, but they seem useless when handling complicated design tasks.

Research background for crowdsourcing PDD
Traditionally, manufacturing SMEs rely heavily on a skilled, technical and professional workforce to increase productivity and remain globally competitive.Crowdsourcing [5] offers an opportunity for SMEs to get access to online communities who may provide requested services such as generating design ideas or problem solutions.Qin Quora [48] , the category of idea platforms in [8], ideascale [49] , social media like WeChat, Facebook, Email, etc. Top ideas et al. [50] explored the challenges and opportunities in adopting crowdsourcing in new product development in manufacturing SMEs and found that crowdsourcing-based product design and development is interesting to many SMEs but there are some barriers preventing them from adopting crowdsourcing into their new product development practice.For example, how to achieve good quality of product design over a crowdsourcing platform is one of the main concerns for many SMEs.Xu et al. [51] have investigated a service model for product Innovation design.Differently, here, we analyse the existing platforms for designs and explore the possible processes and framework of crowdsourcing PDD and the necessary techniques and tools.

Analysis of existing platforms for designs
Among crowdsourcing platforms, those for designs could be classified into two categories: graphic design supported and product design supported.The typical crowdsourcing platforms for graphic designs are 99Designs and crowdSPRING.Taking 99Designs as an example, it works as follows [34] : a) Build a design brief; b) Pick a design package; c) Launch the design contest; d) Receive dozens of designs; e) Give feedback; f) Pick your favorite and get the full design copyright.On this platform, any internet users who are registered on it can participate in the contest and contribute their thoughts and ideas.crowdSPRING has a similar work process to 99Designs.
Jovoto is an open innovation platform, but it could be used for product design [35] .Its work process for product design is: a) project definition; b) brief creation; c) talent matching; d) project directing & guiding; e) results presentation; f) transfer of rights.In this process, all participants are professional designers.They help the customer (the requester who submitted his business goals in the project definition stage) to create the design brief and give feedback to the designers to shape their ideas.Jovoto can support the PDD process, but in the project directing & guiding stage, it seems that the professional designers evaluate the product designs in a conventional way.

PDD process on a crowdsourcing platform
Before introducing the crowdsourcing PDD process, the general PDD process in conventional working environment is presented first.It starts with identifying customer/market needs, establishing product design specifications, conducting product concept design and creating detailed designs [52] .
Referring to Ulrich and Eppinger [53] , a product concept design process is shown in Fig. 3, involving product concept generation, concept evaluation, selection and feedback in an iterative form.This design process starts from market research to identify the customer needs and then establishes a design brief (or product design specification (PDS)).Based on the PDS, a wide range of product concepts can be generated and then evaluated with reference back to the PDS and even the customer needs.After that, a design decision is made to select good concepts for further development in the next stage, or the discarding of bad ones, or feedback is provided based on the evaluation to improve the good concepts.The feedback can be provided to guide the concept improvement, or even to guide the PDS update.This is a typical iterative concept development process.The process does not stop, until otherwise, one or several concepts are accepted.In this process, the activities of concept generation, evaluation, selection and feedback are progressed in a loop and the feedback guides the next design cycle.In the product concept generation stage, the designers can perform product concept design individually or in a collaborative team.After concept evaluation, the designers can improve their designs referring to feedback from the evaluation results.Previous research [54,55] found that about 80% of overall product quality and up to 70% of the development cost is committed in the early stages of design for concept generation and improvement.Thus, in crowdsourcing PDD processes, more attention should be paid to the design evaluation and feedback that are not supported enough by general purposed crowdsourcing platforms.
When designing a specific product on a crowdsourcing platform that supports PDD, a team consisting of crowds with various skills and experience work collaboratively.Since product design heavily depends on information sharing and intermediary results, the crowdsourced PDD processes need to put more emphasis on communication and information sharing, design evaluation and integration with evaluation results during the design process.The cooperation effectiveness plays a vital role on the output quality.When PDD is executed on a crowdsourcing platform, the crowdsourcing-based product design process is summarized in Fig. 4 with reference to [50, 53].It indicates the main activities and challenges during the process.
The steps of the crowdsourcing PDD process are shown as follows: S1.The requester (e.g., a design project manager), defines product design tasks (challenges) through a form of product design brief (e.g., PDS) or a design challenge brief and sets the evaluation criteria and crowdsourcing conditions/terms.S2.The design challenge is broadcast online via a crowdsourcing platform and crowds registered on the platform are invited to perform the product design task.
S3. Once a crowd agrees to take on the product design task, he or she can work on product concepts individually or in a group ongoing formed on the platform.They can also work either online on the platform or offline.
S4.At some points, product concept design results will be submitted by a individual crowd or a group of crowds to the platform for evaluation.The concept design results can be submitted with a concept presentation or design pitch document to better communicate or explain the designers′ ideas or rationales.
S5. Submitted product design concepts will be evaluated via suitable approaches such as automatic evaluation based on machine intelligence or crowdsourcingbased methods.
S6.After concepts have been evaluated, better design concepts will be shortlisted based on the evaluation results and the corresponding evaluation results will be summarized to generate constructive feedback that will be communicated back to the related designers, thus guiding them to improve their designs and entering the next loop of the product concept design process.
Referring to Fig. 4, in order to involve crowdsourcing in the PDD process while ensuring better output quality, the following challenges need to be addressed: Challenge 1.How to organize a group of crowds and what measures need to be adopted to ensure the communication and cooperation among crowds when they work together on the same product.
Challenge 2. How to support the crowds (or designers in the context) to effectively present their design concepts in order to evaluate them fairly and decrease the possibility of missing some good concepts or rationales.
Challenge 3. Given a design concept presentation, how to evaluate it, and what is the best concept evaluation method on a crowdsourcing platform.
Challenge 4. If a design is evaluated by crowds (design peers, experts and other stakeholders), how to integrate the evaluation results with weightings from a large number of crowd evaluators and provide feedback to the concept designers to improve their design.

Framework of crowdsourcing PDD process
The framework of the crowdsourcing PDD process is transformed from the general one.Compared to the general one, the communication and cooperation among the crowds, the evaluation of product design and feedback play a more important role in the crowdsourcing process.The framework is shown in Fig. 5. Differing to the general crowdsourcing process, the PPD one pays more attention to design evaluation and provides feedback to corresponding designers as product design is an iterative process.Thus, the techniques for communication and information sharing, product design evaluation and the integration of evaluation results need to be investigated.

Communication and information sharing
Communication and information sharing play an important role in a crowdsourcing PDD process, which is also demonstrated to be important to virtual manufacturing and collaborative design [56] .In the process, all participating crowds must be well-organized so that they could collaborate effectively.Recently, Gray et al. [57] argued that social interaction is a basic need for human beings and this social need must be fully addressed by crowdsourcing platforms.Their research findings indicated the inevitability of collaboration among crowds [58,59] and the significance of combining collaboration with crowdsourcing workflows.
PDD processes always involve multiple stages, which need to be completed by distributed teams or individuals with professional skills and experience.However, the distributed crowds have always faced with challenges in cul-tural differences and coordination [6] .Effective communication approaches could enable crowds to spend less time understanding their tasks and improve the work efficiency.
In order to improve communication and work output in expert crowdsourcing, Alex et al. [60] investigated a structured handoff method where participants were asked in live (live conference and screen share are used) and recorded scenarios (short screen capture video with voiceover) respectively.Their experiments indicated that higher work quality could result and the adherence to the original intent could be increased by the structured handoff approach.Since crowds are located at various places and they are not available to participant in the task at any time, the structured handoff may be useful for them to know the working process.
Generally, discussion forums, blogs and microblogs (e.g., SinaWeibo [61] ) are commonly used by crowds as their communication medium, which is not real-time and may lead to some delays.Also, such kind of communication is not suitable in a large scale [62,63] .Social media, like Facebook, Twitter and WeChat, are real-time, but they only support the sharing of information and asynchronous edition of documents.In order to satisfy the increasing need of synchronous collaboration, tencent instant messenger (TIM) is developed as a free cloud-based and platform independent office software that not only supports instant messaging and the synchronous edition of simple documents, such as Word and Excel, but also integrates social interaction functions.However, when it is applied to product design and development, the platform can only support the sharing of documents in various formats, but it is inconvenient for users to view and edit them unless corresponding software or tool is installed.

Product design evaluation
Product concept evaluation is an important activity in the PDD process [64,65] .Traditionally, firms depend on their internal designers to review and evaluate design concepts.Better design concepts can be selected with internal designer′s design knowledge and experience.However, this activity usually involves a small number of product concepts, thus when the number of concepts is increased dramatically in a crowdsourcing environment, it may be time-consuming and needs alternative ways to do it.
In order to evaluate concepts more efficiently, a lot of automatic approaches have been developed to perform this task by utilizing the indicators and judgement proposed by designers.These approaches could be classified into two categories: numerical and non-numerical [62,64,65] .Non-numerical methods are simple and graphics-based, and they are easy to use to select design concepts.However, these approaches cannot effectively deal with uncertain, vague and subjective judgement from decision makers.As for numerical approaches [57][58][59][60][66][67][68] , they could support both quantitative and qualitative judge- ment of design criteria. One liitation of these methods is that it is difficult to quantify the design criteria and indicators accurately during early design stages.Until now, little research has focused on design concepts evaluation in the context of crowdsourcing.Chang and Chen [28,69] were the first to address this problem.However, their research only focused on the data-mining based approach.In their research [28] , domain ontology is adopted to hierarchically represent the types, properties and interrelationships of design concepts in order to better support the selection of promising design concepts.Differently, Qi et al. [70] focused on presenting product design information with extensible markup language (XML), thus enabling the data integration, sharing and exchange in later design stages.The structured representation of product design concepts can effectively decrease the time used on understanding and evaluating the design concepts.

Integration of evaluation results
After evaluating product designs, these designs can be ranked according to their scores obtained in the design evaluation phase, thus a list of top designs can be selected.For the shortlisted concepts from the selection process, summarized feedback from evaluation results needs to be provided to the corresponding designers for further refinement and development.The feedback can motivate designers and improve productivity.The feedback to designers might be provided for future engagement and better interaction.Content of feedback can consist of four different categories [71] : descriptive, effective, evaluative and motivational.Each category has a specific purpose.
After evaluation, the obtained evaluation results need to be integrated under the four categories before they are prepared for feeding back to the designers.However, how to summarize concept evaluations from various evaluators in different media forms into a brief and clear feedback statement is a big challenge.Jackson [72] provides a method called Sticky Notes for summarization of large numbers of comments and for small numbers of comments (say 50 or less), he suggests to use MS Excel to organize them into categories.Besides, text clustering methods maybe useful.For example, Ma et al. [73] proposed two models to group comments into topic clusters and yield better intra-cluster topic cohesion.
For crowdsourcing applications, there is a lot of research about how to produce high-quality feedback.For instance, Hui et al. [74] have adopted techniques including anonymity and communal efforts to improve the quality of feedback from crowds.In order to address superficial and disorganized feedback [75,76] from unknown members, previous work [77,78] has also created tools to support structured feedback online.During the product design process, if evaluation results and the corresponding feedback are well-structured, it is more helpful for designers to improve their designs.

Quality control techniques
The emphases of crowdsourcing PDD process have higher requirement for workflow management that ensure the fluent execution of the crowdsourcing process.
Based on [31, 40, 45, 79-81], we summarize the factors that influence product design quality in Fig. 6.It is clear that the final design quality is affected by the generated product concept′s quality and enhancement quality.The product concept′s enhancement quality is ensured by product design evaluation and feedback.The integrated feedback can enhance the generated concepts and promote the design process to the next loop.
Besides the aforementioned quality control techniques, the design evaluation and feedback techniques influence the design quality as well.The existing techniques for these aspects have been presented previously.But in the context of crowdsourcing, new techniques need to be explored in order to address the challenges.

Tools needed in crowdsourcing PDD process 4.5.1 Collaborative design tool
In PDD, design is generally performed by a team of professional designers located in different places, thus a collaborative design tool needs to be provided to help them work together and monitor the design process and progression.The tool provides a virtual workspace for the crowds in a team.

Design presentation tool
When the design is finished, it needs to be submitted to the platform for later evaluation.The evaluation results will be in feedback for further refinement and improvement.In order to describe the product design briefly and clearly, the tool should generate a presentation file by integrating together the common file formats, such as jpg, txt, audio and flash.

Design evaluation tool
After submitting the design presentation file, it is ready for later evaluation.It will be sent together with the evaluation criteria to proper crowds for assessment.The tool will generate an evaluation template with reference to product design specifications and even user needs, crowds only need to fill their evaluation results in the evaluation template, and then submit their evaluation results.

Integration tool of evaluation results
The integration tool can extract and classify evaluation results into various categories, thus reduce the heavy burdens of the designers from reviewing large numbers of evaluation results from the crowds.

Assessment of crowdsourcing PDD process
In order to measure the effectiveness of involving crowdsourcing in the PDD process, the product design obtained on a crowdsourcing PDD platform will be compared to the one accomplished by traditional methods in terms of cost, time duration, performance, ergonomics, aesthetics, safety, reliability, etc.

Discussion and conclusions
This paper analyzes the framework, platform, tools and techniques used in crowdsourcing processes in terms of open call generation, rewards, crowd qualification for working, organization structure of crowds, solution evaluation, workflow and quality Here, we propose a framework for applying crowdsourcing in the PDD process and investigate what techniques and tools are needed in the process while indicating the main challenges.Mainly, collaborative product design in virtual environment, communication and information sharing, design evaluation and feedback generation by integrating evaluation results are four key challenges in the PDD process.
Although specific tools supporting functions similar to activities in product design process, such as communication, have been developed, they are still not well integrated by a crowdsourcing platform to support the activities of product design and development.Meanwhile, the successful integration of crowdsourcing and product design process will offer a possibility for SMEs to get access to a large pool of crowds with various skills and experience, which can effectively overcome their difficulties on the shortage of skilled employees and related resources.In order to deal with these challenges, a crowdsourcing platform that considers all these challenges needs to be developed.Here, we propose some key guidelines for the development of such a crowdsourcingbased collaborative design platform: 1) The platform should be cloud-based.Therefore, crowds can access, edit and share related documents any-time and from anywhere.From the cloud-based workflow, they can make updates in real-time and have a full visibility of their collaborations.
2) The platform should be user-centred.The platform can guideline the crowds perform tasks including product design and evaluation, while providing a comfortable and satisfactory user experience.
3) The platform should integrate a communication tool that supports both private chats (one to one) and group meetings while sharing the related design documents.
4) An assistive design tool or specified design software should be provided in order to ensure crowd participants can view and edit the design documents.
5) The platform can be integrated with blockchain technology to ensure the trustworthiness of crowd contributions and effective protection of intellectual property (IP).Since the crowdsourcing process is open to the crowds who have been registered on the platform, the IP protection faces more risks than in traditional environment.It also benefits the selection of qualified crowds as all design experience on the platform can be retrieved and their actual skills and capabilities have been verified by previous design tasks he/she takes part in.
6) The platform should provide application programming interfaces (APIs) to common social media so that the platform user could invite his or her trusted friends with specific capabilities and experience to the platform to take part in product design activities.If participants are all trust-based, it is more likely to yield satisfactory design results.
7) The platform should provide a tool that can help the crowds evaluate design concepts.As both product design and the evaluation are subjective activities, it is hard to judge automatically whether the design satisfies the design requirements and needs or not.Therefore, a method combining automatic calculation of quantitative variables with manual evaluation of qualitative variables would be a better choice.In the assessment of qualitative variables, crowds are employed to extract relevant information about how design requirements and needs are expressed in the design concept and then assess them.Then design experts verify the evaluation results.
8) The platform should provide a tool that can classify evaluation results into different categories according to evaluation criteria.The evaluation results in the same category can be analysed by clustering approaches so that the similar information will be given back to corresponding crowds only once.helped to improve the paper.

Open Access
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/),which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Fig. 3
Fig. 3 Iterative product design process of consumer product design

Fig. 4
Fig. 4 PDD process on a crowdsourcing platform

Table 2
Ways of crowdsourcing a task

Table 3
Classification of crowdsourcing tools according to various crowdsourcing phases