Introduction

The design process has received attention recently because of its strong relationships with the innovation process (Verganti 2009). The literature contains at least two key arguments concerning this relation about design and innovation. The first relates to the utilitarian meaning of economics: “useful” innovation is not only welcomed but sought after, and design is a path towards innovation. The second argument is related to creativity: innovation is a creative activity, not only in the productive sense of economic value but also in an artistic sense (i.e., originality) (Godin 2014). Baregheh et al. (2009) defined innovation attributes as something new or even improved, which involves changes and phases of creation, generation, implementation, development, and adoption. In the literature, many authors describe innovation as a multi-step process whereby organizations transform ideas into new and improved products, services, or processes (Adams et al. 2006). In the case of a creative product, design plays a major role in defining the physical form of the product to better satisfy customer needs (Ulrich et al. 2011). However, when discussing a creative product, we reference the creative design process, which was proposed to integrate the engineering design process with the cognitive creative process (Howard et al. 2008). Creativity and design can both be linked to innovation as the first contribution to the expansion of available ideas. Moreover, creativity and design can be linked to increasing the chance of successfully commercializing these ideas (Hollanders and Cruysen 2009). In this scenario, Bitard and Basset (2008) described design as creativity deployed to a specific innovation. However, design also corresponds to a structured process that transforms creative ideas into tangible products, services, and systems. Thus, it links creativity to innovation (Hollanders and Cruysen 2009). Consequently, design is an essential part of the improvement or development of new products in almost all industries (Sheldon 2004).

In practice, design can be considered as an integrative activity that, in its broadest sense, exploits the knowledge of various fields and/or disciplines to accomplish distinct outcomes (Margolin 2000). Similarly, design processes can be considered as creativity channeling, which is a difficult task that involves the subjectivity of creativity. Nevertheless, design can suffer from several constraints that reduce the designer motivation. Consequently, innovative creativity and product development should remain in equilibrium, where the constraints should not be so large that creativity is stifled (Swann 2005).

From a wider viewpoint, some researchers considered design as a science (Archer 1964; Hubka and Eder 1992; Cross 2001), where various well-defined elements interact, including problem-solving, decision-making, creativity, heuristics-searching, evolution, learning, negotiation, knowledge, optimization, organization, and satisfaction of needs. Generally, all elements are necessary but not sufficient for a good design. It is accepted that design methodologies and related methods combine the elements to guide designers in product creation (Archer 1979). A considerable amount of research in recent years has focused on proposing systematic models of the design process, with suggestions for creating more robust and efficient methodologies and/or structured approaches (Cross 2001; Manzini and Vezzoli 2003). There has been a slow but constant growth of empirical investigations to understand how design projects are linked to action, creation, and application search and results. As noted by Margolin (2000, p.2) “Due [to the fact] that design investigation does not only refer to products but also to human response, research techniques for design must necessarily be diverse.” Unfortunately, a complete analysis considering human decisions is not always suitable in the design research field, owing to the time and resource requirements to develop a complete analysis of the design process (Cross 2001). From educational viewpoint, some studies tried to relate positive reported of design process by creating systematic practices of designing and delivering instructional products and experience (Edmonds et al. 1994); among these field of knowledge, instructional design is useful to determine the state and needs of the learner, defining the end goal of instruction. In other words, instructional design aims at assisting in the transition of unknown field of knowledge by using instructions (Klein 1989). According to Gustafson (1997), instructional design models can be split in three categories: (i) classroom focused; (ii) product focused; (iii) System focused. From design perspective, the instructional design process is study by following a design-protocol analysis, which involves attempting to understand how design processes are generated in an empirical way and thereby developing models, methods, and tools that enable product creation (Cross 2001). According to Sentz et al. (2019), the creation of instructional strategies and related models arise from cognitive load theory, which emerge in activities that are considered highly complex. In this sense, protocol analysis during the product development process allows to transform the empirical activities in an appropriate knowledge with scientific viewpoint that can be useful for both companies and universities.

It is interesting to note that instructional design and design protocol claim difficulties for scholars and practitioners to make any meaningful comparison between models or assess the models against a useful standard (Edmonds et al. 1994). Indeed, in literature this is not an argument for creation of more models of instructional design, but rather, a call for appropriate ways of comparing and measuring aspects of design process. In this formulation, it becomes of primary importance the willingness of common instruments for measuring different tasks and activities. With this premise, this study aims at researching about design process predictability and measurability by following a systematic approach. However, before introducing the methodology and a case study, a brief overview of the context in which this work takes place, for outlining of the premises of this paper.

Framework of instructional design models in design field

In broad sense, instructional design is a type of problem-solving in itself (Jonassen 2000), at the same time, design methods can be considered as a way of representing the process in which designers do their work (Jonassen 1997). In other words, design methods can be considered as instructional design model how designers implement their theory and research in practice. With this view, Cross and Roozenburg (1991) classified design methods into two groups: descriptive and prescriptive. Descriptive methods describe the sequence of activities occurring during design (Reigeluth 2013; French 1985). Prescriptive models, as suggested by their name, prescribe a pattern of design activities, as shown in the works of Archer (1965); Jones (1984); Pugh (1991); Pahl et al. (2007); Ulrich et al. (2011); and others. For a review, see Riba and Molina (2006). Initial attempts to develop prescriptive methods focused on characterizing the PDP, which is a general description of the essential phases in the design process. The next subsection illustrates the differences and similarities of the various approaches.

Descriptive and prescriptive design methods

According to the analysis of Cross (2008), the activities performed by designers and engineers during the PDP can be categorized into four different phases: exploration, generation, evaluation, and communication. These phases are organized in sequences, iterations, and loops, according to the heuristics of the design process. The descriptive model proposed by French (1985) consists of four phases: problem analysis, conceptual design, schemes elaboration, and detailed design. Creative processes are also described according to four basic stages (with the understanding that individuals do not necessarily progress through the stages in a unidirectional or stepwise manner) (Amabile 1983, 1996). The first stage is problem identification. During this stage, the problem solvers recognize, define, and attempt to understand the problem or opportunity. The second stage is preparation, where problem solvers gather information and other resources necessary to address the problem or to pursue an opportunity. The third stage is response generation, where several ideas for problem solving or opportunity pursuit are designed. The fourth stage—validation and communication—involves consideration of generated ideas, selection of suitable ideas, and formalization or communication of the selected approach (Amabile 1996). The four creativity stages can be observed throughout the design process, but they are shaped in one specific design phase: conceptual design. Conceptual design is a part of the design process where the basic solution path is laid out via elaboration of a solution principle by identifying the essential problems through abstraction, establishing function structures, searching for appropriate working principles, and combining these into a working structure. A solution concept is a description of the form, function, and characteristics of a product, and it is usually accompanied by a set of specifications, a product analysis, and economic justification. A solution concept is usually expressed as a sketch or as an approximate three-dimensional model. It is sometimes accompanied by a brief description (Ulrich et al. 2011).

Pahl et al. (2007) improved the characterization of the design process using prescriptive models for design, identifying four phases in the PDP. Pahl et al. (2007) proposed the Clarification of Task and Plan phase for collecting information regarding the requirements to be embodied in the solution and regarding constraints. The Conceptual Design phase involves establishing function structures, searching for suitable solution principles, and combining the elements into concept variants. During the Embodiment Design phase, the designer determines the concept, layout, and forms for creating technical products or systems in accordance with the technical and economic considerations. Finally, the Detail Design phase involves arranging the form, dimensions, and surface properties of all individual parts, together with the specified materials, as well as re-checking the technical and economic feasibility. All drawings and other production documents are evaluated in Detail Design. Similarly, Ulrich et al. (2011) proposed a PDP based on six phases: planning, concept development, system-level design, detail design, testing and refinement, and production ramp-up. Baldussu (2014) noted that apart from the differences in the name of the third phase, the two aforementioned approaches (Pahl et al. (2007) and Ulrich et al. (2011)) share a common perspective, per the first four steps. Moreover, Baldussu (2014) noted that most of the time, companies develop their own PDP, often as an adaptation of one of the standard approaches (e.g., Cross 2008; Pahl et al. 2007; Ulrich et al. 2011). In fact, the PDPs proposed by Pahl et al. (2007) and Ulrich et al. (2011) were mostly diffused at industries and universities. Nevertheless, one main issue regarding the prescriptive methods is related to integration of the design phases. This issue arises owing to the different capabilities required in the entire design process. Some designers are good at the initial design phase (requirements and conceptual design) but not as good at the final design phase (embodiment and product development).

Role of Protocol Analysis in Design

Instructional strategies and models have been applied in several cases to understand the nature and cognitive load of design process (Christensen and Osguthorpe 2004; Sugar and Luterbach 2016). These studies aim to understand the complexity of design process, as results, researchers will be able to pursue studies that replicate more realistic the design process. Among the empirical approach for studying the design process, protocol analysis is the most commonly used method in research (Jiang and Yen 2009a, b). The protocol approach is considered one of the few approaches that elucidates the cognitive abilities of the designer. For example, the Delft Design Protocols Workshop of 1994 has given a crucial importance to this research field (Cross et al. 1996). A constant aspect in this field has been the search for method(s) with a clear understanding of design research processes (High et al. 1987; Herrera 2010; Becattini and Cascini 2016). As noted by Cross (1999), an important and diverse range of investigation methods has been employed for the research field of design activity, sometimes embedded in different PDPs. Three categories were identified in their empirical work. There were case studies that focused on understanding design skills and behavior and the progress and development of design projects and protocol studies, which were applied to artificial projects with strict protocol record requirements. By examining inexperienced (e.g., students) and experienced designers from this perspective, these protocol studies allow for a thorough understanding of the design process (Phadke 1995; Becattini and Cascini 2016). Designer performance testing is conducted in a controlled environment (Herrera 2010), in which a certain number of individuals are asked to perform a specialized task. However, models of this kind implemented under lab-controlled conditions are mainly related to the field of psychological investigation (Radcliffe and Lee 1989).

Protocol analysis has become one of the most well-established empirical research approaches in the field of design research (Jiang and Yen 2009a, b; Ericsson and Simon 1993). Unfortunately, problematic issues with this approach have been identified. They are divided into two situations: 1) designers who create a new product from scratch but mainly focus on the generation of ideas and elicitation of requirements and;2) designers who improve existing products but focus on idea generation. Consequently, there still is a lack of appropriate investigation on enhancing the repeatability, effectiveness, and usability of design methods for improving the PDP. To date, previous studies have focused only on design process tasks. There remains a need for more complete analysis in the field of design. Protocol analyses are generally applied in a manageable period, ranging from 20 min to a couple of hours. Other types of studies analyze the design process but focus on partial episodes instead of the overall process, involving problem analysis or proposals of design alternatives. The main reasons for this situation may be related to time requirements and technical difficulties in performing the required examinations. In addition, there has been few research to date that examines the intersection of cognitive load theory, conditions-based instructional design theory, and theories of problems solving (Sentz et al. 2019).

Motivation and assumption, hypotheses and objectives

With regard to the objective of setting a first step towards an approach for enhancing the predictability and measurability of Product Development Process, the formalization of design process is herein employed for individuating and highlighting the experiential perception offered by descriptive design method to designers.

It is assumed that design method corresponds to a structured process that transforms creative ideas into tangible products, services, and systems. Adapting the general design methods presented previously, the design phases of the product may be assumed as the stages allowing designers to create the new product, in other words, the design method can be considered as an Instructional Design model to follow (Christensen and Osguthorpe 2004; Sugar and Luterbach 2016). However, the final results of design process itself is influenced by the design approach that designer follow, personal-skill and cognitive load during the product development process. In this context, the purpose of this research study was to assess different design phases by using a common measurable instrument, which aims to compare different aspect of design process. Consequently, the following research question guided the study:

  • Is NASA-TLX a suitable questionnaire towards improved predictability and measurability of design process?

  • How does NASA-TLX stated awareness of the main workload-dimensions in the different design phases?

  • Which are the adaptations that need to have an instructional design model in order to be predictable and measurable constantly?

  • When designers given a common and detailed instructional design model allows them to improve repeatability of design process towards better product results?

  • What issues influence the execution of a successful product during the different design phases?

The present study exposes a procedure for enquiring about said research questions, which may result useful for the evaluation of new product development.

NASA-TLX to improve predictability and measurability of design process

NASA-TLX was developed by Hart and Staveland (1988) to analyze ergonomic factors of prototypes in the aerospace industry. The empirical results of NASA-TLX have been validated in several cases study in different fields of knowledge (Rubio et al. 2004; Cao et al. 2009; Noyes and Bruneau 2007; de Winter 2014). Indeed, Hart (2006) validated applicability and flexibility of NASA-TLX. The results showed that NASA-TLX has proven to be reasonably easy to use and reliably sensitive to experimentally important manipulations over the past 20 years. More in detail, NASA-TLX evaluates a multidimensional scale designed to obtain an estimated workload while the users are performing a task or immediately afterwards. Nonetheless, its effectiveness has allowed researchers to extend its application to investigating the psychological load of individuals performing different activities. NASA-TLX is a multi-dimensional rating procedure that assigns a total workload score based on a weighted average of six sub-scales: mental demand (mental and perceptive activity); physical demand (degree of physical effort); temporal demand (temporal perception); performance (degree of goal accomplishment); effort (amount of physical and mental effort); and frustration level (feeling of pressure, discouragement, and insecurity during execution). NASA-TLX analyzes three dimensions of requirements concerning individuals: mental, physical, and temporal. It analyzes another three dimensions related to the willingness of individuals: effort, frustration, and performance. Consequently, we adopt NASA-TLX to maintain continuity in the measurement of activities. Table 1 presents each dimension of NASA-TLX in more detail (Hart and Staveland 1983).

Table 1 Description of NASA-TLX dimension
Full size table

In more detail, the application of NASA-TLX has two sequential and structured phases: i) weighing and; ii) estimation (Rubio et al. 2001). Weighing occurs after task execution and consists of 15 binary comparisons among the six dimensions. For each pair, one element is chosen that the individual believes is the most important source of their workload. Each dimension is weighted depending on how many times the dimension was selected in the binary comparisons. The weight ranges from zero (if the dimension was never chosen) to five (if the dimension was chosen every time). This reflects the relevance of each dimension proposed by NASA-TLX in the task. Estimation is performed immediately after task completion. The individual estimates, on a scale from 0 to 100, the mental load of the task in terms of each of the six dimensions proposed by NASA-TLX. With the data collected in these two phases, the global index is calculated using the following equation:

$${\text{WI}} = \frac{{\sum\nolimits_{i = 1}^{6} {{\text{weighing}}_{\text{i}} \,{\text{estimation}}_{\text{i}} } }}{15}$$
(1)

The workload index (WI) is not an intrinsic feature of the task but is a result of the interaction among the requirements of the task, the circumstances under which it is developed, its capacities, and the behaviors and perceptions of the individual (Arquer and Nogareda 2001). In the present study, NASA-TLX was adopted as an outcome evaluation instrument for comparing the different design phases and related activities.

Creating a new instructional design model based on design method

This section presents the proposed instructional design model and case study structure for measuring each design phase using NASA-TLX. A key point to use NASA-TLX was represent the main design phases in specific task to be measured. With this premise, the design phases were split into different modules to better track the design process and related activities. From a design perspective, we consider a prescriptive design method a suitable approach to organize activities for the entire process. We can thereby monitor and control the phases (Table 2) by considering the main phases proposed by Pahl et al. (2007), which constitute a common and well-known design process used by practitioners. We consider this proposal suitable for organizing activities for the entire design process, allowing us to monitor and control the design phases using NASA-TLX.

Table 2 Detailed description of the modules and related activities
Full size table

Several modules and activities are established for each design phase to accomplish the research goals. The activities are clustered into four phases: clarification of the task, conceptual design, design embodiment, and detailed design and final product. Table 2 presents the design phases and tasks used in this research. We divide the design method into 11 modules to represent all the phases of product creation. Each module is characterized by a minimum of three activities/task, which can be measured independently as suggested by NASA-TLX. The use of activities allows the measurement of the outcomes for each module using NASA-TLX. In this study, we aim to determine which activities and phases of the design process demand the greatest workload from designers in the creation of a new product.

Procedure of research contribution

One of the limitations associated with instructional design and protocol design studies is related predictability and measurability of activities (Cross 2001; Edmonds et al. 1994). Consequently, this study proposes the NASA-TLX questionnaire as a measuring instrument, independent of any specific activity. NASA-TLX aims to formulate a common and appropriate approach for measuring different design activities by using the same workload dimensions and value scales. In this scenario, the NASA-TLX allows to address the following research challenges: (i) understanding the differences between design phases by using a common measurement approach; (ii) identifying the main solvable issues in the design solution process; (iii) creating a resilient solution that is suitable and adoptable for different users and practitioners; and (iv) obtaining insight into different design activities towards a systematic comparison.

The current study involves two essential parts: (1) controlling and measuring the design process with a common instrument for measuring the workload to understand the different design phases in detail and; (2) organizing the design activities to develop a new product by following a systematic approach (i.e. instructional design model), allowing the control and comparison of different activities related to design (Fig. 1).

Fig. 1
figure 1

Description of research method as a key element to validate the research contribution

Full size image

The next section presents a case study aimed at validating the proposal made in this paper.

Framework of case study

The study employed a quasi-experimental and mixed methods design to gain insight into how designers perceived the proposed instructional design model to follow (Table 2). This case study took advantage of the collection of quantitative and qualitative data provided by each activity from participants. At the end of each module, the designers use think-aloud as primary means of data collection, which provide a valuable information about cognitive process (Ericsson and Simon 1993). Additionally, at the end of the module each designer answer individually the NASA-TLX questionnaire to obtain data about their workload. The data obtain from NASA-TLX correspond to each dimension with scale from 0 to 100. The final data collection instrument used during the study was an interview protocol. This instrument provided the flexibility for participants to respond openly.

Moreover, the application and validation of the case study follows the logic of Robert Yin (2003). The problem to be solved is a contemporary problem in a real-life context. The product focuses on solving problems for handicapped people. Designers were asked to create a working surface that could be attached to a wheelchair. Moreover, the manufacturing process must be affordable. To validate the solution, designers collaborated with a group of handicapped teenagers participating in TELETON (www.teleton.cl). Finally, the products were tested by this group of teenagers.

The case study illustrates how and why different designers addressed each design phase. To justify and enrich the use of NASA-TLX, several gathering tools were used for complementary documentation (Yazan 2015). The validation of this case study requires the work to be structured and controlled throughout the entire design process. These practices require considerable time for both, evaluators and designers. In this case, four young designers participated with the aim of testing how the workload changes in each phase of the design process. None of the designers received any monetary compensation.

The project was divided into 11 modules according to the proposed instructional design model (Table 2), with two sessions per module. Each module was divided into a theoretical and practical part spanning 1.5 and 3 h, respectively (Table 2). A video recorder was used as a complementary data-gathering tool to monitor and record the design activities. This was useful for obtaining meaningful insights into the given situations. In the case study, the development of solutions took more than 40 working hours, from spanning the idea generation phase to the manufacturing of the final product.

Data and measurement instrument

The present study exposes a procedure for inquiring about data retrieval, which may be useful for the experiential evaluation of product development. First, the young designers were conscious that the activities developed in laboratories and classrooms would be recorded and used in this research. The sessions took place in a design laboratory of the Universidad Tecnica Federico Santa Maria. At the beginning of each module, participants received a short presentation with instructions about module activities, considering the expected outcomes proposed in Table 2. At the end of each module, three NASA-TLX questionnaires, framed for each module activity, were provided, delivering the related scores for the workload dimension. The questionnaires were in Spanish language to reduce bias and misunderstandings arising from incorrect comprehension of English text. Additionally, participants were asked to express an overall judgment about how well the modules represent real-life situations. Furthermore, to observe their behavior during module execution, an external observer recorded and captured images during the activities (see Table 3). Subsequently, the deliverables of each participant were evaluated by two expert designers to understand the quality of the outcome. A total of 133 NASA-TLX questionnaires were delivered in this study.

Table 3 Outcome images for each design module (in Spanish)
Full size table

Implementation of case study

In this subsection, we present the outcomes of the different modules using images of the developed activities. During the first phase (design requirements), we described the problem using a graphical representation related to the creation of a new table for handicapped individuals. During this phase, the designers focused on obtaining data using functional models while understanding the goals of their own projects. They generated ideas about the products that allowed them to define their initial shape and form, including operations and features.

During the second phase (design concepts), the designers developed a light prototype to better understand their concepts. At the end of this phase, a design concept was produced that served as a foundation for the next phase. The designers transformed their initial sketches into detailed conceptual designs and developed a quick and light prototype (i.e., cardboard). This activity was useful for envisioning the usability of proposals. The outcome of this process was a detailed conceptual design approaching a real solution. Additionally, the light prototype allowed students to consider realistic manufacturing limitations.

During the third phase (embodiment design), computer-aided design (CAD) models were created to obtain a detailed product specification, producing a list of parts for manufacturing. Additionally, the designers defined specific manufacturing processes to be followed. A product was manufactured in the previous phase, and modifications and problems were updated in the CAD model. Thus, the final product was based on a previously developed tangible solution, which allowed the designers to better solve the problem. Moreover, feedback for the final user was produced. Table 3 presents images of the different design phases.

Results from the application of NASA-TLX

This section presents the results of applying NASA-TLX. At the end of the case study, a total of 133 NASA-TLX questionnaires were obtained by scoring 11 modules of three activities each. Generally, this kind of experiment is developed in a repeatable and systematic way, whereby designers gain a better understanding of the systematic design process. The scores during the data collection process were valid and reliable, since the researcher attended to the possible threats of validity during each work session. The activities were planned to not have a specific outcome in controlled laboratory. Additional data sources, such as video recordings and sketches, were used to complement the findings obtained from NASA-TLX in this research. Video recordings were useful for establishing relationships between the questionnaire answers and designer behavior.

General results by using NASA-TLX

The use of think-aloud aims to confirm quantitative results delivered by NASA-TLX questionnaire. In this scenario, regarding the decisions they made in relation to managing workload. To simplify the understanding of the study and related results, we have defined a scale of ranges for NASA-TLX aiming to obtain a more plausible analysis towards specific conclusions: 0–20% = very low; 20–40% = low; 40–60% = normal; 60–80% = high; and 80–100% = very high. Figure 1 summarizes the WI for the different modules. The overall average of the perceived workload was between 50 and 75% (normal to high workload; see Fig. 2).

Fig. 2
figure 2

NASA-TLX results of the workload related to each module of the design process

Full size image

According to the first research question, “Is NASA-TLX a suitable questionnaire towards improved predictability and measurability of design process?”. The results suggest that this study shows a normal-high workload for the entire design process, which is considered to be positive in terms of motivation to stay active. The results regarding the workload are similar in general terms, implying that this systematic design process was not considered difficult, which is positive for a instructional design model too (Klein 1989). Nevertheless, comparing different module results reveals that module VI had a slightly higher average value of the workload than the others. From the video recordings, this result seems to emerge from the lack of an appropriate approach for anticipating and responding to problems. We observed that this module had the highest values because the young designers must conceptualize and synthesize their ideas with sketches. Nevertheless, a sketched idea must be feasible. Consequently, the trials and errors required for developing a solution were more extensive, involving several iterations.

Comparison of perceived workload among design phases

According to the second research question, “How does NASA-TLX stated awareness of the main workload-dimensions in the different design phases?” To better understand the behavior of each dimension associated with the different modules, Fig. 3 presents a summary of each NASA-TLX dimension during the design process. The results are as follows.

Fig. 3
figure 3

Comparison of outcomes for NASA-TLX among different dimensions

Full size image
  1. i)

    The workload observed in the “effort” dimension was “normal–high,” between 40 and 60%, reaching a peak in module III and exhibiting lower values in modules VIII, IX, and X. The workload reached an average of 80% in the last module. According to the tape recording, the main reason appears to be the motivation to complete the project on time.

  2. ii)

    The average of the “mental” dimension for the entire design tasks was between 50 and 75%, amounting to a “normal–high” value. The lower levels of mental demand coincided with the last steps, in which the designers focused entirely on manufacturing the prototype. It is possible to observe in the videotape that well-defined and structured activities did not require extensive mental activity. Consequently, during manufacturing, the mental dimension was low, because the designers considered these activities to be design execution. The non-existence of unexpected events during manufacturing appears to have a direct influence on the mental demand.

  3. iii)

    Regarding the “physical” dimension, the average workload was lower, under 40%, including the atypical data in the modules I and IX. The high value was identified in the initial modules and was the highest of this dimension. From the recordings, designers in the first module were asked to use wheelchairs to understand the user experience and to become knowledgeable about the problem they wanted to solve. Some designers perceived the task (move wheelchair) as difficult because it required physical effort. The second high-value range occurred during the module related to the manufacturing of the final prototype, meaning that the use of materials and/or tools required physical effort.

  4. iv)

    Regarding the “temporal” dimension, during the design process, the NASA-TLX results had an average of approximately 50%, which can be considered a normal workload. According to the recordings, some of the tasks required long times to be accomplished, whereas others were quickly performed. In this case, personal planning and development, as well as the skill of the designer, were relevant to finish the specific task on time.

  5. v)

    In the “performance” dimension, the data oscillated between 50 and 85% with high dispersion, especially in the last two design phases. This dimension was considered by the same designers as the self-evaluation of their outcome in performing the activities. Interestingly, the “performance” dimension is a completely personal evaluation that is strongly related to personal effectiveness in accomplishing personal tasks and performing module activities.

  6. vi)

    Finally, the “frustration” dimension showed that designers had values lower than 50% in every module, which could be a positive indicator of motivation. Nevertheless, in modules I, II, and III, the data from NASA-TLX reached higher values. In the recordings, these high values of the frustration dimension can be related to the following reason: the designers proposed ideas to solve the main problems, but several were inadequate or were not implementable as real solutions. Thus, the failure of a design proposal increased frustration in the initial design phase.

Generally, the workload related to each module changed over the entire design process according to the different activities performed. The average of the evaluated dimensions was between 40% and 60% (normal), with a relatively symmetrical distribution, except for the physical workload and level of frustration, which showed lower values of mental effort, temporal, and performance. Frustration and physical effort exhibited variations that can be explained by the particular situations involving the design activities. According to the recordings, for example, the presentation and idea evaluation involved anxiety and failures, thereby increasing the frustration dimension. Another example is how a specific manufacturing process activity increases the physical dimension, as it involves increased physical effort for a brief period (Fig. 4).

Fig. 4
figure 4

Results of the dimensions of the overall experience for NASA-TLX

Full size image

Finally, we developed a Pearson correlation analysis to better understand the relationship between the workload and dimensions, as proposed by NASA-TLX. A total of 133 NASA-TLX questionnaires were analyzed, considering 11 modules and three activities each (Table 4). The results show that most workload dimensions were correlated and significant (p < 0.05) among the participants. Regarding the correlation of the workload dimension, several relationships emerged (e.g., mental demand and effort, r = 0.768, p < 0.05; effort and temporal demand, r = 0.641, p < 0.05) (Table 4). However, negative relationships are strongly correlated to the performance and frustration: r = − 0.443, p < 0.05. According to correlation analysis, a larger mental capacity required by the design activity yielded a greater the effort dimension; thus, the frustration was higher.

Table 4 Pearson correlations among dimensions in the NASA-TLX questionnaire
Full size table

According to the empirical evidence, the case study shows how frustration is related to unexpected events. A larger number of unexpected events affecting the design process and proposed solution yields greater frustration. For example, when frustration emerged during the decision process, the designers perceived that their performance was worse.

By combining the data analysis with the tape recordings, it was possible to obtain and summarize knowledge and experiences as key elements for solving problems in design research field. Moreover, the initial design phase (requirements and conceptual design) required knowledge about the situation and alternative solutions to address the problem. According to the recordings, the young designers developed several sketches while attempting to solve the problem. However, many times, they felt frustrated because their proposals were not suitable. However, during the final design phases (embodiment detailed design and product), the number of trials and errors decreased, because all manufacturing activities were planned properly in the initial design phases. Thus, the frustration related to unexpected events was lower. Nevertheless, the time required to develop an activity influenced the expected outcome of the overall activity. We observed that when functionality was a main requirement, design novelties seemed less relevant. Consequently, in the design process, there was a tradeoff between the function requirement and aesthetics (i.e., complementary requirement). The simple statement proposed by Hollanders and Cruysen (2009) seems to characterize the results of this research: “design also corresponds to a structured process that transforms creative ideas into tangible products.” It is thus possible to understand that the design workload is strongly related to previous experience. Thus, new design methods should be supported by expert knowledge systems capable of providing appropriate information throughout the design process, with special attention to the initial phases.

Discussion of case study

In this section, emergent results from the application of NASA-TLX, enriched with tape recordings, are discussed. According to the following research questions “When designers given a common and detailed instructional design model allows them to improve repeatability of design process towards better product results?” In general terms, the design process can be considered a creative process plausible in reality using the available resources (Altshuller 1984). In the context of this study, the new instructional design model was perceived as a normal workload with according to the mental dimension, so can be considered as a positive instructional design model because reduce the cognitive load during the learning process (Klein 1989). Additionally, it required normal effort, according to the dimensions proposed by NASA-TLX. Additionally, this case study was useful in eliciting insights directly related to the effectiveness of the design process. About the research question: “What issues influence the execution of a successful product during the different design phases?”, From comparing interview combined with data emerged from NASA-TLX is possible to understand the main issues that influence the suitability of the product:

  1. (i)

    Idea effectiveness and usefulness. During the initial design phase, designers produced several ideas; many were nonviable. Consequently, frustration and mental demand increased considerably. These insights correspond to initial design phase values obtained utilizing NASA-TLX, regarding the mental demand and frustration (i.e., modules I, II, and III). Consequently, to avoid trial and error, using a systematic method appears to be an appropriate way to reduce the workload during the design process (Weth and Frankenberger 1995; Nikulin et al. 2017). Therefore, we show that there is a strong relationship between frustration and mental demand: r = 0,509 and p < 0.05.

  2. (ii)

    Anticipation of problems. When the designers developed solution concept ideas (second phase) with inappropriate anticipation of the manufacturing process, the frustration dimension increased considerably (module VI). This insight is relevant when creating plausible products. Thus, the coherence between the solution concept and manufacturing appears to be more important than expected. When the manufacturing process was well-defined, frustration decreased, because the designers considered this an execution activity (Becattini and Cascini 2016; Nikulin et al. 2013). For example, in modules VIII and IX, frustration was lower, because the manufacturing activities were fully anticipated during CAD creation. Thus, unexpected events were reduced.

  3. (iii)

    Integration of design phases. In the literature, there are many techniques intended to help designers throughout the different design phases. However, when it is necessary to create a real product, the integration of design phases is relevant for obtaining suitable results because it allows to anticipate problems that were not considered in previous design phases. For example, according to the case study, when a requirement was not met, or a partial solution was not properly addressed during the initial design phase, that requirement tended to produce problems in subsequent phases. Consequently, the dimension of mental demand and frustration was likely to increase. Thus, problem anticipation and mitigation is essential in the entire design process (Nikulin et al. 2018). Indeed, the use of a systematic design process aimed to obtain a normal–high workload for the entire process, which can be considered as a positive experience.

It is important to mention that this research proposed an approach to follow in order to compare systemically the different design phases, consequently, the final results of the design process might be affected according to the nature of the cases and characteristics of the samples who participated in a specific case study. Nevertheless, a measured systematic design process aims to help designers to understand the design phases and the outcomes of each phase, potentially leading to re-organized activities and effort aimed at changing the relevance given to specific behaviors during the design process. Consequently, this research can be useful for team management and for obtaining different design outcomes during the entire design process.

Conclusions

The research conducted in this paper contributes to the design process and advocates the use of more effective predictability and measurability approach to investigate the design process. Moreover, an Instructional Design model as systematic design process has been presented based in main literature review in order to be constant in measuring with NASA-TLX questionnaire. Finally, a real case study has been developed in order to validate the research proposal by both qualitative and quantitative approach. The research question of the authors have been, for a major part, answered: The Instructional Design model as being a guide for obtain more plausible and suitable products. Additionally, NASA-TLX aimed to obtain new insight/knowledge in terms of relevance of good ideas at initial design phases and relevance of “anticipation of problems” in order to reduce “frustration” of design process.

This paper contributes to the literature on the design process and instructional design model by proposing a NASA-TLX, which allows the comparison of design activities. However, the initial design methods need to be modified in order to have clear task to be measurable in practice. Consequently, instructional design method in order to be measurable and predictable need to have specific outcome. In this context, our proposal involved 11 modules, aimed at analyzing the different design phases and measuring the related NASA-TLX dimensions for evaluating product development. Moreover, using NASA-TLX provided a complete overview of the relationships between elementary activities and the consequences of trivial or intuitive solutions, which increased the frustration of designers. The analysis of the NASA-TLX dimensions helped determine what supported or limited the activities of the designers during the generation of suitable and effective products. Moreover, NASA-TLX was used as a measurement tool to obtain information from designers about all the design activities. NASA-TLX allowed the structured comparison of each phase and a comparative analysis among the different phases. In this study, the design experience followed an enriched systematic and prescriptive design method that was structured around specific goals. The results helped us understand how the designers performed design activities and how each dimension proposed by NASA-TLX was influenced by the different design phases by using correlation analysis.

To validate the author’s proposal, a real-life problem was addressed involving the development of a table for wheelchairs. The proposal has two main strengths: facilitating measurement of the activities and guiding designers on identifying and correlating design activities with personal skills. The proposed approach allows for the anticipation of how a dimension of NASA-TLX will behave according to different design modules. Simultaneously, the tape recordings identify the main causes that emerged from the behavior of the designers. The designers learned how to strengthen each dimension according to the design phase, which is useful for reducing trial-and-error costs that sometimes result from a lack of field-specific knowledge regarding product development.

The analysis revealed interesting results for understanding the design process from a holistic perspective. Our findings extend to the field of design cognition, which allows the improvement of the organization of the design phases according to how designers react to the different activities and related tasks. For example, some results showed that the design process could not be considered a physical activity. Rather, the design process should be considered a mental activity with frustration levels, which are dependent upon turning an idea into reality. Thus, methods and models in the design process should focus on transforming ideas into reality by reducing trial-and-error costs.

The present study presents limitations of various natures. First of all, it was conducted by arbitrarily picking one need, which, in spite of being relevant, might not be the most appropriate. This consideration leads to another limitation, related to more philosophical issues: workload may be predicted in the end, nevertheless a certain margin of unpredictability has to be considered, as it is not possible to foresee all of the alternative that designers use to develop their solutions. For this reason, experiments on true experiential assessment should be as realistic as possible, in order not to constrain the diverse possibilities of interpretation of the design activities. Hence, it is important to perform these experiments with the big sample size, in order to be able to observe the highest variability of “nuances”. However, the personal/individual analysis was useful for understanding the complexity of individual behavior when designers developed a new product. Finally, the validation of the present proposal is an issue, because there is a lack of knowledge regarding the relationship between the phases and the workload. Additionally, no alternative approach for comparing the results was found. Hence, further developments of the proposed approach should make inroads to a new procedure for validating results, besides a critical interpretation of the correlations.

In conclusion, whereas different refinements should be performed, the present study assessed the workload during the design process. It is expected that this approach may aid the development of learning and teaching solutions in education for increasing understanding and reduce the complexity of individual project development in different field, with a focus on creation in more robust and measurable instructional design models. This study indicates that the benefits gained by using a systematic design process method reduce trial-and-error costs. Considering the results obtained in this study, future research should aim to understand the experiences of designers during the initial stages of the design process and to manage their product expectations and problem anticipation, in order to reduce frustration, uncertainty, and stress. Next, we plan to apply the proposal findings to a multidisciplinary project to understand the complexity of teamwork performance in real-life projects.