Introduction

The engineering education curricula are slowly but constantly changing to address the deficiencies long discussed in engineering literature (Felder et al. 2000) and to adapt to the needs of companies and society. The need to renovate engineering education spending more effort in teaching design methods that are closer to industrial practices is highlighted in academic and industrial discussion since the ‘90s (see for instance Bordogna et al. 1993; National Research Council 1995), based on the idea that the students’ capabilities to solve problems, synthesize and analyze knowledge, and propose creative solutions, shall be the relevant skills to develop during the engineering education (Atman and Bursic 1996). Particular attention has been paid to the education in the engineering design area, whose importance has often been underestimated in the traditional engineering curricula (Dym 1999). According to Dym (1999), this happened in the past because of a combination of factors, including the inexperience of academics in being directly involved in design issues, and a general perception that in design education there was no real “scientific content”, since it could not be reduced into meaningful mathematical terms. Nowadays design is increasingly seen as a series of cognitive activities and practices that can be modeled, thus as a legitimate area of education. In recent years, researchers in engineering design have introduced a plethora of frameworks and models to be applied in different design phases, that have contributed to enhance the perception of design as a scientific and engineering-oriented discipline.

In detailed design phases, models have been developed to support the quantification of design parameters obtained as results of mathematical and logical processes (e.g. Finite Element Analysis). In more preliminary phases, where parameters are not yet defined, and mathematical correlations are unknown, models have been developed to support decision making mostly through qualitative assessment, based on expert judgment and personal experience (Eppinger and Ulrich 1995). In this area, a number of customized methods and tools to enhance decision-making have been proposed and tested in industry (e.g. Hazelrigg 1998; Ullman 2001; Collopy and Hollingsworth 2011). Among those, the concept Value Driven Design (VDD) (Collopy and Hollingsworth 2011) has lately emerged in the scientific literature and has been used as a framework for the development of methods and tools enabling the value assessment of products and services from a multidisciplinary perspective. While initially mainly applied in the aerospace industry, VDD-inspired methods have been recently deployed in other industrial contexts (Bertoni et al. 2016; Panarotto et al. 2017), reflecting a shared industrial need to use models (named “value models”) to preliminary quantify the long-run effect of an early design decision (i.e. understanding the impact of a decision from a multi-disciplinary and multi-stakeholder perspective).

Although, while the teaching of detailed design methods, such as numerical simulations or finite element analysis, is nowadays well established in academic courses, the same cannot be said for methods to model the “value” of design alternatives from a preliminary design stage. A challenge when teaching VDD is given by the fact that VDD methods are often developed by researchers and practitioners jointly collaborating in solving a specific company-related design problem. Therefore, generalizability and comprehensibility for applications outside the scope of the research are not prioritized, making difficult to understand the potential value of new methods for those not involved in the specific context in which the methods have been developed (i.e. lacking background knowledge and understanding of the problem and of the underlying logic of the solution).

Researchers in engineering design have discussed the barrier between the development of new methods in research and their generalized application in industrial settings (Wallace 2011; Gericke and Eckert 2015), highlighting that it is essential not only that those methods provide valuable results, but also that their role is clearly understood, to be successfully integrated into the everyday design activities. It is necessary that both experienced and novel engineers share the same perception of how to use a method, of its relevancy and of its potential. Looking at the engineering design education it is possible to affirm that in case of methods for detailed design, the self-evidence of the results (e.g. fractures or deformations) enables the students to clearly understand the benefits of a specific approach, while, when teaching how to use VDD methods, the teacher finds him/herself in the challenging situation of communicating the potential of the different approaches to an audience that lacks the contextual experience related to the complexity and uncertainty of a multidisciplinary early design analysis. This makes it difficult for the students to understand the value of a method in respect to its ability to manage and synthesize a variety of cross-disciplinary information. It is the role of education to fill this endogenous experience and knowledge gap between practitioners and new engineers, and the teaching activities shall be designed to provide the occasions for the students to learn the rationale and logic of the new methods, and to develop the individual skills and reflective capabilities to best fit the industrial needs.

This paper addresses the challenge of teaching VDD methods, with a specific focus on the use of value models for design concepts trade-off. The study presents the experience of teaching value models using realistic design sessions, in which the students practiced the use of VDD methods and reflected upon their relevancy. The use of realistic design sessions had a double objective. First, it aimed to provide the students with an effective way to develop their skills in evaluating and performing value analysis of different design concepts. Second, it aimed to prepare the students to effectively become “new engineers” by practicing the use of value models in a way that replicates a real industrial setting, allowing them to mimic the behaviors of practitioners when taking part of a early design evaluation. With the intent to verify the fulfillment of those two objectives, the paper describes the findings obtained by the analysis of the data collected from realistic design sessions run with both students and industrial practitioners on a common design case, i.e. a wheel for race bicycle.

The paper first defines the concept of value in the VDD domain. This is complemented by a description of the importance of value modeling in preliminary design, in relations with educational challenges and approaches in use. Third section presents the course set-up and frame the realistic design session into the CDIO syllabus. Fourth section presents the research design, and “Analysis of the data of the design sessions” section describes and analyze the results. The “Discussion and conclusion” section discusses the relevance of findings describing limitations and further improvements of the educational activities concerning Value Driven Design.

Defining value and value driven design

In the engineering and industrial context, methods and tools for value analysis, along with different interpretations of the concept of value, have been proposed since the 60s. In scientific literature, despite the centrality of the value concept, different interpretations exist about what the characteristics of value are and how stakeholders determine it (Day 2000). Miles (1962) first introduced the value analysis concept, defining the value of a product simply as the ratio of its performance over its cost. Later, authors from the fields of marketing and business management have either conceptualized value as the maximum amount a customer should be willing to pay, assuming full information about the product and competitive offerings (Shapiro and Jackson 1978; Forbis and Mehta 1981), or as a subjective perception linked to experience and extrinsic attributes such as brand name, logos, charm, social status and perceived quality (Zeithaml 1988). In the early ‘90s, in accordance with a more subjective interpretation of value, Normann and Ramirez (1993) stressed the notion of value-in-use, which highlighted the importance of the relationship between the company and the customer, whose perception of value is not limited to the product performance, but can be considered as an emotional bond established between a customer and a producer (1996). Researchers in marketing and service logic have stressed the subjectivity of this value perception stating that “Value is uniquely experientially and contextually perceived and determined by the customer” (Grönroos and Voima 2013, p. 146). Research in the domain of engineering design and systems engineering has instead highlighted the need to consider value as a numerical, quantifiable, encoding of preferences, reflecting the relative desire of someone to have one thing as opposed to other things (Collopy and Hollingsworth 2011). In such context, Value Driven Design (Collopy and Hollingsworth 2011) has become a popular umbrella term that collects several methodologies (O’Neill et al. 2010), ranging from previous work on Tradespace Exploration (Ross et al. 2004), to Value Centric Design (Brown and Eremenko 2008) and Value Driven Optimization (Castagne et al. 2009). It indicates an “improved design process that uses requirements flexibility, formal optimization, and a mathematical value model to balance performance, cost, schedule, and other measures important to the stakeholders to produce the best possible outcome” (AIAA 2015). In the VDD context, value has been defined as the level to which a product, or a technical solution, fulfills a set of needs from different stakeholders along its lifecycle (Bertoni et al. 2013). In the presence of complex engineering systems this includes more than performance and cost assessment, encompassing aspects such as: epistemic, emotional and image value (Steiner and Harmon 2009), relationship-based values (Kowalkowski and Kindström 2009), and value robustness to unexpected changes in the system (McManus et al. 2007).

The importance of value modeling in preliminary design

Decisions made in a preliminary design stage radically impact the specifications and configurations of a future product, playing a key role in defining lifecycle services, after-sales policies, suppliers and partners (Krishnan and Ulrich 2001). Nevertheless, at this stage, designers are in the situation of having limited knowledge about the consequences of a design decision, since the concepts under consideration are often immature. Later in the process, when a more established and developed knowledge base is available, major decisions have already been made and capital has already been committed, making it more costly and time-consuming to make changes (Ullman 1992). Studies in systems engineering have highlighted that up to the 70% of the total cost of a development project is already committed in preliminary design, and that the cost to remove defects radically increases afterward (INCOSE 2015).

Understanding the right design direction is therefore essential in preliminary design, and the use of models to help engineers deal with a wide and heterogeneous set of information is largely discussed in literature. In current practices, the use of requirements, expressing the engineering performances of a future product in a measurable and verifiable way, is considered as the main reference approach. However, a requirements-centered process is often unable to assure optimal decisions for an engineering system at an overarching level (Chen et al. 2013) and to add value to the solution space (Cheung et al. 2008, Collopy and Hollingsworth 2011). Researchers (e.g. Soban et al. 2011) claim that a qualitative assessment of the ‘goodness’ of a design is preferable against a quantitative encoding of preferences when qualitative data, assumptions and forecasts prevail. For such reason, methods are used in preliminary design to enhance engineers’ awareness about the value of a future design going beyond the mere calculation of product technical performances but encompassing the measurement of the customers’ satisfaction and of the product lifecycle performances. Quality Function Deployment (QFD) (Hauser and Clausing 1988) is one of the most used method in this respect, mainly because of its transparency in mapping engineering parameters to customer needs (Al-Ashaab et al. 2013). QFD has some limitations, in particular being unable to realistically model non-linear phenomena (Erginel 2010; Zhang et al. 2015), such as the relationship between quality and customer satisfaction (Anderson and Sullivan 1993) or the relationship between qualitative customer requirements and measurable technical requirements (Guenov 2008). Building on such considerations VDD literature has proposed different models to preliminary assess the value of a design concept. Novel engineers and students approaching the learning of such models need not only to understand the logic upon which they function, but also to understand the reasons why, in practice, a value modeling technique is more effective than another in terms of easiness of use, reliability of the results and uncertainty of the evaluation.

The challenge of teaching value modeling in preliminary design

As pointed out by Mclaren (2009) design is a very wide subject that can potentially cover the range of all disciplines within engineering, and it is recognized to be one of the core activities that an engineer undertakes. There are many approaches to teach design in engineering education, ranging from more “traditional” approaches focusing on specific modules treating detailed technical features of machine design, to more innovative approaches requiring cross-disciplinary skills in group work, logistics or management. When teaching models for VDD and systems engineering a challenge resides in making the student aware of the “broadness of the system” (Muller and Bonnema 2013), to make them understand the benefits and drawback of such approaches (Bertoni 2019). As highlighted by Muller and Bonnema (2013), systems engineering students might easily perceive the models as “open doors”, missing a frame of reference and a background to understand what such models or techniques offer, thus categorizing them more or less as the application of “common sense”.

A second challenge resides in the intrinsic multidisciplinary and multidimensional nature of a value assessment activity, i.e. encompassing both a wide set of engineering skills and a perspective over the system lifecycle, often outside the technical horizon of the typical engineering education. To this concern, the topic of interdisciplinary engineering design education has received a lot of attention in recent years by many higher education institutions worldwide. Academic literature shares the understanding that modern engineering designers need to possess multidisciplinary skills and those need to be part of the engineering education and training, as complementary to technical and subject-specific skills (Luft et al. 2013). For instance, Bronet et al. (2003) have highlighted the need to reroute traditional mechanical engineering programs toward higher integration of social and aesthetic aspects, while Ollis (2004) has identified a set of basic elements for the creation of multidisciplinary design courses. Jablokow (2008) has expanded the analysis of multidisciplinary skills for engineers by focusing on problem-solving and leadership capabilities, and Benedetto et al. (2010) have provided an example of cross-academic collaboration in creating multidisciplinary education programs towards innovation in engineering, architecture, and design. Additional contributions have come from Pop-Iliev and Nokleby (2011), proposing an approach for teaching concurrent design engineering, and from Vignoli and Onghia (2015) taking a more holistic standpoint on the engineering education, proposing an approach inspired by the Reggio Emilia method.

As a partial answer to the such challenges, the use of design sessions, in which students are given realistic design challenges in a controlled environment, has been recognized by Cash et al. (2012), as beneficial for design research. The use of design sessions to compare the behaviors of students and practitioners have often served as a testing environment for new design methods and tools, before moving to further development or to industrial implementation (e.g. Ellis and Dix 2006; Mitsui et al. 2009; Kleinsmann et al. 2012; Bertoni 2013). How to run small-scale design studies have been rigorously analyzed (Cash et al. 2012), emphasizing the benefit linked to the identification of trends and research directions. The use of design sessions should, however, not only be regarded as a source of data for design research, rather, it can recreate a learning environment that replicates the challenges and the context of a real industrial setting (Cobb et al. 2003). In such artificial contexts, researchers have highlighted that, when working on a design task, the major differences between students and engineering practitioners are present in the problem scoping and in the information gathering phases (Atman et al. 2007). The problem resides in the fact that most of the time the students do not realize the complexity of a problem, they underestimate the trade-off between design variables, thus increasing the risk to make bad decisions. According to Napp (2004), decision-making methods and theories for complex design environment should, therefore, be a major focus for improving design education. The setup of realistic design sessions capable of recreating a situation as similar as possible to a real industrial scenario can be regarded as an educational strategy to cope with this challenges.

Course set-up

The research presented is based on the data collected from realistic design sessions with students and industrial practitioners run respectively during the course of Value Innovation at Blekinge Institute of Technology and during industrial trainings and executive courses.

The term “realistic design session” is used to refer to a setting in which students are asked to deal with a design problem which mimics a real industrial challenge, but that it is run in a controlled environment (i.e. a university course), being aware that the design session results will not have any real impact on the company performances.

The course Value Innovation involved students at master level from the program of Mechanical Engineering, Industrial Economics, and Sustainable Product Service Systems Innovation. All students taking part in the course held a bachelor’s degree in an engineering related field, thus they possessed the basic knowledge to address engineering problems. According to the intended learning outcomes (ILO) of the course, the students shall, in the end, be able to:

  1. 1.

    Run a need analysis and a market analysis.

  2. 2.

    Develop and evaluate the value contribution of different design concepts.

  3. 3.

    Create the prototype of value adding solutions.

  4. 4.

    Work effectively in project teams.

  5. 5.

    Run value analysis.

The course accounted for 7.5 ECTS points in the European Credit Transfer System, based on learning achievements and students’ workload. It was structured as a combination of frontal lectures and autonomous group work based on a course project defined with a partner company. The course lasted for 2 months, and the design sessions were run 6 weeks after the beginning of the course. The first 6 weeks of the course were used to teach innovation theories, stakeholders’ analysis, customer needs analysis, creativity methods, and prototyping.

The design sessions were repeated in the frame of the course in three consecutive academic years from 2013 to 2015, thus 3 different classes of students took part to them.

The use of design sessions was planned to contribute to a major extent to ILO 5, and to a complementary extent to ILO2 and ILO4. This is because the students in the sessions were not asked to develop new solutions, rather they worked in teams to run a value analysis on design concepts already defined.

Framing the realistic design sessions in the CDIO syllabus

Literature on engineering design and systems engineering education highlights that the redesign of engineering training and education has to be approached from both a students and teacher perspective. Nadelson et al. (2015) stressed the necessity to enhance teachers’ capacity of teaching engineering design, proposing a support to significantly influence teachers’ knowledge about the engineering design process. Addressing both the students and the teachers’ perspective, an advancement in the field of engineering design education has been driven by the CDIO initiative (Crawley et al. 2014), promoting the context for an undergraduate engineering education to be a combination of four actions, namely Conceive-Design-Implement-Operate. The teaching and learning activities in the CDIO framework concern the direct participation of students in concrete design tasks, active learning, participation to workshop and project-oriented group tasks and collection of feedback from teachers. After its formulation, more than 150 academic institutions have joined to the CDIO initiative (CDIO 2018) with the vision that engineering graduates should be able to: Conceive-Design-Implement-Operate 
complex value-added engineering systems in a modern team-based engineering environment to create systems and product (Crawley et al. 2014). The CDIO initiative does not prescribe any specific approach, rather it shall be seen as a guiding framework to reform engineering education. To this concern the CDIO initiative encompass twelve standards to describe a CDIO program. Those are meant to support the transition of an educational programs toward a CDIO framework. Among those, the standard number 8 recognizes the relevant role of teaching and learning based on active experiential learning methods. In this line, research has recognized since the ‘90s the role of active learning in improving students’ attitude and writing and thinking ability (Bonwell and Eison 1991), and toward improving the capability to remember the topic of a lecture (Prince 2004).

The reform of an engineering education toward the CDIO principles implies an overall revision of an educational program from both a structural and content perspective. This shall be approached from a much higher level than a single educational activity as described in this paper. Nevertheless, the realistic design sessions were designed to contribute to achieve a few goals highlighted by the CDIO syllabus 2.0 (Crawley et al. 2011) in line with the university objective to promote the transition towards CDIO. In this context, the realistic design sessions are an example of the implementation of active learning. The activities were meant to contribute in addressing part of the CDIO goals dealing with analytical reasoning and problem solving, system thinking, interpersonal skills, and systems engineering and management in the conceiving phase. Table 1 summarizes the goals expressed in the CDIO syllabus that were addressed when designing the realistic design sessions. It has to be remembered, that such sessions are only one of the activities run in the frame of the Value Innovation course. The description of other course activities, addressing the examination of CDIO-related objectives in the frame of an industrial project, is outside the scope of this paper.

Table 1 List of the goals of the CDIO syllabus that the realistic design sessions contributed to address
Full size table

Research design

The research at the basis of this paper had the overall objective to investigate how to effectively teach value-driven design to university students. The proposed realistic design sessions were designed as a learning activity in which the students could develop their skills in evaluating and performing value analysis. They focused on preparing the students to become “new engineers” by practicing the use of value models in a way that replicated a real industrial setting. The research objective was declined in the following research question:

RQ1

How can value models be taught in academia overcoming the endogenous experience and knowledge gap between industrial practitioners and engineering students?

The research hypothesis was stated as follows:

Hp1

The realistic design sessions effectively teach how to use value models by enabling the students to obtain results that do not differ from those obtained by industrial practitioners.

The hypothesis was tested by running the same design sessions with both teams of students and teams of practitioners and analyzing the differences and similarities in the results. The details of the research design are described in the following sub-sections.

Participants

Two groups of participants took part to the design sessions: university students and industrial practitioners. The first group consisted of a total 61 students attending the course of Value Innovation in year 2013, 2014, and 2015. The students were divided into design teams of either 3 or 4 people, rendering a total of 17 design teams in the 3 years. The size of the teams was set to 3 or 4 people in order to balance the eventual presence of strong or weak participants, to make the discussion easy to follow and to avoid parallel discussions (Cash et al. 2012). This is because smaller teams increase the time spent on silent “thinking”, while larger teams increase the possibility of concurrent verbalization and parallel conversation (Cash et al. 2012). The students were randomly assigned to the teams. The realistic design sessions were organized as activities in the frame of the course in substitution to a regular classroom lecture. The participation was not mandatory but recommended by the teaching team. The students did not receive any incentive to take part to the design session and the activity was not part of the grading, nevertheless the students were aware that the ability to use the method practiced during the design session could have been part of the final examination of the course. One teacher took the role of “experimental controller” as suggested by Cash et al. (2012), to facilitate the design session. His role was passive, he did not support or suggest answers to the design teams, still he could be asked to clarify the logic of the method used. To minimize possible biases given by the relation between students and teacher, the experimental controller was neither the course coordinator, nor the examiner of the course.

In parallel, the same design sessions were run with industrial practitioners involving 78 people divided in 23 design teams with different industrial affiliations. Thirteen teams of practitioners were employed by an aerospace component manufacturer and took part to the design sessions in 2013, voluntary registering to the activity in the context of a company training day. Seven teams of practitioners worked for a telecommunication company and took part to the activity in 2014, in the frame of an executive course run in collaboration with the university. In this case it was not possible to verify the voluntary willingness to take part to the activity by the participants. Three teams of practitioners consisted of engineers having direct experience in companies but also having at the same time an academic position. Those teams were built by practitioners working in the aerospace industry, in the construction machinery industry, in product-service systems development, and in sustainable development. Even in those cases the same teacher took the role of experimental controller in the same fashion as for what done with the teams of students.

Procedures

The design sessions were organized as a 75 min activity. A design episode was created concerning the value assessment of different design configurations of a wheel for a racing bicycle. The choice of the racing bicycle as the reference product was driven by the necessity to limit as much as possible the knowledge barrier toward the evaluation of the new designs. The use of a bicycle was believed to be easily connected to the participants´ background knowledge and direct users’ experience.

All teams received the same task to be delivered in the same timeframe, and the same setting was followed by both students and practitioners.

Among the possible approaches for value modeling available in literature, a specific value modeling approach named EVOKE (Bertoni et al. 2018) was used in the sessions. EVOKE was developed and tested in systems engineering and product-service systems design and was selected because recalling a number of methods that are common in many approaches for preliminary design assessment. First, EVOKE uses numerical correlations (i.e. 0, 0.1, 0.3, 0.9) typical of the QFD method (Hauser and Clausing 1988); second, it uses a weighting system to define the relative importance of the “value drivers”, in a very similar fashion to the weights of the customer needs expressed in QFD; and third it uses of non-linear merit functions to set the type of correlations (i.e. the preferred behavior in terms of maximization, minimization and optimization) amongst the variables on the rows (i.e. the “Value Drivers”) and the engineering characteristics on the columns (i.e. the “Quantified Objectives), similarly to what used in the Customer Oriented Design Analysis approach (Wooley et al. 2001). For such reasons, EVOKE was considered as a good reference model for the students to practice, in order to understand the value and the limitations of using value models. The detailed description of the value modeling approach using EVOKE is outside the scope of this paper and can be found in Bertoni et al. (2018).

The task concerned the choice of the best design concept of a bicycle wheel. The participants were asked to populate two QFD-like matrixes as proposed by the EVOKE approach. The first matrix required the setting of only correlation coefficients (0–0.1–0.3–0.9) and the second matrix required both correlation coefficients and non-linear functions. The set-up of the design sessions is summarized in Fig. 1.

Fig. 1
figure 1

Set-up of the design sessions

Full size image

The sessions featured an initial 10-min introduction. This phase was important to align the teams and establish a common base for the experiment. At this stage, the teacher introduced the objectives of the design session, explained the specific tasks related to populating the EVOKE matrixes, provided contextual information about the component to be analyzed, and explained the meaning of needs and requirements featured in the provided EVOKE template (in paper-based A3 format). The teacher presented the bicycle users’ expectations with a list of derived systems level needs. Additionally, he presented the component level needs from the standpoint of the wheel manufacturer. The 10 min introduction followed a standardized scheme supported by a power-point presentation so that the information presented was the same for all the teams.

In the second step, the teams were given a 20-min time slot and assigned the task of populating each intersection between system-level needs (i.e., related the bike) and component-level needs (i.e., related to the bike wheel). System-level needs, seven in the sessions, were thought as criteria mirroring how customers experience the use of the bike along its lifecycle. These criteria were: Vibrations and Noise, derived from the overall customer expectation related to ‘comfort’, Top Speed, Grip and All-terrains, related to expectations about ‘road performances’, and Maintainability and Robustness mirroring the ‘operating cost’ of the bicycle. Component-level needs were considered a first step towards decomposing customer needs into requirements for the wheel. These were defined as Stiffness, Friction, Weight, Manufacturability, and Reparability (Fig. 2). They mirrored how customers experience the component and did not propose any specification for the solution; hence they did not feature a unit of measurement. The participants were asked to set four types of correlations in the matrix: 0 (no correlation), 0.1 (weak correlation), 0.3 (medium correlation), 0.9 (strong correlation). The students were already familiar to such conventional notation as it was previously taught in another occasion (i.e. the Systems Engineering course) during the study program. Once the matrixes were filled the participants were asked to take additional time to self-reflect on the confidence that they had on the correlations. This was made by asking the teams to set a degree of confidence for each of the component level need considered in the matrix. The assessment of the level of confidence mimicked the knowledge maturity scale proposed by Johansson et al. (2011), corresponding to the following definition:

Fig. 2
figure 2

Decomposition of expectations, needs, and requirements in the session

Full size image

  • 5 = Members are very confident and recognize exact correlations.

  • 4 = Members are confident and agree on the correlations after a short discussion.

  • 3 = Members are fairly confident and agree on the correlations after discussion.

  • 2 = Members are poorly confident and have different opinions on the correlations.

  • 1 = Members are not confident with any value and recognize opposite correlation.

No right or wrong answer existed for the knowledge maturity assessment, rather the activity was undertaken to promote the participants’ reflections on the assessment. In this way participants were asked to rethink about what they had just done, so to make them more aware of the uncertainty and risks that were embedded in the preliminary design evaluation.

After the first phase, a 10 min wrap-up session allowed the teacher to verify alignment between the teams and to clarify the remaining technical issues about the use of the provided material. Furthermore, this interruption allowed the teacher to introduce the second stage of the experiment, to explain the meaning of ‘component-level requirements’, and to specify the concept of non-linear merit functions.

In the second phase, the teams were asked to establish correlation coefficients (0–0.1–0.3–0.9) and non-linear functions (i.e. Null, Maximization, Optimization, Minimization) at the intersection of component-level needs and requirements. Four requirements were selected at this stage: Tire Diameter (mm), Tire width (mm), Spoke thickness (mm) and Use of composite (% in weight). Participants were asked to set 20 correlations and 20 functions, for a total of 40 items, in 25 min, rendering a pace of one item every 37.5 s. This is similar to the pace featured in the first iteration (approximately one item every 35 s). There are two reasons for relying on such an intensive exercise. Even if a longer session would have granted the participants more reflective time, previous research (Tsenn et al. 2014) has shown benefits in constraining experimental activities in design episodes shorter than 1 h. Also, it was important to verify if students intuitively understood the logic of the value assessment and if they were able to quickly come to a decision. As for the first stage, the participants were asked to self-reflect on the confidence they had in setting the correlations and the non-linear merit functions, and were asked to express such confidence level on a scale from 1 to 5.

The two-phase process allowed the participants to familiarize with the QFD matrix and with the logic of linking features at different levels of granularity. During the second stage, the participants could focus their attention on populating the matrix and setting relationships, rather than discussing the technicalities of the exercise and the mechanisms of EVOKE.

Data collection and analysis

The data from the design sessions run by the students were collected in two occasions. Firstly, at the end of the sessions, templates were collected from the teams and the results were manually imported to a software environment to be later analyzed. Secondly, a self-assessment questionnaire was distributed to the students at the end of the course to collect feedback concerning the design sessions to verify the achievement of the intended learning outcomes. The response to the self-assessment questionnaire was done on a voluntary basis. In the course questionnaire, two questions were considered as particularly relevant to understand if the use of design sessions reached its intended learning outcomes, those concerned if the students believed to be capable to develop and evaluate concepts on value contribution, and if the students believed they were capable to perform value analysis. These questions were answered on a scale from 1 to 4 corresponding to the following definition: not at all (1), to a minor extent (2), to a good extent (3), to a large extent (4). The complete set of questions of the self-assessment questionnaire is attached in “Appendix A”.

The data collection concerning the activities with practitioners was limited to the transcription of the results from the EVOKE matrixes in a software environment. No questionnaire was distributed to the practitioners.

The test of the hypothesis was designed by defining the type of team (i.e. students or practitioners) as independent variable, and each correspondence between component-level needs and component-level requirements as depended variables (including both the correlation coefficient and the selected function). The presence of differences between the two independent groups (i.e. students vs practitioners) was tested with the Mann–Whitney U test, as described in detailed in “Hypothesis verification” section.

Ethical issues

The research did not entail handling of sensible personal information and physical intervention on human beings. Moreover, the research was conducted with methods that do not affect or put at risk the research subjects physically or mentally. Based on such characteristics the research did not require to obtain the approval of the research ethical committee, in accordance to the Ethical Review of Research Involving Humans approved by the Swedish Ministry of Education and Cultural Affairs (SFS 2003:460).

Analysis of the data of the design sessions

The analysis of the data collected from the questionnaires is provided in “Analysis of the data from the questionnaires” section, while “Hypothesis verification” section describes the analysis of the data collected from the templates.

Analysis of the data from the questionnaires

The questionnaires gave both the possibility to the students to self-assess their progress with a numerical scale and to add further written reflections if desired. Two specific statements addressed the measurement of the achievement of the intended learning outcomes of the course related to the use of the realistic design sessions. The statements were:

  • After the course completion, I feel confident I can develop concepts and evaluate their value contribution.

  • After the course completion, I feel confident I can run a value analysis.

Out of the 61 students that took part in the design sessions, 28 answered the voluntary questionnaire. The results are summarized in Fig. 3, showing that the large majority of students believe to be able to develop and evaluate design concepts and perform value analysis to a good and major extent. The answers of the students to the self-assessment reflected positive feedback toward the achievement of the intended learning outcomes, thus providing positive indications toward considering the use of realistic design sessions as an approach to teach the use of value models in preliminary design.

Fig. 3
figure 3

Distribution of students answers to the self-assessment questionnaire. A numerical scale was used with 4 corresponding to “to a major extent” and 1 corresponding to “Not at all”

Full size image

Hypothesis verification

In respect to Hp1 (The realistic design sessions effectively teach how to use value models by enabling the students to obtain results that do not differ from those obtained by industrial practitioners) the students’ correlations set in the templates were compared to those of the practitioners. To test the hypothesis, the correlations between needs and requirements by all the teams were summarized and divided into two groups: the answers given by the groups of students and the answers given by the groups of practitioners. The comparison consisted in verifying if the correlation coefficients and the non-linear merit functions were set irrespectively from the type of the group. To allow this comparison a non-parametric Mann–Whitney U test was applied to each correlation between needs and requirements and to each selected function. The non-parametric Mann–Whitney U test was the most suitable statistic since it allowed comparing the difference between two independent groups, accepting both ordinal and continuous variables as dependent variables. In order to apply the Mann–Whitney U test four assumptions needed to be verified, namely: (1) the dependent variable should be measured at the ordinal or continuous level; (2) the independent variable should consist of two categorical independent groups; (3) there should be independence of observations, and (4) the two variables are not normally distributed and it should be determined if the two distribution have the same shape. The data collected satisfied the first, second and third assumption, while the assumption of equal distributions between the two groups needed to be tested. To do so, a test of homogeneity of variance was applied on each correlation and on each function set by the two groups. Table 2 shows the results from the test of homogeneity variance based on median and with an adjusted degree of freedom concerning the correlation coefficients set between the component level needs and the component level requirements. Table 3 shows the results of the same test but concerning the type of non-linear merit function set between the component level needs and the component level requirements.

Table 2 Results from the homogeneity of variance test concerning the correlation coefficients set between the component level needs and the component level requirements
Full size table
Table 3 Results from the homogeneity of variance test concerning the non-linear merit functions set between the component level needs and the component level requirements
Full size table

The level of significance showed in the last column on the right was larger than 0.05 for each correlation. This indicated that the null hypothesis of the non-parametric version of the homogeneity of variance test was accepted, meaning that also the assumption of equal distribution was accepted. This implied that it was correct to interpret the p value of the results of a Mann–Whitney U statistic.

After the acceptance of the fourth assumption, the statistic was applied to compare the two groups of respondents. Given the limited number of observations (40 observation for each correlation between needs and requirements), the analysis was run on the exact significance of the test rather than on an asymptotic approach, and the two-tailed version was selected as the more appropriate. Table 4 shows the exact significance level of the Mann–Whitney U test obtained for both correlation coefficients (second column) and non-linear merit functions (third column) considering groups of students and groups of practitioners as independent variables.

Table 4 Results of the Mann–Whitney U test for the choice of correlation coefficients (second column) and the choice of non-linear merit functions (third column)
Full size table

The results in the table show that none of the correlations featured a significance level smaller than 5% (i.e. 0.05), thus the null hypothesis was accepted in all the cases. The results of such analysis suggested that being part of the group of students or of the group of practitioners did not influence how participants populated the value models.

Discussion and conclusion

Engineering education is in evolution driven by the constantly changing needs of society and industry. Reforming the way engineering education is organized is a challenge that needs to be approached from different directions considering the wide spectrum of engineering disciplines. This paper has focused on a specific aspect of the engineering activity, i.e. the use of VDD methods in preliminary design, and how this can be taught to university students.

Value Driven Design is considered as a rather novel topic in the engineering design and systems engineering literature and no formalized methods applicable to education have yet to be presented. The use of value models in VDD requires individual thinking, judgment, and interdisciplinary cognitive capabilities going beyond the mere application of a method. For this reason, teaching VDD cannot be limited to the description of the logic of a method, rather it requires an approach stimulating students to think, judge and use their interdisciplinary knowledge to learn and reflect on the positive and negative qualities of the methods. The realistic design sessions presented in this paper recreated a design decision environment that mimicked the situation of a real industrial environment, in line by what proposed by Muller and Bonnema (2013). The adoption of the new teaching activity in the course was framed in the university effort to move toward the transformation objectives highlighted by the CDIO initiative.

Realistic design sessions engage the students in a scenario in which they directly experience complex issues and deal with them in a restricted timeframe. In the specific case of this paper, practicing the use of value models through group discussion and shared decisions, rather than through presentation of theory, was planned as an enabler for the students to more effectively reflect upon the tradeoff between benefit and effort spent on a value model, ultimately nurturing students’ discussion about benefits and drawbacks of VDD methods. This capability is particularly relevant in those areas of design in which there are no right or wrong answers, rather the best solution depends upon a mix of variables often difficult to know beforehand.

The analysis of the data collected through the course questionnaire showed that, after the activity, the students felt more self-confident in being able to perform analysis and value assessment in early design. Such results could be seen as an achievement of the “learning by doing” process granted by the design sessions. However, there is a gap between being confident in using a method and actually using it in the right way. This gap has been addressed by the testing of Hp1, assuming as positive a situation in which students and practitioners behave in the same way when doing value analysis. The results presented in Table 4 show that the 17 teams of students obtained results not having any statistically relevant difference from those obtained by 23 teams of practitioners, strengthening the hypothesis that the use of the realistic design session allowed the students to understand and apply the new method correctly, despite the lack of previous experience.

On the overall, the results of the data analysis indicate that the use of realistic design sessions contributed to the achievement of the intended learning outcome of the course (i.e. ILO_2 Develop and evaluate the value contribution of different design concepts, and ILO_5 Run value analysis), nevertheless, some limitations and possible improvements emerged and are worth being discussed.

Firstly, although the analysis indicates the sessions to be beneficial in the frame of the course, the students faced a design episode that referred to a knowledge domain relatively close to their experience (i.e. the use of a bicycle). The involvement of the students in a larger scale activity going beyond the 75 min timeframe would considerably expand the data collection possibilities. This would, at the same time, open to a larger possibility of introducing biases in measurement and analysis. In this respect framing the experiments in a student-driven project encompassing the development of new a product or system, from needs identification to requirements definition, is perceived as a relevant area for the further development of the educational approach. This could give the students enough freedom to test and select the type of value model more suitable for their specific situation, aiming to further foster team discussion and individual reflections.

Limitations are present in relation to the generalizability of the findings and to the research approach. The use of realistic design sessions has been decided in relation to the renewal of the structure and intended learning outcomes of the course of Value Innovation, and it has been applied on three consecutive years. No previous similar activity in the frame of the same course took place, and this renders a situation in which it is not possible to compare the results, in terms of achievement of intended learning outcomes, in relation to those of a control group subjected to a different teaching activity (i.e. not using realistic design session). Furthermore, different value modeling methods exist in VDD literature, respectively used in different industrial settings, thus it is not possible to claim the generalizability of the results for the teaching of any type of value model. In the experiment, the EVOKE approach was selected as representative of the main common characteristics of VDD methods because embedding the most popular features of conceptual design assessment methods, i.e. the use of QFD-like correlations with rank-weighted needs and the use of non-linear correlation functions.

In conclusion, in line with the university strategy to move toward a CDIO educational architecture, more research has to be spent on to the introduction of broader initiatives for teaching VDD, going beyond the introduction of a single activity in the frame of a course. “Learning by doing” is a remarkable development direction, however embracing such principle does not equal to provide quality education that really meets the intended learning outcomes. The risk is that of being overwhelmed by a plethora of teaching activities and methods claiming to provide improvement just because partially adhering to the CDIO architecture or because applying the “learning by doing” principle. A key challenge for education research in VDD is, therefore, to be able to discriminate between different methods and initiatives, so to bring forward those that provide scientific evidence of their effectiveness toward achieving higher learning outcomes for the students. The method proposed in this paper, and the analysis of its effectiveness, is an expression of such a belief and encourages a more detailed development of metrics to evaluate the effectiveness of educational approaches in engineering design.