Introduction

Higher Education Institutions (HEIs) face the challenge of promoting in their curricula the demands of labor markets in a world that is constantly changing, more technologically-driven and presenting ill-defined problems (Griffin et al., 2012; Wan and Gut 2011). Among the top five skills in demand for 2025 are problem-solving and creativity (World Economic Forum, 2020), often referred to as twenty-first century skills in terms of the new abilities students should be able to do to surpass the basic skills and knowledge expectations of the past, especially when considering the implementation of new technologies (Binkley et al., 2012; Lemke, 2002). One educational approach that fosters the development of these key skills, is labeled as design thinking (Luka, 2020; Scheer et al., 2012). Design thinking has been used as a means for value creation and innovation in different fields including business, medicine, science and various stages of education (Pande & Bharathi, 2020).

Design thinking was first coined by Simon (1969) and it has during recent decades gained popularity in HEIs contexts (Kleinsmann et al., 2017; Liedtka, 2014; Matthews & Wrigley, 2017; Razzouk & Shute, 2012; Spee & Basaiawmoit, 2016). Moreover, HEIs perceive the curriculum implementation of design thinking as a safe learning space where students can work in multidisciplinary teams boosting their skills and knowledge beyond their fields of study (Wrigley & Straker, 2015). Results from a recent design thinking review study show that it is mostly associated with skills in educational settings in terms of: collaboration/teamwork, creativity, problem solving, and empathy (Guaman-Quintanilla et al., 2018).

Nevertheless, researchers criticize the lack of a systematic assessment of design thinking results (Liedtka, 2014), as well as the lack of accurate, performance-based measures to study its impact (Razzouk & Shute, 2012). Moreover, Steinbeck (2011) criticizes the lack of comprehensive design thinking assessment approaches that fit the complex nature of design thinking and its application in multidisciplinary settings. In addition, Spee and Basaiawmoit (2016) point at the absence of statistically robust empirical studies to underpin design thinking effectiveness.

In this study, we assessed the impact of a compulsory first-year course called ‘Analysis and Problem Solving’ in an Ecuadorian University. In this compulsory course students learn how to analyze and solve a real-life problem using design thinking. This course offers as such as an promising opportunity for measuring students’ problem solving and creativity skills, as a result of using design thinking in university settings. Data were collected at three time points during a semester taking into account three raters: students as self-evaluators, peers and facilitators. To the best of our knowledge, no studies have adopted a multi-actor perspective to study design thinking impact. Therefore, the present study sheds light on the impact of design thinking, motivated by: (a) the HEIs growing interest on design thinking; (b) problem solving and creativity, as the most-mentioned skills related to design thinking; and the (c) lack of robust empirical studies to measure the impact of design thinking.

The remaining part of the paper proceeds as follows: First, we review the literature to define design thinking guidelines on a conceptual and theoretical framework. Next, the research method and description of the quasi-experimental procedure is provided. Finally, based on our findings, a discussion that puts forward an agenda for future research and implications for universities curricula is presented.

Conceptual and theoretical framework

Design thinking

Authors argue regarding a clear definition of design thinking (Blizzard et al., 2015; Goldschmidt & Rodgers, 2013; Johansson-Sköldberg et al., 2013; Kimbell, 2011). One of the most cited definition of design thinking is: “A discipline that uses the designer’s sensibility and methods to match people’s needs with what is technologically feasible and what a viable business strategy can convert into customer value and market opportunity.” (Brown, 2008, p. 2). A detailed analysis of design thinking literature in Higher Education settings, helped defining the following main characteristics: human-centered approach, solution of ill-defined problems, creativity, teamwork/collaboration, prototype-driven (Carlgren et al., 2016; Fleury et al., 2016; Ito et al., 2015; Lugmayr et al., 2014; Patel & Mehta, 2017; Razzouk & Shute, 2012; Wrigley & Straker, 2015). Design thinking supports dealing with complex real-life problems, following a human-centered approach and demands students to be avid interdisciplinary collaborators (Brown, 2008; Scheer et al., 2012). Literature also shows that the teachers’ role in design thinking is to be seen as a facilitator, rather than an instructor (Scheer et al., 2012). In a design thinking intervention, the facilitator presents students with the context that comprises of the challenges, collaborative groups, and tools and activities (Pande & Bharathi, 2020). Furthermore, facilitators stimulate students to unleash their creative potential promoting a conducive environment to develop reflection about what worked and what can be learned from things that did not work (e.g. Kelley & Kelley, 2013).

In addition, Wrigley and Straker (2015) propose five-steps along an ‘Educational Design Ladder’ to foster the development of design thinking. The authors point out that by following these steps, students acquire knowledge and skills to apply design thinking in different contexts and projects. Moreover, building on their approach about ‘design-based learning’ (Luka, 2020) or ‘design methodology’ (Huang et al., 2019), the scholars put forward approaches such as: the four-stage process – also called the ‘Double Diamond’ – proposed by the UK Design Council (2007); the popular five-stage model of the Hasso Plattner Institute of Design at Stanford University (d.school) (2010), among others. The stages/models help developing operational interventions to attain the aforementioned impact on problem solving and creativity.

Based on the literature, we propose the following definition of design thinking: Design thinking is a way of working and thinking that goes beyond the pure design context, as it is a way of solving ill-defined problems using methods and mindsets typically associated with designers, but adapting them to different real contexts and applying a human-centered and prototype-driven approach, which fosters creativity and promotes the value of teamwork.

In the following sections we analyze the learning theory that helps describing and explaining the impact of design thinking, as well as how it is connected to the development of problem solving and creativity skills.

Constructivism and its relation to design thinking

Design thinking challenges the teacher-centered approach. The latter is also known as ‘the sage on the stage’ teaching delivery method where a teacher transmit their knowledge (by lecturing) to students who passively absorb the content (King, 1993). In contrast, design thinking is linked to constructivist learning theory (Noweski et al., 2012; Pande & Bharathi, 2020; Scheer et al., 2012). Constructivism serves as an umbrella term to bring together a diversity of views sharing two main ideas: (a) learning is not seen as transmission of knowledge but as an active process of constructing knowledge; and (b) instruction is a process of supporting the knowledge construction process (Duffy & Cunningham, 1996). Jonassen et al. (1995) argue that constructivism raises the assumption that knowledge depends on how people create meaning from their experiences. In addition, the aforementioned authors also emphasize the importance of the social context in the design thinking learning environment based on the socio-cultural constructivist approach of Vygotsky (1978).

Constructivism beliefs about learning assert the need for embedding students in real-world situations where they function as a part of a community that contributes to solving real-world problems (Lave & Wenger, 1991). Furthermore, constructivist environments should engage students in their construction of knowledge through collaboration that insert learning in a meaningful context and reflection on what has been learned (Jonassen et al., 1995). Regarding to evaluation, constructivism focuses on the thinking process and the students’ abilities to argue and support decisions that are related to the development of self-reflection and metacognitive skills (Karagiorgi & Symeou, 2005).

Since the learner is at the center of the constructivist teaching–learning process, teachers are seen as facilitators rather than instructors (Murphy, 1997; Noweski et al., 2012; Pande & Bharathi, 2020). This role implies guiding and supporting students in their construction of knowledge, providing instruction and feedback, monitoring progress and evaluating (Neo, 2003). Additionally, Duffy and Cunningham (1996) state that teachers should play the role of coaches to guide learners and support them as scaffolds.

The link between design thinking and constructivist learning theory remains fuzzy. Pande and Bharathi (2020) tried to map constructivist tenets to design thinking phases. Design thinking main characteristics (human-centered approach, solution of ill-defined problems, creativity, teamwork/collaboration, prototype-driven) can also be linked to constructivist tenets (Jonassen, 1994); we explain these connections as follows: Constructivist tenets provide multiple representations of reality, represent the natural complexity of the real world, provide real-world, case-based learning environment, rather than pre-determined instructional sequences. In this regard, design thinking enables students to deal with complex real-life problems (Scheer et al., 2012). Moreover, design thinking uses a human-centered (also known as user-centered, user-driven, customer-oriented) approach, therefore students are encouraged to develop more empathy towards different views and feelings (d.school 2018), providing the means to bring out real users’ perspectives while analyzing the problem and testing possible solutions. Other key characteristic of design thinking found in most of the literature is the focus on prototyping; in this sense, prototyping develops the skills and competences of students by encouraging them to work on authentic real-life problems (Jussila et al., 2020).

Another tenet of constructivism focuses on knowledge construction, not reproduction. Jonassen (1997) argues that students facing ill-defined problems synthesize their own understanding of the situation rather than find a straightforward solution for an already defined problem. Since design thinking is characterized by working with ill-defined problems, this suggests a connection with this tenet. Besides, according to Scheer et al. (2012) the design thinking process pushes the teacher to become a facilitator of a constructive learning; i.e. designing the space to let students experiment different mental models and methods, balancing instruction and construction, so pupils can convert abstract and general principles into meaningful practice.

Constructivism also builds on working on authentic and contextualized task. In this sense, during a design thinking intervention, the facilitator introduces students with a context that includes the real-life problem (Pande & Bharathi, 2020). Moreover, contextualizing may be also manifested through the focus on a human-centered approach within a design thinking process, which helps identifying insights and delivering innovation that reflects what users want (Brown, 2008).

Additionally, design thinking entails “for different situations apply different perspectives and new perspectives generate new situations” (Scheer et al., 2012, p. 13) which reveals another key tenet of constructivism: Enable context-and content dependent knowledge construction. Moreover, prototyping (as a design thinking main characteristic) requires students, users and others to experience and interact with prototypes for building successful solutions (d.school 2018) which means the knowledge is constructed by students based not only in the content (e.g. prototyping techniques) but also in the context through validation of those prototypes.

Fostering reflective practice is a next constructivist tenet promoted by design thinking. Design thinking promotes students to reflect on their problem solving approach (Scheer et al., 2012). Besides, as part of the human-centered approach of design thinking, students try developing empathy with the actors involved in the situation, empathy involves emotion and logic, which stimulates insights, inspiration and intuition (Glen et al., 2015). Prototyping is another opportunity to foster reflection as students explore ways to turn prototype failures into opportunities (d.school, n.d.).

Finally, the last constructivist tenet is about supporting collaborative construction of knowledge through social negotiation. In this regard, most of the design thinking literature points at collaboration (or teamwork) as one of its main characteristics. For instance, Brown (2008) suggests a design thinker needs to be an enthusiastic interdisciplinary collaborator. Both the collaboration and human-centered approach of design thinking supports the Kanselaar's (2002) idea of learning as the negotiation of meaning since shared ideas among users and/or other stakeholders, as well as teammates (knowledge-building community), should be accepted and agreed upon. For instance, prototyping helps to overcome the boundaries between different stakeholders, reducing miscommunication, as well as the amount of effort required to discuss and share ideas (Björklund et al., 2017). Furthermore, in design thinking, team competences are promoted to express opinions and share knowledge; also, students and teachers build trust (Scheer et al., 2012).

Problem solving

In the education context, problem solving promotes higher-order skills and it is positioned as one of the key skills learners have to develop (Jonassen, 1997). Hesse et al. (2015) provide the following definition for problem solving: “an activity in which a learner perceives a discrepancy between a current state and a desired goal state, recognizes that this discrepancy does not have an obvious or routine solution, and subsequently tries to act upon the given situation in order to achieve that goal state. It is accompanied by a number of mental and behavioral processes that might not necessarily take place in sequential order but can run in parallel” (p. 38). The authors emphasize the collaborative nature of problem solving by working on intertwined activities and building on each other’s contributions. Building on the scope and nature of the present study, we especially build on the rephrasing of the above definition by Rhodes (2010): “The process of designing, evaluating and implementing a strategy to answer an open-ended question or achieve a desired goal.” (p. 41).

Constructivism consistently emphasizes problem-solving as a key learning outcome (Kanselaar, 2002; Murphy, 1997; Neo, 2003). Since design thinking is aligned with constructivism, problem-solving is clearly an associated skill (Alhamdani, 2016; Anand et al., 2015; Bhatnagar & Badke-Schaub, 2017; Khalaf et al., 2012; Lugmayr et al., 2014; Taajamaa et al., 2013). This connection can be explained as follows. One of the main characteristics of design thinking is working with real and ill-defined problem. According to Voss and Post (1988) working with these type of problems challenge students’ problem- solving skills. This is confirmed by Jonassen (1997) who argues that when dealing with ill-defined problems, students must (re)frame the design problem, recognize divergent perspectives, and collect evidence to support or reject the alternative proposals. Moreover, in instructional approaches to implement constructivist learning principles, the focus is mainly on developing the skills related to solving the problem in an authentic context (Duffy & Cunningham, 1996), as also happens in design thinking. During the process for solving problems, students establish a reflective practice with the situation. Reflection is one of the tenets of Constructivism, and as explained in the previous section, design thinking also promotes reflection while dealing with ill-defined problems, developing empathy, and testing prototypes. As mentioned earlier, other common aspect of constructivism and design thinking is the emphasis on collaboration. In this regard, during a problem-solving process the achievement of a group is considered to be more effective because it brings together different abilities and skills, which might lead to better collective outcomes (Siemon et al., 2019).

Creativity

As mentioned earlier, creativity is a dominant skill in the design thinking literature. However, the concept also introduces conceptual confusion (Egan et al., 2017; Hernández-Torrano & Ibrayeva, 2020). Psychology-driven definitions of creativity stress originality and usefulness (Runco & Jaeger, 2012; Stein, 1953). However, Sawyer (2006) goes beyond this approach by stating that creativity is “the emergence of something novel and appropriate, from a person, a group, or a society” (p. 33). Sawyer adds that creativity consists of combining divergent and convergent thinking and switching back and forth at different points in the creation process. This is reiterated by Lindberg et al. (2011) who put forward a problem space and solution space in design thinking. Both spaces include diverging and converging activities. Based on the scope and nature of the present study, the creativity definition of Rhodes (2010) defines our approach in the best way: “It is both the capacity to combine or synthesize existing ideas, images, or expertise in original ways and the experience of thinking, reacting, and working in an imaginative way characterized by a high degree of innovation, divergent thinking, and risk taking.” (p. 27).

The literature stresses the linkages between constructivism and creativity (Asmar & Mady, 2013; Lim, 2014; Neo, 2003). Creativity is mentioned in the design thinking literature either as a characteristic of the process (e.g. Lewrick et al., 2020), or as one of the skills being developed throughout the process (Alhamdani, 2016; Clemente et al., 2017; West et al., 2012). The constructivist framework helps to understand the link between creativity and design thinking. For instance, divergent thinking —one well-studied aspect of creativity (Caughron et al., 2011)—is fostered in design thinking since students need to explore and be flexible for handling real-world information they often have to collect themselves. Another example stresses how during the ideation process students have to make choices (IDEO n.d.). And during brainstorming, a technique commonly used in design thinking (Lewrick et al., 2020), students create and build on a pool of choices, and develop argumentations in relation to a final choice (IDEO n.d.). Moreover, in design thinking courses, facilitators help students to unleash their creative potential promoting a conducive environment to develop reflection. This helps them to learn from things what did (not) work (Kelley & Kelley, 2013). The link with creativity also pushes the instructional design of related environments. According to Caughron et al. (2011) systematic reviews of literature about creativity training stress the provision of challenging materials, real world exercises, along with multiple opportunities for practice by trainees, among others. These design guidelines reflect both constructivist and design thinking characteristics, as explained in previous sections.

Additionally, some scholars have put forward the linkage between problem-solving and creativity (Caughron et al., 2011; Martz et al., 2016; R. K. Sawyer, 2006). Sawyer (2006) points out that creative processes involve problem solving and decision making; similarly, decision making requires creative inspiration. Sawyer additionally stresses that creativity involves both problem solving and problem finding. Martz et al. (2016) suggest an interdependence between problem solving and creativity; and Star and Rittle-Johnson (2008) found that problem solvers show flexibility -one of the creativity dimensions (Torrance, 1974)- when dealing with a challenging part of a problem.

Building on the available theoretical and empirical literature, the authors of this study put forward the following two hypotheses:

Hypothesis 1 (H1)

Students who participated in a design thinking course improve their problem solving skills.

Hypothesis 2 (H2)

Students who participated in a design thinking course improve their creativity skills.

Method

Design

The present study was set up as a field experiment in which university first-year students, regardless of their major, enrolled in the course ‘Analysis and Problem Solving’ (also referred to as the design thinking course), in an Ecuadorian University. This compulsory course seeks to develop in students their problem solving and creativity skills when developing solutions for real-life problems by applying design thinking. The study was set up throughout a complete academic term (from May to August 2019) and included administration of a pretest (t0), a mid-semester test (t1) and a post test (t2). During 14 weeks of classes students received a wide range of in-class training tasks and work in cooperative teams (five to six students) to find a solution for a real problem presented by an Ecuadorian organization (e.g. NGO, small business, etc.), called ‘sponsor’. At the end of the course, teams were expected to present a prototype solution to tackle the problem. In this study, students’ progress in terms of problem solving and creativity was investigated by analyzing their answers when solving a case study (see Research instrument section). Students tackled such case at the start (t0), in the middle (t1) and at end of the course (t2). On the base of a rubric, students, the facilitator, and student peers rated the case solution. In other words, the research data emerged from three different actors at three data points in time.

About the course

The Analysis and Problem Solving course adopts a design thinking approach (Fig. 1) based on the combination of both the model of the Hasso Plattner Institute of Design at Stanford University (d.school) (2010) and the “Double Diamond” model (Design Council, 2007); reported by Santos Ordóñez, González Lema, Puga, Párraga Lema, & Vega, (2017). The latter follows 6 stages.

Fig. 1
figure 1

Design thinking approach used in the Analysis and Problem Solving course

Full size image

The first diamond of the design thinking approach clusters three stages: Research, Empathy and Define; while the second diamond consists of the other three stages: Ideate, Prototype and Validation. Each diamond represents the divergent (left side) and convergent (right side) thinking the students are expected to adopt. Moreover, the students were informed of the possibility of iterating through all design thinking stages; for example, students could do additional exploratory research at any stage, or re-define the problem after discovering a new insight while testing an early prototype.

Meeting one of the tenets of constructivism, and at the same time one main characteristic of design thinking, each facilitator searched for real problems (5 or 6) before starting the semester. Those problems were proposed mainly by NGOs or small businesses (called ‘sponsors’). During the first week of classes, facilitators presented the problems to their students and students indicated their interest about each problem by ranking them. Next, facilitators set up multi-disciplinary teams and assigned teams one of the problems to work upon during the semester, i.e. each team worked on a different problem. Students worked collaboratively in the same team during the whole semester analyzing the problem and proposing a solution (called course project) following the design thinking approach presented in Fig. 1. Collaboration is also one of the tenets of constructivism and a characteristic of design thinking.

During the course, students applied specific techniques and tools in each design thinking stage, guided by the facilitator. They immediately applied those techniques and tools to tackle their projects during the class and when doing related homework. The course outline that includes the description of each design thinking stage as well as the techniques and tools learned in each stage is presented in Online Resource 1. Class sessions were very active and alive with plenty of time for students to apply and discuss the techniques and tools learned. Classrooms were equipped with a projector, rounded tables (one table per team), whiteboards, markers, post-its and other provisions to boost group communication. The course was divided into two large subparts, in the first part students worked along the three stages of the first diamond (Fig. 1). By the end of the third stage, the teams presented the progress on their course projects applying explicitly the three design thinking stages: they presented what (and how) they learned about the context of the problem and the stakeholders involved, insights discovered and how they (re)frame the problem. In the second part of the course, students adopted the three stages of the second diamond. They subsequently used their creativity to ideate as many solutions as possible to the defined problem. Next, they evaluated and filtered those ideas. In a next step they built prototypes to materialize the best ideas, testing them with real potential users. By the end of the course, the teams presented their projects, explaining their solution process and presenting an argumentation as to the adequacy of their problem solution. All the sponsors were invited to attend these presentations. Online Resource 2 depicts an example of the development of a prototype solution proposed by one team, during the course. Their sponsor was Asperger Foundation – Ecuador who supports parents of children with Asperger’s syndrome. In this case, they focused on “clients” who had limited access to the services provided by the foundation. From the many ideas being generated in the Ideate stage, the best idea was selected to develop an app for parents of children with Asperger’s syndrome. Next, they built a low-fidelity prototype using basic materials to outline the data flow and to check the usefulness and usability of their proposed functionality. This helped validating their prototype with the sponsor and potential users. This stage helped getting feedback in view of making adjustments. By the end of the course, the team built a high-fidelity prototype using an online tool that aimed at delivering a close instantiation of the final product. This was presented during the final exam in the presence of the sponsor, facilitator and classmates. The sponsor – in the case of the Asperger foundation – was very satisfied with the results.

The students attended 28 sessions of 90 min (3 h per week). The course structure, contents, activities and materials were the same for all sections and facilitators followed the same guidelines. Finally, it should be noted that the other courses taken by students in their first semester are mainly focused on the development of hard skills instead of skills such as problem solving and creativity.

Participants

In this study N = 910 freshmen students participated who were enrolled in the Analysis and Problem Solving course. The average age was 19 years (SD = 2.17). The students’ background information is presented in Table 1. The majority of the students were men, coming from urban areas. As expected, since this course is offered to first-year studens, most of them did not have working experience. Most of the participants were enrolled in a Science and Engineering program.

Table 1 Students’ background characteristics
Full size table

Given the large number of students, the course was taught to subgroups of students (sections). Each section comprised 35 students -on average- from different disciplines. In total, 27 sections of the course guided by 26 facilitators were included in this study. The facilitators were informed about the research study beforehand to minimize risk of bias. Informed consent was signed by all students after ethical clearance from university authorities.

Research instruments

In this study the research instruments used were (a) a case study developed by the authorsFootnote 1 that was followed by a series of tasks/questions to be answered by students individually; (b) a rubric based on case specific criteria to map the quality of the problem solving and creativity skills. The same case study was presented at t0, t1 and t2. This case was based on a real-life problem related to user-experiences of a new service for riding bicycles at a university campus. To observe the use of problem solving and creativity skills, students were asked to complete four tasks related to the case study using a paper-based form.

  • Identify and properly define the main problem of the case.

  • Before proposing a solution to the problem you defined, what would be your strategies or steps to follow?

  • Propose and describe at least one innovative solution for the problem defined in question 1.

  • Reflect on the solution process(es) resulting from the former task.

The case study and the tasks were not used as scaffolds of the design thinking process, they were never discussed during the classes and they did not have any influence in the students’ course grades. Students did not have access to the case or tasks beforehand, and they were not informed that the same case and tasks would be applied three times. The students received 25–30 min to read the case study and solve the tasks/questions. Next, the solutions to the case study proposed by each student for the 4 tasks/questions were rated by peers, by themselves, and by their facilitator. The students identified themselves with a code only; thus, rating by peers and by facilitators involves a blind process. The involvement of the three raters is critical to respect the underlying constructivist assumptions related to design thinking; i.e. given the emphasis on students’ personal involvement and engagement, the key role of collaboration with peers and the supportive role adopted by the facilitators, all three should put forward their measurement of the impact on problem solving and creativity. In addition, the answers posed by the students for the 4 tasks/questions were rated on the base of rubrics. The evaluation instrument used by the three referred raters was based on an adapted version of the VALUE (Valid Assessment of Learning in Undergraduate Education) rubricsFootnote 2 (Rhodes, 2010), and it focused on the identification and analysis of indicators related to problem solving and creativity. Each rubric criterion presents a description to be rated along five performance levels: A, B, C, D, E; where A represents the highest score and E the lowest. Given the scope of the course, the problem solving rubric used in the study focused on 4 out of the 6 original VALUE rubric criteria: (1) Define problem, (2) Identify strategies, (3) Propose solutions, (4) Evaluate potential solutions. The creativity rubric included 4 out of 6 of the original VALUE rubric items: (1) Acquiring competencies, (2) Taking risks, (3) Innovative thinking, (4) Connecting, synthesizing, transforming. Eight criteria were provided to develop a rubric to rate case solutions from a problem solving and creativity angle. The rubric started with the following instruction: “In order to use the rubric efficiently, we suggest you the following criteria to be applied to each of the answers about the case. Task 1 could be evaluated using the criterion Define problem; Task 2 with Identify strategies; Task 3 with Propose solutions, Innovative thinking, Taking risks; Task 4 with Evaluate Potential Solutions, Acquiring competencies, and Connecting, synthesizing, transforming”. The former induced a systematic approach in which raters first read the student input for each task and next applied the specific rubric criterion. This resulted in defining an explicit performance level (A, B, C, D, E). Some examples of the answers for question #3 (Propose and describe at least one innovative solution for the problem defined in question 1) were:

Example 1: “Students must take a cycling simulation test that recreates real-life campus’ road conditions to verify that students know how to use the bicycle correctly on different types of slopes. If a student passes the test, he will be allowed to use the bicycles”

Example 2: “Visible and intuitive signs will be placed along the road of the bicycle lane to let cyclists know the moment and the type of gear shift that should be applied to the bicycle, depending on the peculiarity of each section of the road.”

Example 3: “An online training for students embedded in one of the university’s technology platforms about the correct use of bicycles on campus. At the end of the training, students must take a test, and those who pass the test will be registered on a database of people allowed to borrow a bicycle on campus.”

Moreover, a pilot study collected feedback from course facilitators and undergraduate students (not involved in the present study) regarding the research instruments. Thus, this feedback allowed the authors to check the translation (from English to Spanish) and the consistency in interpretations. Note that these VALUE rubrics were not shared as part of the course; i.e. they were not part of the learning process.

Prior research has reported acceptable validity and reliability of VALUE rubrics (Finley, 2012; Mcconnell & Rhodes, 2017; Rhodes & Finley, 2013; Simper, 2018). Nevertheless, the translated and adapted versions of the instruments were reassessed in the context of the present study using univariate and multivariate techniques. Table 2 summarizes reliability analysis of the instruments at t0, t1 and t2 obtaining Cronbach’s alpha; i.e., how closely related rubric items are as a group. Given the use by three different raters, the coefficients suggest that the items for each skill (problem solving and creativity) have relatively acceptable internal consistency.

Table 2 Cronbach Alpha’s results for each rubric items, per rater, at each time point
Full size table

To check construct validity, the problem solving and creativity items were considered in an exploratory factor analysis (principal component analysis with Varimax rotation) by imposing a two-factor solution (see Table 3). The analysis was repeated on the base of data from students, peers, and the facilitators at t0, t1 and t2. The analysis results show how all four creativity items load on the same factor (Factor A) at t0, t1 and t2 based on data from the three actors, except when processing t0 student self-evaluation. The problem solving items loaded on the second factor (Factor B). However, two items often loaded on the creativity factor (“Propose Solutions” and “Evaluate Potential Solutions”). At t2, most of the items fitted the specific factor as expected. Thus, the results reiterate the conceptual overlap between problem solving and creativity as stated earlier (Sawyer, 2006). In sum, this factor analysis allow us to conclude that the instruments were valid in the present context.

Table 3 Results of exploratory factor analysis for Problem Solving and Creativity rubrics’ items
Full size table

Research procedures

Building on the course structure, data were collected three times (t0, t1, t2) during actual class time and using paper-based forms. During the first week the first data collection was carried out (t0), then 8 weeks later (week 9) for a second time (t1) and 6 weeks later (week 15) for the last time. Given the natural (or authentic) setting of this study, the number of students participating in each data collection point differed. Details about the number of participants and procedure of the data collection in each time point and per rater are presented in a chart that can be found at the Online Resource 3. Each data collection session lasted about 60–75 min. The data collected during this research did not affect students’course grades. Results of this study are not reported at the level of course’s sections.

Before the intervention, the 26 facilitators agreed to participate in this study. Prior to the study, they received a training session, which focused on data collection and the application of rubrics to evaluate students’ performance in terms of problem solving and creativity. Hence, a standardized procedure was followed by all facilitators as they received instructions and a “kit” with the forms for the collection sessions.

At t0, in each course section, the students signed an informed consent, filled out a background questionnaire, and received the case study next to a sheet with the tasks/questions to be answered. Once the time for reading the case and answering the tasks was up, the facilitator collected all the answers’ sheets. Immediately after this part of the session, the facilitator assigned randomly to each student one answer sheet from one of the classmates (blinded version) and a rubric (see Research Instrument section) to rate the answers. At that stage, the facilitator explained how to use the rubric. Once completed, the facilitator collected the original answer sheets and filled out rubrics. Next, students were given their own answer sheets (identified by a personal code) as well as the same rubric to rate their personal answers. At the end of the session, the facilitators collected all sheets. Afterwards, the facilitators applied the rubric themselves to evaluate the work of each student. To facilitate this part of the process, SurveyMonkey software was used. The same procedure was applied at t1 and t2. As stated earlier, students were not aware of when and how they would be evaluated.

Analysis approach

To test the two hypotheses on whether students would show a higher performance in their problem solving (H1) and creativity (H2) skills as a result of a design thinking course, the researchers adopted repeated measures ANOVA. In addition, the perspectives of the students, their peers, and the facilitator were considered. The analyses were carried out with SPSS version 25. A significance level of p < 0.05 was put forward. Effect sizes (Cohen’s d) were additionally calculated in view of interpreting the importance of the analysis results.

Results

Descriptive data analysis

Table 4 summarizes descriptive data of the dependent variables at t0, t1, t2. The results of the attained average scores reflected an apparent improvement over the time in all cases.

Table 4 Descriptive results of the dependent variables
Full size table

Figure 2 illustrates a consistent increase in problem solving scores according to the raters (students’ self-evaluation, peers, facilitators). Figure 3 shows the students’ performance in creativity based on the three raters. Hence, there it is evident an overall improvement from t0 to t2.

Fig. 2
figure 2

Mean in “Problem solving” scores (N students = 693; N peers = 682; N facilitators = 607)

Full size image
Fig. 3
figure 3

Mean of “Creativity” scores (N students = 691; N peers = 682; N facilitators = 607)

Full size image

The results of a correlation analysis of the scores are presented in Table 5. The correlation measures describe positive and significant correlations.

Table 5 Correlations of evaluators’ ratings
Full size table

Hypothesis 1 (H1): students who participated in a design thinking course improve their problem solving skills.

Self-evaluation results: the repeated measures within-subjects ANOVA show a statistically significant effect on problem solving performance over the time, according to the students’ evaluation, as presented in Table 6. When looking at the analysis of contrasts results, we find a significant change from t0 to t2, as well as from t0 to t1, and from t1 to t2. We observe an effect size of d = 0.85 in the change from t0 to t2. The effect size between t0 and t1 is larger than between t1 and t2.

Table 6 Within-Subjects ANOVA and Cohen’s d test results for Hypothesis 1 (problem solving)
Full size table

Peers evaluation results: problem solving performance seems significantly affected over the time (see Table 6). Contrasts results reveal a statistically significant increase from t0 to t2, as well as from t0 to t1, and also from t1 to t2. In contrast to self-evaluation’s effect sizes, a smaller effect (d = 0.60) is observed when comparing t0 and t2. The effect size from t0 to t1 is larger compared to t1 to t2.

Facilitators evaluation results: repeated measures ANOVA reflect that problem solving performance shows a statistically significant improvement over the time (see Table 6). When zooming in the contrasts results, we observe significant changes from t0 to t2, as well as from t0 to t1, and also from t1 to t2. The ratings provided by facilitators show the largest effect size from t0 to t2 (d = 1.17), compared to the other two raters. The effect size between t0 and t1 is much larger than the change from t1 to t2.

Therefore, we can reject the null hypothesis and accept H1: Students who participated in a design thinking course improve their problem solving skills.

Hypothesis 2 (H2): students who participated in a design thinking course improve their creativity skills

Self-evaluation results: the repeated measures within-subjects ANOVA results reveal a statistically significant increase in creativity performance over the time (see Table 7). When looking at the contrasts results, we observe statistically significant changes from t0 to t2, as well as from t0 to t1, and also from t1 to t2. We observe an effect size of d = 0.94 when comparing t0 to t2. The effect size of changes between t0 to t1 is larger than the changes observed from t1 to t2.

Table 7 Within-Subjects ANOVA and Cohen’s d Test results for Hypothesis 2 (creativity)
Full size table

Peers evaluation results: following the above statistical analysis approach, Table 7 shows that creativity performance has a significant increase over the time. Moreover, contrasts results indicate statistically significant improvements from t0 to t2, as well as from t0 to t1, and also from t1 to t2. There is a slight smaller effect size related to changes from t0 to t2 (d = 0.79), compared to the one observed in students’ self-evaluation. The effect size from t0 to t1 is larger compared to t1 to t2.

Facilitators evaluation results: creativity performance shows a statistically significant improvement over the time, as shown in Table 7. When looking at contrasts results, we also found significant changes from t0 to t2, as well as from t0 to t1, and also from t1 to t2. Similar to what happened in Problem Solving results when analyzing facilitators’ ratings, this type of rater reflects the largest effect size from t0 to t2, (d = 1.28), compared to students’ self-assessment and peers. The effect size of changes from t0 to t1 is considerably larger than the effect size related to changes from t1 to t2.

Therefore, we reject the null hypothesis and accept H2: Students who participated in a design thinking course improve their creativity skills.

Discussion

The present study explored the effects of design thinking on problem solving and creativity skills in first-year university students at three points in time (t0, t1, t2). A key aspect of the present study is the multi-actor perspective since the data were collected from different sources: self-reports, from peers and from facilitators. In addition, a strength of the present study is the robust design based on an operational definition of design thinking and the adoption of reliable and valid measures to study the impact on problem solving and creativity.

According to the data collected from the raters, the design thinking intervention had a positive significant impact on students’ problem solving (H1) and creativity (H2) skills when comparing t0 vs t2, as well as the other time slots t0 vs t1, and t1 vs t2 (see Table 6 and Table 7). A relevant observation is that from t0 to t1 the effect size of related changes seems to be larger than from t1 to t2. Nevertheless, all the time slots show a positive and statistically significant improvement.

Note that by t1, students had experienced the three design thinking stages: Research, Empathy and Define. Thus, in general terms, students were expected to adopt a human-centered approach, develop insights into their project-related problems and (re)defined their specific problem focus. At t2, the students had finished the other three design thinking stages: Ideate, Prototype, Validation; i.e., in the second part of the course students were encouraged to generate as many solution ideas as possible and tested the best ones with real users until proposing one that better fits to the problem previously defined. The scope of this study does as such not allow to determine at what specific stage the strongest development in problem solving or creativity skills did occur. Nevertheless, based on the techniques and tools students adopted at each stage, we might conclude that the first diamond of design thinking (research, empathy, defined) could contribute to a larger extent to the development of problem solving skills; while the second diamond (ideate, prototype validation) might have contributed more to boost students’ creativity skills.

Regarding Hypothesis 1, the results can be linked to the main features of the design thinking design and characteristics that were inspired by constructivist learning theory (Pande & Bharathi, 2020). The systematic mapping of problem solving phases onto the design thinking phases might have offered students sufficient structure and support to deal successfully with the complex problem solving projects; or as Lindberg et al. (2011) pose: Design thinking promotes a problem exploration space where instead of creating general hypotheses or theories about the problem, people get an intuitive (not completely verbalized) understanding through observing exemplary use cases or scenarios; and synthesizes this information to point of views. Our findings about the positive impact on problem-solving skills after a design thinking intervention confirm earlier research (Alhamdani, 2016; Bhatnagar & Badke-Schaub, 2017; Khalaf et al., 2012; Lee & Benza, 2015; Linton & Klinton, 2019; Lugmayr et al., 2014; Matsushita et al., 2015; Taajamaa et al., 2013). In addition, the emphasis on structured collaboration, explicit reflection on experiences, plans and products and the explicit availability of feedback by the facilitators and ‘sponsors’ might have boosted the design thinking potential. The importance of structure in the adoption of constructivism is a constant in the literature (Alesandrini & Larson, 2002). However, the lack of structure is often used to explain failure in the adoption of constructivist learning theories (see Kirschner et al., 2006). Therefore, current constructivist approaches often explicitly stress and embrace a ‘guided discovery’ principle (Mayer, 2004).

Furthermore, hypothesis 2 was also accepted. Creativity is often looked at through a constructivist theoretical lens (Edwards-Schachter et al., 2015) and it is one of the main skills associated with the outcomes of design thinking (Tsai, 2021). Prior research reported a positive effect of design thinking on the development of creativity skills (Balakrishnan, 2021; Benson & Dresdow, 2015; Clemente et al., 2017; Lee & Benza, 2015; Saggar et al., 2017; West et al., 2012). To explain our positive results, we can refer to specific design thinking design guidelines that pushed students’ divergent thinking in a pre-structured way. As Lindberg et al. (2011) point out there is a ‘solution space’ in design thinking where people are asked for a vast number of ideas in parallel and to make these explicit through sketching and prototyping techniques. In this way, ideas are expected to be converted into cognitive representations. Both spaces, problem space and solution space, in design thinking generate “a system of checks and balances to ensure that the conclusive solution will be both innovative and suitable for the social system that the design problem addresses” (Lindberg et al., 2011, p. 6). Since the design guidelines offer structure, Sawyer (2011) stresses the need to balance this structure with more open instructional approaches when pursuing an impact on creativity. For example, one study based on self-perception questionaries (Ohly et al., 2016) found no design thinking-impact on creative self-efficacy after an intervention (n = 69). According to the authors of that study, the group format used in that course as well as a weak focus on creativity (shorter than a third of the course) may be the cause. The latter reinforces the importance of a proper structure, planning and guidance by the facilitator during a design thinking course which is fundamental, especially for promoting creativity.

Our findings for both hypotheses show a decrease in effect size (Cohen’s d) in the facilitator scores when comparing t0 to t1 versus t1 to t2 (Table 6 and Table 7). Thus, the facilitator expectations might have introduced bias in students’ evaluation (Jussim, 1989). In fact, some facilitators reported that their expectations on students’ performance were higher at the end of the course than at the mid-term. Nevertheless, it is interesting that for both skills, changes in facilitators’ scores reflect the largest effect sizes from t0 to t2, compared to students’ self-evaluation and peers’ scores in the same time slot. The above suggests the largest effect sizes in both skills’ improvement is related to the level of experience of raters. In the context of this study, facilitators could be considered the best trained raters, not only because their knowledge and experience in design thinking; but also because of their experience as teachers in the Analysis and Problem Solving course and other courses.

A key limitation of the present study is the lack of a control group. This is difficult to achieve since all first-year students in this university must enroll in the Analysis and Problem Solving course. Hence, future research could compare first-year scores from students enrolled in a different university in comparable programs. A further limitation to be addressed in future research is the background variables of students (professional experiences, gender) and the potential impact of mediating variables, such as motivation and/or self-efficacy. Moreover, the multi-actor focus of the present study could also be enriched by adding data resulting from qualitative research. As exemplified above, perceptions of the facilitators might be biased due to their expectations of progress related to time. It would be also interesting to complement the results obtained with the projects developed by students’ teams as part of the course. However, since the projects represent a significant percentage of the course grade, we prefer not to include them as measurement instruments since this would violate the informed consent agreement and could have introduced bias. As mentioned earlier, there is research pointing at the validity and reliability of specific VALUE rubrics; nevertheless, in our current context an inter-rater reliability study could result in more convincing arguments to adopt the instruments used. Finally, some readers may have a concern about a potential Hawthorne effect; however we rule out that risk since (a) students were informed that their participation in this course would not affect their grades; (b) students did not know exactly when and how they would be evaluated; (c) students’ names and personal information remained anonymous during the entire process; (d) students did not have access to consult their case’s answers at any point of the study; and the instruments as well as the results related to the study were not discussed during the semester.

Conclusions

The current study was an attempt to address critical features of current and past empirical research to study the impact of design thinking on creativity and problem solving skills of students in higher education. Therefore, we aimed at answering the call of Razzouk and Shute (2012) to set up a new wave of design thinking research that builds on valid performance-based assessments to examine the effects of the design thinking process on various students’ skills. Building on data from a large sample size and an intervention that operationalized design thinking design principles in a replicable way, we added a multi-actor focus to the impact study by looking at the performance ratings from students, peers and facilitators. In addition, the researchers are open to exchange their approach, materials, procedures and research instruments to facilitate a wider adoption of design thinking in higher education and achieve the key aim of developing the skills demanded by labor markets.