Introduction

Over the past decade, there has been increased interest in bringing engineering design projects into pre-college settings in North America (Academies & of Sciences, Engineering, & Medicine [NASEM], 2020; NGSS Lead States, 2013), Europe (Clark & Andrews, 2010; Jones et al., 2018), Asia (Salas-Pilco, 2021) and others. One prominent motivation behind this push to include engineering design in pre-college settings is that these projects can help students develop interdisciplinary skills that transcend engineering problems, including design thinking (Dym et al., 2005; Li et al., 2019; Razzouk & Shute, 2012) and reflective practices or reflection skills (Schimpf et al., 2020; Goldstein et al., 2019; Crismond & Adams, 2012). Reflection is sometimes examined as a core component of design thinking (Cross, 2011; Kimbell, 2011) but has also received considerable attention as a broader skill applicable across many areas in education (Jarvis, 2006; Moon, 2004; Rogers, 2001; Mezirow, 1991). Reflection is a mental process whereby someone returns to a past experience or idea in order to inform one’s next action or decision (Turns et al., 2017; Moon, 2004). Thus, reflection is a critical skill students can develop from engineering design projects that can be transferable across design and non-design problem-solving contexts.

Despite the promise of design projects in pre-college settings, how these younger students learn to become designers and develop design skills, practices, and mindsets is not well understood (Goldstein et al., 2021; Dasgupta, 2019; Wendell et al., 2017), in stark contrast to the wealth of research on undergraduate and professional designers (Atman, 2019; Rahman et al., 2019; McComb et al., 2017; Crismond & Adams, 2012; Cross, 2011; Lawson & Dorst, 2009; Atman et al., 2007). Moreover, while reflection has been studied for undergraduate students and professions, these studies tend to focus on the outcomes of reflection (Ali, 2020; Baaki et al., 2017; McKenna et al., 2009; Tracey & Hutchinson, 2018) or mechanisms to support reflection or reflective practice (Moore et al., 2016; Walther et al., 2011; Kavanagh, 2008) but not how students or professionals engage in reflection as a practice. Further complicating this research is the fact that both the design process and how design thinking, and associated skills are enacted are highly fluid (Calgren et al., 2016; Dorst & Cross, 2001). Following this, we expect reflection strategies for pre-college students to manifest in multiple emerging productive and perhaps some less productive ways (Engel & Conant, 2002). A greater understanding of these productive or unproductive early reflection strategies can help teachers and researchers more clearly recognize these practices and develop new approaches to support student learning.

A key challenge remains, however, in how we accurately and reliably capture students engaging in these productive beginnings but early and likely underdeveloped practices or strategies, particularly during a complex activity like engineering design. Classroom observations can capture these, but not all students can be observed simultaneously and having a larger observation team (e.g., researchers) is impractical and may be obtrusive to the classroom. Moreover, while videotaping can capture these strategies, analyzing video data is time-intensive and often only practical in research contexts. Learning analytics can offer an alternative for capturing students design strategies and practices that is unobtrusive (Shute, 2011), and scalable for all students in a classroom setting (e.g., Vieira et al., 2018) and produces rich and reliable data (Rahman et al., 2019). Learning analytics leverage digital platforms to collect detailed student behavior to support analysis of this data toward understanding and supporting learners (Siemens, 2013). In recent years learning analytics has been applied to studying engineering design to better understand how design skills, practices, and strategies are developed or can be supported (Schimpf & Xie, 2017; Akhtar et al., 2017; Vieira et al., 2016; Worlsley & Blinkstein, 2014). In this work we use the computer-aided design platform Energy3D (see Xie et al., 2018), for designing eco-friendly homes and renewable energy systems, with middle school students to study their reflection strategies. Energy3D was created for pre-college settings and features a dedicated design journal embedded within the system so students can pause and reflect at any time during or after a design session and an action-logging system that captures students’ design actions, including interactions with their design journal, as they work on their project.

Our driving research questions for this work ask:

  1. 1)

    What reflection strategies do students engage in throughout their design process?

  2. 2)

    To what extent do students’ reflection strategies relate to their final artifact design performance?

In the remainder of the paper, we present an overview of how engineering design is used in pre-college environments, review research on reflection, and then cover learning analytics and how it has been incorporated into design education research. Next, we review the study context, the CAD tool and data collected. Afterwards, we explain our analytical approach. Our findings then cover the cluster analysis results for reflection strategies and the regression analysis. We then discuss the findings and conclude the study.

Literature review

The literature review has three major sections. First, we review engineering design’s increasing use in pre-college settings and why it is important to study this population in particular. Second, we cover what reflection is and describe reflection research to date. Third, and finally, we review learning analytics in general and discuss learning analytics in engineering design research.

There is a growing interest in incorporating and expanding engineering learning experiences into pre-college settings, often through design projects. This trend is driven in part by initiatives such as the A Framework for K-12 Science Education (National Research Council 2012) and the Next Generation Science Standards (NGSS Lead States, 2013) in the United States, which present a new vision for science education that encompasses engineering practices. Several other ongoing challenges and research findings have unveiled numerous reasons for deeper integration of engineering design into K-12 or pre-college settings, including: providing an integrative STEM learning experience (English et al., 2020; Moore et al., 2014), supporting learning in other STEM domains, including science (Aranda et al., 2020; Purzer & Quintana-Cifuentes, 2019) and math (English et al., 2020; Chiu et al., 2013); raising interest and awareness of engineering at a time when enrollment in the field is not keeping pace with demand (Polmear et al., 2023; Brophy et al., 2008; Clark & Andrews, 2010); helping students develop interdisciplinary or transferable ways of thinking, skills, and practices including design thinking (Schimpf et al., 2022; Li et al., 2019).

However, as noted previously, while there is a substantial body of research on how undergraduates and professional develop and exhibit design skills, practices and mindsets (Atman, 2019; Crismond & Adams, 2012; Cross, 2011) there is considerably less research on how precollege students develop these skills. As Siverling et al., (2021, p295) argue, “As engineering is still relatively new to precollege education, it is important to explore how precollege students engage in developmentally appropriate engineering practices and use these practices to learn science, technology, engineering, and mathematics (STEM) content.” The work of Crismond and Adams (2012) is often cited in this space, despite being focusing on design skills and practices for undergraduate students. This is in part due to the comprehensiveness of the work, which synthesizes findings on many key design skills including problem scoping, concept generation, experimentation, and reflection, but also due to their identification of the informed designer. The informed designer is an intermediate state between novice and expert designer and can serve as a more plausible set of learning targets for precollege students. For example, on reflection as a design skill, they contrast a novice designer who would “do tactic designing with little self-monitoring while working or reflecting on the process or product when done” whereas an informed designer would “practice reflective thinking by keeping tabs on design strategies and thinking while working and after finished” (Crismond & Adams, 2012 p749). In short, their informed designer designation gives educators learning outcomes to target for developing design skills or practices, while recognizing mastery or expertise is unlikely to develop for students who will often have limited engineering experience.

An important related concept to understanding and measuring early or emerging skills or knowledge is productive beginnings (Goodhew & Robertson, 2017; Watkins et al., 2014). Productive beginnings has its origins in science education research and refers to emergent understandings, abilities, or resources that students begin to develop, and which can lead to deeper understanding or abilities if fostered (Goodhew & Roberston, 2017; Hammer et al., 2012; Engle & Conant, 2002). In science education productive beginnings has been used to examine students early conceptual knowledge or scientific thinking. For instance, Hammer et al. (2012) analyze how third-grade students engaged with the concept of energy over a series of lessons and unveiled students’ productive beginnings of thinking scientifically about energy. Productive beginnings is starting to be used in engineering education research in areas such as engineering judgement (Swenson et al., 2019) and engineering design practices (Watkins et al., 2014). Watkins et al. (2014) analyzed fourth grade students’ conversations about design problems to identify the ways these students scoped or framed the problem. They argued that past research oversimplified how problem scoping was examined in younger students which resulted in missing or misinterpreting students’ practices. Using in-depth analysis of video recordings of student’s discussions about a problem that was presented to them in class, they identify several ways students productively engaged in problem-scoping including reflecting on the nature of the problem and weighing or considering different criteria and constraints. Despite having little design or engineering experience, the students exhibited productive beginnings of problem scoping. Thus, taken together, the informed designer and productive beginnings concepts can help guide analysis of precollege students early or emerging capabilities with design skills and practices.

Reflection is generally defined as a type of thinking or mental process used to revisit or reconsider a prior experience, idea, or other mental structure to then  take some action or achieve an outcome (Turns et al., 2017; Moon, 2004). The objects of reflection may be wide and varied, but typically we reflect on things we have experience with or understanding of, although external sources may also inspire reflections (Moon, 2004). Some authors have extended this general definition, in particular Schön, who distinguishes between reflection-in-action and reflection-on-action (Schön, 1983). According to Schön, reflection-in-action involves reflecting on some aspect of an active practice one is engaged with (e.g., an engineer presently designing). In contrast, reflection-on-action involves revisiting an aspect of a prior experience or practice (e.g., revisiting a previous project that may inform a current project).

Reflection, as either a professional practice or a component of education, has received substantial attention across several fields. One notable thread of research has focused on the ways to support students to reflect on their own learning or development (Moore et al., 2016; Walther et al., 2011; Kavanagh, 2008; Luppicini, 2003; Turns et al., 1997). For example, Walther et al. (2011) propose a framework of five indicators tied to students’ emotional responses related to critical learning experiences. The indicators are novelty, challenge, progression, exploration, and insight and each are associated with different emotional responses to learning experiences. The researchers developed a series of prompts based on these indicators in order to facilitate reflection on past learning and report some success in the efficacy of the prompts for engineering and art students.

Another large area of reflection research analyzes how reflection can encourage important learning outcomes (Ali, 2020; Baaki et al., 2017; McKenna et al., 2009; Moon, 1999; Tracey & Hutchinson, 2018; Valkenburg & Dorst, 1998). For instance, Tracey and Hutchison (2018) used reflective writing to facilitate professional identity development for graduate students taking an online design course as part of a Learning, Design and Technology degree program. In their reflections, students engaged deeply in practices that assisted identity development, such as authorial presence and analysis; however, discussion of emotions showed the lowest occurrence in student responses. They draw implications for assisting identity development and how to better support students' emotional self-awareness in regard to becoming designers.

Although there has been considerable research on how to support reflection and the impact reflection can have on various academic or other student/practitioner outcomes, research on the process of reflection is more limited. Prior work seeking to explain reflection as a process or practice has focused on or been embedded in prescriptive models like Kolb’s Experiential Learning Model (1984) and Jarvis learning model (2006) or as theoretical constructs such as Mezirow’s critical reflection (1991) and Schön’s (1983) reflection-in-action and reflection-on-action. There have been fewer analyses focusing explicitly on the naturalistic, situated, and contextualized ways students or professionals engage in reflection as a fluid practice, particularly for the area of engineering design.

Turning to learning analytics, Siemens (2013, pp 1382) has defined the area in this way: “[l]earning analytics is the measurement, collection, analysis, and reporting of data about learners and their contexts for the purposes of understanding and optimizing learning and the environments in which it occurs.” Learning analytics rely on commercial or research technology platforms for collecting said data, typically operating stealthily or unobtrusively in the background of the technology (Rahman et al., 2019; Shute, 2011). In other words, data is collected from students without interrupting their learning experience within the platform, in contrast to surveys, question prompts or other modes of data collection. Some common tools that have been used to collect data for learning analytics include learning management systems (Marks et al., 2016; Cerezo et al., 2016), game-based learning environments (Shute & Ramihi, 2021; Serrano-Laguna et al., 2017), and simulation tools (Hernandez-Lara et al., 2019; Wen et al., 2018). Although still emerging as an analytical approach, learning analytics has been growing in popularity with several dedicated literature reviews published (e.g., see Aldowah et al., 2019; Mangaroska & Giannakos, 2019) and conferences such as International Conference on Learning Analytics and Knowledge. Moving specifically to the study of design learning or design education, learning analytics are also gaining more attention (Vieira et al., 2018; Akhtar et al., 2017; Worsley & Blinkstein, 2018; Schimpf & Xie, 2017; Vieira et al., 2016; Worlsley & Blinkstein, 2014). For example, Vieira et al. (2016) used a computer aided design (CAD) system for creating sustainable homes that logged students’ actions to identify the design experiment strategies of 55 middle school students, including systematic and non-systematic experiments. The results revealed a correlation between the number of experiments run, both systematic and non-systematic, and the quality of students' final design models. In another example, Akhtar et al. (2017) studied 331 undergraduates from a CAD drawing module in a lab setting and examined several learning analytics including students’ average time spent in class, total attendance, and average seating row in the lab. A one-way ANOVA was used to identify the statistical significance of the factors mentioned above in relation to student achievement. They discovered a positive relationship between attendance, time on task, and sitting with peers and a negative relationship between students’ distance from the lecturer. These studies show some of the ways learning analytics have begun to be applied in the study of design education.

Study context and methods

Participants and classroom context

This study was conducted at a large public urban middle school located in the Midwest United States that enrolls over 90% of its students from minority populations. Our total sample was 105 students (ages 13–14) from four science classes. Seventeen students were eliminated from the analysis. Fourteen of these were removed as their reflection levels within their design process were negligible, and another three were eliminated for having incomplete final design artifacts, ultimately leaving us with 88 students for the study.

The Net-Zero Energy design challenge presented to students asked them to individually design an eco-friendly house by creating three unique designs that would minimize energy consumption annually, either through conservation or generation of energy or both. Students were instructed to stay within a $250,000 budget, with a requirement that each side of the house needs at least one window and no more than 40 total solar panels. This project was completed in Energy3D, a computer-aided-design (CAD) platform that estimates energy generation and consumption through a simulation engine, along with an integrated design journal (Xie et al., 2018). The platform was explicitly built for precollege students to engage in design challenges and simplifies several features from professional CAD software while still allowing students to creatively design energy efficient homes. For example, there are several prebuilt parts (e.g., roofs, walls, doors) that can be clicked and dropped into the design space which can then be manipulated or transformed by the student, instead of requiring each part to be assembled from the ground up. Additionally, the curriculum was designed to be age-appropriate while retaining sufficient challenge for middle to high school students; moreover, the project has been run successfully in several other settings with similar age groups of students (Goldstein et al., 2021; Magana et al., 2021; Goldstein et al., 2018; Vieira et al., 2016). Students worked individually under the guidance of one science teacher over the course of two weeks to design their solutions to the design challenge. The science teacher had over 25 years of experience teaching general science courses and four years of experience teaching engineering and modeling with Energy3D. She introduced the software and design challenge to the students’ using computers loaned from a local University due to limited resources at the middle school. In the beginning of the project the teacher emphasized the importance of reflection for engineering and instructed students to use the embedded journal in Energy3D to take notes and reflect on their design ideas throughout the project. The teacher highlighted that documenting design changes would be helpful for creating a successful solution. Figure 1 illustrates a student house built in Energy3D with an accompanying design journal entry at the bottom of the screen.

Fig. 1
figure 1

Energy3D Interface with example student model and design journal

Full size image

Past research has elaborated several reasons that may make design problems complex including: (1) they often require knowledge across multiple domains or systems (Schimpf & Singh, 2022); (2) they consist of a large number of components and the distinct relationships between components (Summers & Shah, 2010); and (3) they are ill-structured with multiple viable solutions and multiple paths or processes to a viable solution (Jonassen, 2000). It is also possible to constrain some of these aspects of design problems to reduce their complexity. In the Net-Zero Energy challenge the initial problem is constrained by providing students with a broad solution direction – building an energy efficient and eco-friendly home. However, several other factors keep the problem complex. Students need to draw on knowledge about houses and their components, the environment and earth systems (such as seasonal differences), and energy systems including natural (the sun) and human-made (solar panels). There are many components’ students can design with including walls, roofs, windows, doors, trees, solar panels and others, and there are a variety of relationships between any given set of components. Finally, there are multiple viable solutions students could pursue as well as multiple problem-solving processes to reach these. As such, the problem is relatively complex, particularly for this age group, and a problem context like this provides an opportunity to investigate students’ unique reflection strategies or processes.

Data collected

Data was collected over the approximate two weeks of the design challenge and included: recording learning process data from Energy3D and a final design artifact from Energy3D. Process data in this project was continuously logged as students addressed the design challenge in Energy3D (please note Energy3D has been renamed to Aladdin for future users). The log data, or process data, contains all of the design actions students perform such as construction activities (e.g., add wall, edit wall), analyses (e.g., run annual energy analysis), simulations (e.g., animate sun, show shadow, etc.), and journal documentation activities. Design artifacts are the final saved versions of student houses. These files can be opened in Energy3D to view the form and aesthetics of the final design, and run analyses to find the final construction costs and annual energy use. Thus, this design artifact file allows both a high-level look at how well a student approached the design task, as well as a very detailed look at how students address design criteria and constraints. Previous research used this final design artifact to calculate a design quality score that encompasses multiple complementary and competing dimensions called the Trade-off Value (Goldstein et al., 2021). The Trade-off Value is calculated as the sum of the percentile ranks of student performance concerning human dimensions, technical dimensions, and economic performance. The maximum score is a function of the number of trade-off dimensions involved within the particular design challenge and is a maximum of 400 for this particular context (see Goldstein et al., 2021 for more details). A higher Trade-off Value represents a final design artifact that simultaneously attends to design dimensions of cost, performance, and desirability, while a low Trade-off Value represents a final design artifact that addresses only one or two design dimensions at the expense of the other dimension(s).

Data analysis

In this study, we use cluster analysis, as a way of separating data into clusters based on similarities and finding unknown groups within the data. We used the R software package to conduct our analysis. The particular form of cluster analysis we use is kmeans, which takes each case in a dataset and assigns it to a set number of clusters. Kmeans assigns a case to a cluster on similarity between the values of the case and the mean values or centroids of the cluster (Jain, 2010). Kmeans attempts to reduce the sum of squared error between the clusters by taking the data, assigning a case to a cluster based on similarity scores that compare absolute distance between variables on a new case with cluster profiles, and then iteratively it moves other cases to clusters depending on level of fit (e.g., distance from centroid). Kmeans results display cluster sizes (how many cases fall into each cluster), cluster means (means of each cluster for each variable in the data), and total sum of squares (amount of total variation accounted for by the cluster assignments).

This study builds on the analysis and feature extraction originally reported in Schimpf et al. (2020). The different features extracted for this study are as follows: Duration (percentage of sessions students reflected in), Intensity (average degree of reflection across sessions), and Lean (average percentage of design session dedicated to reflecting). Note that for Intensity, it was normalized across the minimum and maximum values across all participants with reflection data. The lowest Intensity value was 24, or 5–6 words per session whereas the highest was 321 or about 65–80 words per session. Here design sessions refer to separate class periods in which students worked on their design project. For this study, we took the data logs portraying the number of times students reflected during their design process in Energy3D. We used a kmeans to group students into different clusters representing their reflection strategies. In the data analysis discussion we present what kmeans model we selected and present the details of the model itself in the results. Potential cluster solutions from two to six were examined to find the best representation of the underlying data. Larger cluster solutions (seven or more) resulted in small, fragmented clusters and were therefore not considered. Silhouette plots were generated, which visualize cases fit within clusters and how close the data points are to neighboring clusters (e.g., see Schimpf & Castellani, 2020). The plots present the number of clusters, how many cases are in each cluster, and the measure of fit within the cluster (-1 to 1 scale). And overall average width or fit across all the clusters is generated by the silhouette plots; higher average width represents better fit. We also looked at to each solution’s sum of squares explained, which indicates how much of the variation in the data is accounted for by the cluster grouping. Lastly, we reviewed the cluster profiles themselves, to identify the best cluster solution. This represents a multidimensional evaluation approach that leverages both statistical metrics (silhouette plots, sum of squares) and domain expertise (profile differences relative to the domain of engineering design reflection). We used this process to evaluate cluster solutions two through six. The results for the sum of squares and silhouette plots average width can be seen in Table 1. Notable jumps in sum of squares can be seen as the solution went from K = 2, to K = 3 and K = 4, with increases of 10% or greater; afterward gains became smaller. Additionally, the average width for the silhouette plots increase until K = 4 and then begin to decrease. Examining the cluster profiles themselves K = 3 was similar to K = 4 but lacked one of the groups presents in the four cluster solution. K = 5 and K = 6 added more groups, but these were similar to profiles for groups already present in K = 4, suggesting the algorithm was splitting existing groups rather than identifying new ones. Given these three metrics, we decided on cluster solution four which provided us with four clear groups.

Table 1 Description of K-means outputs for K = 2 to K = 6
Full size table

Finally, we conducted a regression analysis using students cluster IDs or which cluster they were in to predict their tradeoff values. Cluster ID was turned into a set of dummy variables for this analysis. This meant each student had a one representing their presence in a cluster and zeros representing the clusters they were not in. We selected a reference group in order to have a base that we could then compare the other clusters. As shown in the results below, we selected the lowest cluster in Duration, Intensity, and Balance as it allowed for a good comparison against the other strategies used. Our hypothesis was that students with reflection strategies that exhibited high duration or intensity would have greater performance on their final tradeoff values.

Results

This section presents the four clusters we identified through kmeans, each representing a unique reflection strategy. After providing an overview of the clusters, we share exemplars that visualize how each strategy was employed. Following our past work (Schimpf et al., 2020), these illustrative examples are visualizations of students' design processes showing when students reflected, modeled, and analyzed their eco-friendly home over several design sessions. Next, we present a summary of the tradeoff values students achieved in each reflection strategy group. Lastly, we present the regression results.

Reflection strategies

Table 2 presents the kmeans results. The table shows four student reflection processes using Duration, Intensity, and Lean, as described in the methods section. Cluster 1, the Consistent Heavy Modeling Leaners, (N = 13) represents students that spent a medium amount of sessions reflecting (medium Duration), with a heavier amount of reflection in each session (high Intensity), and their overall process across sessions had less emphasis on reflecting (modeling Lean). Cluster 2, the Sporadic Light Modeling Leaners, (N = 21) represents students that spent a low amount of sessions reflecting (low Duration), with a low amount of reflection in each session (low Intensity), and their overall process across sessions had less emphasis on reflecting (modeling Lean). Cluster 3, Frequent Heavy Non-Leaners, (N = 9) represents students that spent a high amount of sessions.

Table 2 K-means clustering output for K = 4
Full size table

reflecting (high Duration), with a high amount of reflection in each session (high Intensity), and their overall process across sessions had an even emphasis on reflecting (No Lean). Cluster 4, Frequent Light Non-Leaner, (N = 45) represents students that spent a high amount of sessions reflecting (high Duration), with a low amount of reflection in each session (low Intensity), and their overall process across sessions had an even emphasis on reflecting (No Lean).

Figure 2 shows one example of how the reflection strategy Consistent Heavy Modeling Leaner was realized. This particular student has roughly evenly split their days between mostly reflection or mostly design (i.e. constructing their house), with a medium-to-higher intensity of reflection episodes for those sessions where they reflected (as will become clearer with subsequent exemplars).

Fig. 2
figure 2

Student exemplar of reflection strategy consistent heavy modeling leaners (cluster 1)

Full size image

Turning to Fig. 3, a stark contrast with the prior reflection strategy is observed. For this exemplar, reflection is infrequent or light, and what reflection happens is low in intensity. Their process is dominated by design or modeling actions, like above.

Fig. 3
figure 3

Student exemplar of reflection strategy sporadic light modeling leaners (cluster 2)

Full size image

The exemplar for Frequent Heavy Non-Leaner, Fig. 4, shows a notable contrast from the other two strategies. Here reflection happens in all sessions and its intensity, as displayed on the y-axis to the left, is notably higher. It is worth noting that this exemplar has lean more toward reflection (0.66) more so than the cluster centroid (0.617), as kmeans groups around its centroid or mean values, but individual cases may deviate from the mean on some attribute while still fitting best within the cluster.

Fig. 4
figure 4

Student exemplar of reflection strategy frequent heavy non-leaner (cluster 3)

Full size image

Finally, Fig. 5 represents the Frequent Light Non-Leaner. In this exemplar the student commonly reflected in design sessions, however, setting aside session 1, most of their reflections came in short bursts, with markedly lower intensity than the Frequent Heavy Reflectors strategy seen above.

Fig. 5
figure 5

Student exemplar of reflection strategy frequent light non-leaner (cluster 4)

Full size image

Reflection strategies relationship to students’ tradeoff values

Table 3 describes the summary statistics for the tradeoff values of each student split between Clusters 1–4. Cluster 1 mean: 183.495 (std dev: 50.522), Cluster 2 mean: 208.913 (std dev: 41.402), Cluster 3 mean: 213.380 (std dev: 48.806), and Cluster 4 mean: 203.539 (std dev: 41.620). Cluster 2 and 3 have the highest means, Cluster 1 has the lowest mean, with Cluster 4 in the middle.

Table 3 Clusters 1–4 tradeoff value summary statistics
Full size table

Table 4 presents the regression results for tradeoff values using dummy variables to compare Cluster 1, 3, and 4 to Cluster 2 (lowest cluster based on Duration, Intensity, and Balance). Cluster 2 serves as the reference group, as described in the methods section. There was no significant change in tradeoff value when moving from the reference group to any other of the three cluster groups. Cluster 1 (p-value: 0.171), Cluster 3 (p-value: 0.820), and Cluster 4 (p-value: 0.231).

Table 4 Regression outputs for student tradeoff values
Full size table

Discussion and implications

In the first part of the discussion, we examine which of the reflection strategies may be productive for the novice designers in this study. We then compare and contrast some differences in the reflection strategies. Following this, we discuss the finding of no relationship between students' reflection strategies and their trade-off values, identify some potential explanations for why this may have happened and outline future research directions to examine the potential impact of reflection strategies.

Crismond and Adams (2012) synthesizes key design practices and distinguishes how novice and informed designers employ these practices. They note that novice reflection is often tactic or absent altogether. Specifically, they summarize that novice designers “[d]o tacit designing with little self-monitoring while working or reflecting on the process and product when done” (Crismond & Adams, 2012, p. 749) whereas informed designers’ “[p]ractice reflective thinking by keeping tabs on design strategies and thinking while working and after finished” (Crismond & Adams, 2012, p.749). From the cluster labels, it seems the Sporadic Light Modeling Leaners are likely to fall under Crismond and Adams (2012) novice description as their reflection was minimal across their design process and they only engaged in light reflection at those few points they undertook this design practice. However, for the three other clusters, the groups engaged in moderate to high levels of reflection across the design process or had heavy episodes of reflection when engaging in this practice. Therefore, we argue that the remaining three reflection strategies: Consistent Heavy Modeling Leaners, Frequent Heavy Non-Leaners, and Frequent Light Non-Leaners all represent productive beginnings (Engle & Conant, 2002; Swenson et al., 2019) of reflection.

Acknowledging that there may be multiple productive beginnings for engaging in reflection allows students to leverage this practice more naturally by drawing on their own thinking, experiences, and ways of learning. Instead of attempting to rank students’ strategies as better or worse, we examine what promise these strategies have for supporting students in navigating complex design problems. To do so, we return to Schön’s (1983) concepts of reflection-in-action and reflection-on-action. Recall that reflection-in-action happens while people are engaged in some practice or task and reflection-on-action happens after the practice or task is complete. In this context, students were primarily using reflection-in-action as they used their journals while working on their design artifacts. Reflection-on-action was not examined in this context, as students completed the project over a largely consecutive period and because as novices, they had limited prior design projects to reflection on. While reflection-in-action is a critical practice for many future professionals including engineers or those engaged in engineering practice, less has been said about what forms, or at what frequency, reflection-in-action may happen within a given project, particularly for novice designers. These strategies may lend some deeper insight into how reflection-in-action may begin to manifest. For example, the Consistent Heavy Non-Leaner, represent pausing to reflect regularly throughout the design process in moderate-to-high reflection episodes. These regular episodes may be helpful for students who are making steady progress. In contrast, Frequent Heavy Non-Leaners take many opportunities to reflect for longer dedicated periods on their design process or artifact. These longer periods may benefit some students who need more time to consider what they have accomplished and plan what they hope to undertake next. Finally, Frequent Light Non-Leaners take many opportunities to briefly reflect on their design process or artifact. These short bursts may benefit some students who need to make quick checks on their progress so far and plan for their immediate next steps.

Collectively, these strategies illustrate some of the ways reflection-in-action may be employed. For researchers, these strategies provide greater analytical depth for understanding how reflection is used and for examining how these strategies may impact early design learning and performance. For educators, these strategies can act as indicators of how students are employing reflection and provide hooks or entryways for educators to support novice designers in better integrating reflection with their design processes.

Returning to the regression results shown above, there was no statistical relationship between students utilizing one of the identified reflection strategies and their performance on their final design artifact, as measured by our trade-off value score. However, despite the absence of this relationship, the authors argue that students' reflection strategies are an important discovery. Reflection and the various ways it may be employed can facilitate deeper understanding and planning during complex problem solving across many problem contexts (Ali, 2020; Crismond & Adams, 2012; Luppicini, 2003; Mezirow, 1991; Moon, 2004). Furthermore, Crismond and Adams (2012) make an explicit call to analyze the distinct strategies used by student designers: “Keep design diaries and portfolios; compare/contrast design cases of approaches used by different groups” (pp. 749). The identification of the four reflection strategies above make progress toward this goal and provide educators and researchers with a fuller picture of how novice students may engage in reflection. Reflection also connects to literacy, writing, socio-emotional learning and related concepts (Chan & Lee, 2021; Gelmez & Bagli, 2018; Moon, 2004), therefore promoting or better supporting this skill could have multiple beneficial outcomes. Relatedly, reflection provides an opportunity for personal authenticity (Schimpf et al., 2022), that is where class-based projects enable students to make connections with their everyday experiences, interests, and beliefs.

Nevertheless, the lack of a relationship between reflection strategies and design outcomes deserves greater investigation. We offer a few preliminary possibilities for why this disconnect may exist. First, our student designers are novices in middle school, likely with very limited exposure to design. Additionally, the school district is an economically disadvantaged area, which often has limited resources in general and, more specifically for activities like engineering design that may lay beyond the typical educational goals for a district. This may have further exacerbated students' limited exposure to engineering design. It is well established that developing skills as a designer takes considerable time (Atman et al., 2007; Cross, 2011; Lawson & Dorst, 2009), so although students were able to engage in distinct reflection strategies, they may not yet be able to fully leverage these toward the creation of high performing design artifacts.

Related to the above point, another conflating factor may be that reflection and trade-off skills operate as distinct tracks of learning. In short, students may develop these skills independently and only later arrive at applying them in unison. As previously noted, reflection is a domain-general skill (Moon, 2004; Mezirow, 1991; Schön, 1983) and students may have had some experience developing these skills through other projects, such as in science or writing classes. Trade-off skills, on the other hand, are more closely associated with engineering (Dym et al., 2014; International Technology Education Association, 2000). Therefore, the difference between domain generality and domain specificity of these skills may leave them on separate learning tracks and require more time for students to draw strong connections between them.

A third and final reason may be that at this stage for novice designers, other design practices have a more immediate impact on their design outcomes. For instance, several studies have found an association between testing practices and learning or performance outcomes (Chao et al., 2017; Vieira et al., 2016) and other studies have found that after an engineering design project-based learning experience, students show a statistically significant greater recognition of the importance of testing practices (Goldstein et al., 2017). It may be that learning about design as a novice, practices like testing are more easily leveraged, and other practices, like reflection, require more time and experience to fully coordinate within the design process. Future studies should further explore these possible explanations for the lack of connection between reflection strategies and design outcomes to better understand how we can support students' development as designers and reflective thinkers.

Limitations

Aside from the null results on the regression analysis, several other limitations of this study should be noted. First, although we gave students the agency to decide when and how they would reflect during the design process, not all students engaged in reflection. Since the cluster analysis variables were based on reflection actions, seventeen students with no reflection actions had to be removed from the analysis. This outcome further suggests that simply allowing students to select their own reflection strategy may be insufficient for supporting an entire class of students – even if many students do engage in productive reflection strategies as discussed above.

An additional limitation comes from the data collection platform itself. While Energy3D can collect each time a student accesses and enters text in their journal, it cannot capture activity happening outside of the platform. For example, if students were to reflect on their design project in their head those actions would not be captured. While the teacher regularly encouraged students to use the system to record all their reflections, we cannot control for all instances where students may have reflected outside the platform. Other data collection modes also are limited in fully capturing student reflections if they are unwilling or unable to share. Some of the richness of students’ reflection strategies may therefore be lost.

Conclusion

While there is a growing interest of bringing engineering design projects into pre-college settings, it is not well understood how younger students develop design skills, practices and mindsets (Goldstein et al., 2021; Dasgupta, 2019; Wendell et al., 2017). In this study, we sought to analyze how younger students engaged in reflection strategies in the design process, as reflection is both important to design and represents a more general skill or ability that can be leveraged across a variety of problem-solving contexts. Past work that studied reflection has tended to focus the outcomes of reflection (Ali, 2020; Baaki et al., 2017; McKenna et al., 2009; Tracey & Hutchinson, 2018) or ways to support people to use reflection (Moore et al., 2016; Walther et al., 2011; Kavanagh, 2008), but not how people employ reflection naturalistically and contextually as a practice. Thus, we sought to study both what reflection strategies pre-college students use and whether these strategies impacted their final design outcomes.

In order to study this, we had 105 middle school students (ages 13–14) undertake an engineering design project to create a green energy home that produced net-zero energy over the course of the year, stayed under budget and met several aesthetic criteria. Students created their homes using the CAD platform Energy3D (Xie et al., 2018). This platform unobtrusively logs students’ actions such as designing a home and using the built-in design journal. Students were instructed to use the design journal to reflect during their design process as they felt appropriate, allowing us to capture their natural reflection strategies. We leveraged a learning analytics approach, based on prior work (Schimpf et al., 2020) to extract key features of their reflection strategies from the data log: (1) duration or number of design sessions where reflection happened, (2) intensity or average reflection frequency across sessions, and (3) balance average percent of design sessions dedicated to reflection over other tasks (i.e., home design). To identify common reflection strategies, we used kmeans cluster analysis to group students based on these three features. To assess their design outcome or performance we used the Trade-off Value proposed by Goldstein et al., (2021), which is a multiple-dimension quality score considering dimensions like energy performance, cost, and aesthetics.

Results indicated that students engaged in four distinct reflection strategies that varied in duration (sporadic to frequent), intensity (light to heavy) and lean (lean to non-lean): Sporadic Light Modeling Leaners, Consistent Heavy Modeling-Leaners, Frequent Heavy Non-Leaner, and Frequent Light Non-Leaner. However, on the second question, no association was found between using any of the aforementioned reflection strategies and design performance as measured by trade-off value scores.

Sporadic Light Modeling Leaners engaged in reflection minimally, in contrast, the three other groups engaged in moderate to high levels of reflection across the design process or had heavy episodes of reflection. Consequently, we argue these three reflection strategies represent a productive beginning of reflection that teachers can use to identify and support students to develop this practice. While no relationship was found between these strategies and students design outcomes, several intervening factors may be affecting this, including that students had minimal prior design experience, students may develop design skills independent of other skills at early stages of learning and hence there will be no or a weak relationship between them, and finally that for novices, other design practices, such as testing, may have a more immediate impact on students design outcomes. Nevertheless, reflection remains a key still for engineering design and the results reported here demonstrate some of the ways this skill may start to develop in novices as well as some directions to explore how reflection develops relative to other design skills. By uncovering some of the ways novice designers contextualize this practice to their design process, we also open the door to exploring how more experienced designers engage in reflection strategies. Lastly, this work helps demonstrate how learning analytics when deployed with sufficiently detailed and comprehensive data schemas (e.g., see Schimpf & Goldstein, 2022) can help us uncover these naturalistic or contextualized practices as well.