What’s your goal? The importance of shaping the goals of engineering tasks to focus learners on the underlying science

Abstract

Engaging in engineering tasks can help students learn science concepts. However, many engineering tasks lead students to focus more on the success of their construction than on learning science content, which can hurt students’ ability to learn and transfer scientific principles from them. Two empirical studies investigate how content-focused learning goals and contrasting cases affect how students learn and transfer science concepts from engineering activities. High school students were given an engineering challenge, which involved understanding and applying center of mass concepts. In Study 1, 86 students were divided into four conditions where both goals (content learning vs. outcome) and instructional scaffolds (contrasting cases vs. no cases) were manipulated during the engineering task. Students with both content-focused learning goals and contrasting cases were better able to transfer scientific principles to a new task. Meanwhile, regardless of condition, students who noticed the deep structure in the cases demonstrated greater learning. A second study tried to replicate the goal manipulation findings, while addressing some limitations of Study 1. In Study 2, 78 students received the same engineering task with contrasting cases, while half the students received a learning goal, and half received an outcome goal. Students who were given content-focused learning goals valued science learning resources more and were better able to transfer scientific principles to novel situations on a test. Across conditions, the more students valued resources, the more they learned, and students who noticed the deep structure transferred more. This research underscores the importance of content-focused learning goals for supporting transfer of scientific principles from engineering tasks, when students have access to adequate instructional scaffolds.

Introduction

Learning science from engineering tasks

Engineering design curricula are on the rise in schools today. Many scholars have touted the benefits of these curricula for teaching valuable engineering practices, developing broader twenty first century skills, and raising interest in engineering careers (Bottoms and Anthony 2005; Koh et al. 2015; Lammi et al. 2018). Another key goal of many engineering curricula is to enhance the learning and transfer of core science concepts (e.g., Barron et al. 1998; Kolodner et al. 2003; Silk et al. 2009; Worsley and Blikstein 2014). While learning science is not the only, nor the most important outcome for engineering tasks, several engineering curricula have been developed to support students’ ability to learn science concepts (Brophy et al. 2008; Fortus et al. 2004; Kolodner et al. 2003).

However, just having students complete engineering design activities is not sufficient to teach students science. For example, work by Petrosino (1998) illustrated that simply having students make and launch rockets did little in the way of teaching students science concepts. Subsequently, engineering tasks and curricula have been developed to incorporate an assortment of instructional scaffolds meant to aid learning and transfer of science content knowledge. These scaffolds may include multiple challenges that allow students to abstract scientific principles that are common across different tasks (Berland et al. 2013b; Fortus et al. 2004; Silk et al. 2009), specific opportunities for reflection on the application of science concepts to engineering designs (Berland et al. 2013b; Fortus et al. 2004; Gero et al. 2013; Schunn 2011; Silk et al. 2009; Svarovsky and Shaffer 2007), science concept-focused brainstorming (Worsley and Blikstein 2014), formal instruction on scientific principles in the form of readings, lectures, class discussion, and individual tutoring (e.g., Fortus et al. 2004; Kanter 2010; Kolodner et al. 2003; Silk et al. 2009), and activities that directly address students’ common scientific misconceptions (Schnittka and Bell 2011). Curricular scaffolds like these are used to direct students to think about scientific principles in relation to their work on the engineering task.

Yet, despite all this good intention, students may fail to effectively learn and transfer science content from engineering tasks, largely because they (or their teachers) are too focused on completing the engineering task and not enough on understanding the science underlying the task. For instance, rather than tear themselves away from an engaging engineering task to consult science learning resources, students often rely on trial-and-error to guide their construction process (e.g., Ahmed et al. 2003; Berland et al. 2013a, b; Kolodner et al. 2003). Additionally, non-expert teachers may allow students too much time for messing about with task materials and too little time for activities that help students learn the core scientific principles of the task (Gertzman and Kolodner 1996; Hmelo et al. 2000). As a result, engineering tasks often turn into “arts and crafts” activities (Holbrook and Kolodner 2000) where students focus solely on making their construction, and fail to deeply reflect on associated science concepts (e.g., Gertzman and Kolodner 1996; Hmelo et al. 2000; Kanter 2010; Roth et al. 2001; Silk et al. 2009; Vattam and Kolodner 2008). Additionally, having students focus solely on engineering outcomes can hurt a student’s ability to learn science concepts from the engineering activity (Worsley and Blikstein 2014).

The question then becomes how to encourage students to engage with learning resources and instructional scaffolds so that they can effectively learn science content from engineering activities. We propose that content-focused learning goals can motivate students to attend more to the scientific content of an engineering task. This is important, because helping students notice the deep scientific structures of a task may be crucial for helping them learn, and subsequently transfer, science content knowledge to novel contexts.

Content-focused learning goals versus outcome goals

One way to focus student attention on core scientific principles during engineering activities is by setting content-specific goals. This idea is not new. In 1998, Barron and colleagues noted that setting “learning appropriate goals” is an essential component of any project-based learning curriculum that aims to help students develop science knowledge. The authors note that during engineering tasks, students typically focus on the outcome of the task, such as whether or not their construction is successful. So for the purpose of teaching science content, these activities need content-focused learning goals, which focus student attention on learning instead of constructing.

While the term “learning goals” has been used in the literature to define many different constructs (e.g., Dweck 1986; Elliot and Harackiewicz 1994; Latham and Brown 2006; Gardner et al. 2016; Rothkopf and Billington 1979), this paper is specifically concerned with content-focused learning goals versus outcome goals. We define these terms in reference to how the terms learning goals versus outcomes goals are used in the goal setting literature (e.g., Locke and Latham 1990, 2002; Locke et al. 1981). Content-focused learning goals are defined by their ability to focus students on acquiring information, ideas, or strategies to accomplish a task. In our studies, the content-focused learning goal is meant to direct students to think about the deep scientific structures that underlie the task. Here, deep structure refers to the core scientific principles of the task, which are relevant to producing a successful engineering product. We contrast these content-focused learning goals with outcome goals, which focus students on their performance on the task itself.

We propose that content-focused learning goals may improve learning by leading students to engage with the task in more effective ways. One study on a science inquiry task by Schauble et al. (1991) found that students with an engineering goal focused more on task outcomes. Meanwhile, students with a science goal gained a better understanding of the structural relationships between variables of the problem, which led them to make more appropriate conclusions from the task. However, there is very little research looking at the use of content-focused learning goals designed to teach science content. In particular, we know of only one study that has explored the impact of content-focused learning goals on transfer outcomes (Schunk and Swartz 1993). Thus, there is a serious need for more research on the effects of content-focused learning goals on science learning processes, and especially transfer outcomes.

That being said, the distinction between content-focused learning goals and outcome goals has been examined in other educational domains. For example, a study by Rothkopf and Billington (1979) found that high school and college students who were given content-focused learning goals read a text faster, and attended more to the relevant portions of the text, making their overall processing of the learning resource more efficient. Other work also suggests that content-focused learning goals improve student attention to learning resources (Ames and Archer 1988; Locke and Bryan 1969). Work has also shown that a learning goal, which focuses students on the core scientific principles of a game, improves learning more than an outcome goal, which focuses students on performing well in the game (Miller et al. 1999). Work in the domain of writing has shown that students given a learning goal that asked them to focus on the deep structure of a task (writing strategy) both performed and transferred better than students who were given an outcome goal (producing written paragraphs; Schunk and Swartz 1993).

Engineering tasks are particularly at risk for focusing students on outcome goals, since most engineering tasks are framed as outcome goals. For example, a common engineering task goal might be to build the highest structure possible that will withstand an earthquake test (Apedoe and Schunn 2012), or to construct a working water purification device (Riskowski et al. 2009). These outcome goals may lead students to believe that producing the desired outcome is the sole purpose of the task, rather than a means to learn science content. Content-focused learning goals may therefore be a key means to ensure science learning from engineering tasks. The present research investigates how content-focused learning versus outcome goals might affect students’ ability to learn and transfer science content knowledge from engineering activities.

Transfer and noticing

Transfer is another important objective of engineering tasks that aim to teach students science content (Kolodner et al. 2003). Ideally students can not only learn associated science content from an engineering task, but also apply that science content appropriately in novel tasks and contexts. While the transfer of science knowledge from engineering tasks has not been extensively studied (but see Malkiewich and Chase 2019; Chase et al. 2019; Kolodner et al. 2003), transfer has been widely studied in the context of schooling more generally (Bransford et al. 1999; National Research Council 2012). While many argue that transfer is a major goal of K-12 education, it has proved an elusive goal, with some claiming that empirical demonstrations of transfer are rare (Detterman 1993; Lave 1988). This has prompted some scholars to call for instructional design that explicitly “teach[es] for transfer” (Perkins and Salomon 1988).

One way to support transfer is to help students notice the deep structure of the task. By “notice” we mean a student’s ability to both perceive and use specific information when a host of information is competing for the student’s attention (see Malkiewich and Chase 2019; Lobato et al. 2012). Some ecological psychologists claim that learning to perceive deep structures can improve performance in other situations that have the same deep structure (Gibson 1988; Pick 1992). Some transfer scholars argue that successful transfer is dependent on the ability to notice a deep structure as invariant across contexts, and to know how to act in accordance with the presence of that deep structure (Chase et al. 2019; Day and Goldstone 2012; Greeno et al. 1993; Lobato et al. 2012). There have been several empirical demonstrations linking noticing with transfer (Day and Goldstone 2012; Gick and Holyoak 1983; Schwartz et al. 2011; Shemwell et al. 2015). However, there is little work connecting deep structure noticing and transfer in the context of engineering activities (but see Chase et al. 2019; Malkiewich and Chase 2019).

We hypothesize that noticing the deep structure (in this case the scientific principles of the task) helps students to transfer science knowledge from engineering tasks, and that content-focused learning goals may support this noticing process. It stands to reason that students who adopt a content-focused learning goal will have a better chance of noticing the deeper scientific structure embedded in the engineering task than those who are solely focused on producing a successful engineering product. Work already suggests that content-focused learning goals improve the quality of student exploration of the problem space during inquiry science tasks (Schauble et al. 1991). However, we do not know of any work that has specifically measured the effect of content-focused learning goals on students’ ability to notice the deep scientific structure of an engineering task.

One way to help learners notice the deep structure of a task is to use contrasting cases. Contrasting cases are examples that differ on key features, to make certain variables or relationships more salient (Bransford and Schwartz 1999; Gibson and Gibson 1955). For example, tasting glasses of wine side-by-side would make it easier to notice differences in each wine’s flavor profile. Contrasting cases have been shown to aid students’ ability to notice the deep structure of a problem, which in turn can improve transfer of science and math knowledge out of that learning context (Bransford et al. 1989; Chase et al. 2017; Roll et al. 2011; Schwartz et al. 2011; Shemwell et al. 2015). However, little empirical work has examined the effect of contrasting cases on transfer from engineering activities (but see Gentner et al. 2016; Silk and Schunn 2008).

Thus, learning goals and contrasting cases may interact to support student transfer. For one, goals affect how much students pay attention to learning resources (e.g., Ames and Archer 1988; Locke and Bryan 1969; Rothkopf and Billington 1979) and contrasting cases affect deep structure noticing. Without content-focused learning goals, students may not pay attention to cases, relying instead on other problem solving strategies like copying or trial-and-error. Without contrasting cases, even well-intentioned students may not be able to notice the deep scientific structure of a learning task. Work has shown that novices struggle to determine which features of a scientific problem are relevant (Chi et al. 1981). Furthermore, cases typically require deep processing, and content-focused learning goals may encourage students to engage in that deep processing. Therefore, contrasting cases may be most effective as instructional aids when paired with a content-focused learning goal.

Goals and learning resources

Another critical part of the science learning (and possibly transfer) process is how students value, access, and use available learning resources (Aleven et al. 2016; Schwartz and Arena 2013). Learning the science that underlies engineering tasks cannot be done in the engineering task alone. During engineering tasks, students often have access to several learning resources, such as textbooks, other students, lab notebooks, or help from a teacher. Unfortunately, it is often hard to tear students away from engaging hands-on engineering tasks to interact with more traditional learning resources such as readings. For instance, instead of accessing relevant science learning resources, students may prefer to use trial-and-error techniques to work through engineering tasks (e.g., Ahmed et al. 2003; Berland et al. 2013a, b; Kolodner et al. 2003). Nonetheless, it is important for students to consult learning resources outside the engineering task, to deepen their understanding of the relevant science content in the context of the engineering task.

Beyond engineering education, several scholars consider whether, when, and how students use available learning resources as both (1) an important outcome in its own right—students need to learn how to learn when given a choice in resources (Aleven et al. 2016; Schwartz and Arena 2013), and (2) a critical factor that influences learning (Aleven et al. 2003, 2016; Renkl 2002; Shute and Gluck 1996; Wood 2001) and sometimes transfer (Renkl 2002). Unfortunately, much work, particularly in interactive learning environments, shows that students often ignore or rarely use learning resources, when given the choice to access them freely (Lee 2017; Mandl et al. 2000).

Likewise, research suggests that learning goals can lead people to perceive greater value in seeking help from learning resources, report using them more frequently, pay closer attention to them, and use them more adaptively than those who adopt outcome goals (Ames and Archer 1988; Arbreton 1998; Locke and Bryan 1969; Rothkopf and Billington 1979; Ryan and Pintrich 1997). In this research we take the first step towards understanding how content-focused learning goals might affect students’ attitudes towards learning resources in the context of an engineering task by measuring how much students value those resources. We also explore how students’ valuing of learning resources relates to learning and transfer outcomes.

The present research

This work investigates how content-focused learning goals and contrasting cases interact to affect how students learn and transfer science concepts from an engineering task. In Study 1, students were given an engineering task to build a cantilever out of Legos. This task involved center of mass concepts. Students were given either a content-focused learning goal or an outcome goal, and half the students in each goal condition were given contrasting cases that highlighted the deep structure of the problem (center of mass concepts). After Study 1 concluded, a second study was designed to see whether initial findings could be replicated while addressing some limitations of the first study. Study 2 also focused more deeply on goals, such that only goals were manipulated across conditions. The second study also examined how students valued resources as a potential mechanism of science learning and transfer.

The research was driven by three main research questions: (1) How do goals and contrasting cases affect student performance on the engineering task as well as subsequent learning and transfer of science content? (2) How do goals and contrasting cases affect learning processes, such as deep structure noticing and valuing instructional resources? (3) How do learning processes, such as noticing, valuing resources, and task performance relate to learning and transfer?

For learning and performance, we hypothesized two main effects, such that both content-focused learning goals and contrasting cases would improve student performance and learning. We predicted that content-focused learning goals would be needed to notice the deep structure in the cases. In turn we predicted that deep structure noticing would improve transfer of science content knowledge out of the engineering task. For these reasons, we also predicted an interaction between goal and contrasting case use, such that students would need both a learning goal and contrasting cases in order to transfer.

Study 1 method

In Study 1, all participants were given an engineering challenge to build a Lego cantilever across three iterations. Half the participants were given a content-focused learning goal: to come up with a rule to determine the location of a cantilever’s center of mass. Half the participants were given an outcome goal: to build the most effective cantilever. In addition, half of all participants reflected on a set of contrasting cases, while half did not. This yielded four conditions: learning goal with cases, learning goal without cases, outcome goal with cases, and outcome goal without cases. Measures included performance on the engineering task, a learning subtest, a transfer task, and a transfer subtest. For the contrasting cases conditions, we also assessed whether students noticed the deep structure—the concept of center of mass—that was embedded in the cases.

Participants

Participants were 86 11th grade students at a racially diverse, urban public high school in the Northeastern United States. The school population was 43% White, 16% Black, 29% Hispanic, 9% Asian and 3% other, with 58% of the student body receiving free or reduced-price lunch. The school ranked in the 24th percentile of high schools on state test scores. Students from several advanced science classes at the school opted into the study. Students from across these classes were pulled from their typical science class to participate in the study during one of seven study periods. Within each assigned study period (which from here forward we will simply refer to as “period”), students were randomly assigned to a learning or outcome goal, and cases or no cases. Ultimately this created four conditions within each period: learning goal with cases (n = 19), outcome goal with cases (n = 23), learning goal no cases (n = 24), and outcome goal no cases (n = 20).

Engineering task

During the engineering task, students attempted to build a freestanding cantilever that could hang 10.5″ off the edge of a table using Legos (Fig. 1). The activity involved learning and applying center of mass, a core concept in mechanics education, and a component of the AP Physics curriculum (Bundy 2013). This task is similar to the challenges students receive in many engineering curricula. For example, students are often asked to build a structure that would benefit from applying scientific principles to improve the structure’s design (e.g., Apedoe and Schunn 2012; Kolodner et al. 2003; Worsley and Blikstein 2014).

Fig. 1
figure1

Example student cantilevers from the engineering task

Center of mass is defined as the location of an object’s point mass. It is calculated as a weighted average. Each part of the object can be considered as an individual mass which is “weighted” differently depending on its distance from some discrete reference point. Parts of an object that are heavier or farther away “pull” the center of mass closer to them. Below is the equation for calculating an object’s center of mass:

$$X_{{{\text{center}}\;{\text{of}}\;{\text{mass}}}} = \frac{{m_{1} x_{1} + m_{2} x_{2} + \ldots + m_{n} x_{n} }}{{m_{1} + m_{2} + \ldots + m_{n} }} = \frac{{\sum m_{i} x_{i} }}{{\sum m_{i} }}$$

For this challenge, each Lego acts as a mass, and the distance of each Lego from the end of the structure (xi), along with its weight (mi), affect the location of the center of mass (Xcenter of mass) of the cantilever. Since a structure can balance on its center of mass, to complete the challenge, a participant’s structure has to optimize the placement of each Lego. By distributing the large Legos farther back onto the table, the center of mass of the structure moves farther back in the cantilever, allowing the structure to stick farther off the table.

Manipulation

In the contrasting cases conditions, students reflected on two sets of cases (see Fig. 2). Contrasting cases were depicted on worksheets showing front and side images of all cases. The bases of each case were the same. On top of each case was a colored weight that was made out of the same Legos, each with a different shape. Cases were designed to highlight the effect of mass and distance on center of mass. For instance, by contrasting cases A and C (Fig. 2), students might notice that the location of the weight mattered, because the case with the weight farther back sticks off the table more. Cases also addressed common student misconceptions, such as whether the height or width of the structure was important. For example, by comparing cases E and F (Fig. 2), students could notice that these two structures stick off the table the same amount, even though one is taller and one is wider, because they had the same amount of weight in the same part of the structure.

Fig. 2
figure2

Contrasting cases for reflection 1 (A–D) and reflection 2 (E–H), shown from top and side. Dots indicate cantilevers’ centers of mass

During two reflections, students in the cases conditions answered a series of questions to draw specific comparisons between cases which should highlight the features of mass and distance. For example, students were asked to think about the similarities and differences between the cases, and then answer, “What do you notice about the structures that stick out the most?” and “what do you notice about the structures that stick out the least?” This type of comparative prompting can help students process contrasting cases more effectively (Kurtz et al. 2001). At the end of the worksheet, students were asked to describe what factors affected the location of the center of mass on a cantilever, and why. The students who did not receive cases spent that time reflecting on their building process. For example, students without contrasting cases were asked, “What difficulties might you come across during this challenge?” and “What is difficult about working with Legos?” These process-focused questions were inspired by curricula that emphasize the design process during these types of activities (e.g., Lachapelle and Cunningham 2007).

The second manipulation was the goal students were given during the engineering task. Outcome goal students were instructed to “Build a structure that can stick 10.5″ off of a table”. Learning goal students were told to “Figure out a rule that indicates where a structure’s balance point is. Make sure your rule explains why some structures stick out more than others.” Learning goal students were told that an adequate rule would help them build a structure that could stick 10.5″ off the table, so the engineering task was framed as a way to test the quality of their rule.

Procedure and materials

Students participated in the study for one period a day, for five school days (Fig. 3). On the first day, students took a pretest and received their task goal for the engineering activity. During days two and three, students worked on the engineering task individually, over three “build periods.” This was meant to emulate a common aspect of engineering tasks, where students are given multiple chances to iterate upon their designs (e.g., Apedoe and Schunn 2012; Fortus et al. 2004; Klahr et al. 2007; Kolodner et al. 2003). Between builds 2 and 3 students received a short lecture about center of mass. The lecture was given by the principle investigator, and discussed how center of mass is affected by both mass and distance in a multiplicative way, such that if the mass of one end of an object increases, or the distance of a mass within an object increases the center of mass moves. This emulated common engineering instruction where students have time to stop, think about what they are building, and get some direct instruction on the core principle of the task to improve their designs (e.g., Kolodner et al. 2003). Students were not told exactly how to apply the lecture content to the task, because that would have cut off student exploration during build 3, as well as students’ ability to make that connection on their own.

Fig. 3
figure3

Study 1 procedure

Before and after build 1, students did a reflection. In the contrasting cases condition, students reflected on the first set of Lego cases (Fig. 2). The no cases condition did a process reflection described above. All students reflected with a partner to promote deep processing, then filled out their own individual reflection sheet. Students did a second reflection, with the same partner after build 1. During the second reflection, students in the contrasting cases conditions looked at another set of cases (Fig. 2). Students in the no cases conditions did a second process reflection.

On the fourth day, students received a full “tell” where the principal investigator provided the optimal solution to the cantilever building task. This idea comes from the preparation for future learning literature (Bransford and Schwartz 1999), and the productive failure literature (for a review see Loibl et al. 2017) where students first struggle with an ill-defined problem then get formal instruction, called a “tell”. The tell in this study was meant to fill in knowledge gaps for students who were unable to effectively connect the center of mass lecture content to the engineering task. It elaborated on the concepts of the earlier lecture by explaining the equation for center of mass in relation to the shape of the optimal cantilever. It also addressed several common student misconceptions, such as why making the structure taller or wider did not actually “add mass.” Finally, the tell addressed how to find the center of mass of a two-dimensional object, and why the center of mass needs to be over the base of an object in order for that object to balance.

On the fifth day, students did a transfer task to evaluate how their understanding of center of mass affected performance on a different engineering activity. Students then took posttests that had learning and transfer components.

Measures

Task performance

Task performance was measured by experimenter records of how far off the table each student’s structure hung at the end of each build period (measured in inches).

Tests

Students took pencil-and-paper pre and posttests, which contained learning and transfer subtests. All questions were scored on a 3 or 4-point scale, ranging from incorrect, to partially correct, to correct. For examples of these coding schemes see Tables 1 and 2. Two different researchers blind-coded 20% of the data. For each question that required inference (e.g., not multiple choice), an inter-rater reliability of κ > .70 was achieved, and one master coder scored the rest of the data. An average item score was calculated for learning and transfer subtests, after transforming each question to a 0–1 scale, so that each subtest item was weighted equally.

Table 1 Coding manual and example student responses to the second learning subtest question
Learning subtest

The learning subtests evaluated students’ declarative knowledge about center of mass using two questions that were identical at pre and posttest. The questions asked “What is center of mass? Give a definition” and “Explain what features of a structure affect the location of its center of mass. Describe precisely HOW these features impact the center of mass”. These two questions evaluated rote understanding of the lecture content. For the coding manual of the second learning subtest question, see Table 1.

Transfer subtest

Transfer questions asked students to apply knowledge about center of mass to new problems in a variety of contexts that differed from the engineering design activity in both functional context and modality (Barnett and Ceci 2002). For example, in the transfer question shown in Fig. 4 (coding manual Table 2), students have to take the center of mass equation that they learned during the tell, and apply it to an irregularly shaped two-dimensional object. A worked problem is provided to students at the beginning of the transfer subtest. Students need to figure out how that problem applies to the target transfer problem given later in the test. This is a novel situation because (1) students did not practice applying the center of mass equation during instruction, and (2) to solve this problem correctly, students have to learn a new procedure for calculating the center of mass of an object in two dimensions, while both the cantilever task and the tell focused only on center of mass on the horizontal dimension. This question was meant to emulate preparation for future learning type transfer questions used in other studies of transfer (e.g., Schwartz and Martin 2004).

Fig. 4
figure4

Students see the worked example above on the front page of their test. Later in the test, they answer question 2

Table 2 Coding manual and example student responses to the transfer subtest question shown in Fig. 4

The pre transfer subtest contained four questions (three short answer and one multiple choice item). The post transfer subtest contained four questions that were isomorphic to those on the pretest plus seven new items, for a total of 11 questions (seven short answer and four multiple choice). Isomorphic posttest questions were structurally the same as those on the posttest, and were worded in the same way, but the stimuli were modified to make the posttest questions distinct.

On the transfer posttest, two questions asked students to think about how distributing the weight of a system or object differently would impact the center of mass. The other two questions asked students to explain why a novel object’s center of mass was in a particular location. The remainder of the questions on the transfer posttest continued to test students’ ability to apply center of mass concepts in contextualized problems. For more example transfer posttest questions, see Appendix 1. Reliability for the transfer measure was low at pre (α = .40) because students had little knowledge of center of mass so they tended to answer questions randomly, leading to low inter-item correlations. Reliability on the post transfer subtest was higher (α = .67). Here we considered a cutoff less than α = .70 acceptable because we were not looking to measure student performance on a single learning construct. Rather, a student’s transfer score was meant to measure his or her ability to apply center of mass knowledge to a large swath of different problems, set in different contexts, intended to test different ways in which a student could transfer what was learned. In addition, Cronbach’s alpha is typically low when there are few items on a test (Tavakol and Dennik 2011).

Transfer task

The transfer task asked students to make a paper bird balance on a straw by adding paper clips to it (Fig. 5). To be successful on this task, students had to move the center of mass on the bird to the left and down. This was considered a transfer task because students had to use center of mass concepts in a new task with different materials. Furthermore, to be successful, students had to think about center of mass in a new dimension (up and down, along the y-axis) from the learning task (which only involved the x-axis). To measure student success, birds were divided into four quadrants and experimenters counted how many paper clips students put on the bottom half of their bird. Putting weight on the left half of the bird was not considered transfer, because it involved the same horizontal weight placement principles students learned in the engineering task. In contrast, placing weight low measured how well students took new information presented in the tell and applied it to a novel context.

Fig. 5
figure5

Transfer task. Students were instructed to make the bird balance on a straw by adding paper clips to it. The dot labeled “COM given” indicates where the center of mass of the bird was when the bird was given to students without any paper clips on it. The dot labeled “COM target” indicates where the center of mass needed to be moved to in order for the bird to balance on a straw. Students did not see these COM labels or quadrants. Adding paper clips to quadrants 1 and 3 moves the center of mass down, closer to the COM target

Deep structure noticing

Whether or not students noticed the deep structure was only measured for students in the contrasting cases conditions. When looking at the cases, students answered a series of questions, such as “What do you notice about the structures that stick out the most?” Reflections were dichotomously coded for whether students noticed the deep structure or not. Students were considered to have noticed the deep structure if they could verbalize the multiplicative relationship between mass and distance. For example, the following rule received a score of 1: “The balance point is determined by the positioning of the Lego’s weight. The heavier the back side, the larger the extension”. Two different researchers blind-coded the same 20% of the data independently. An inter-rater reliability of κ = .79 was achieved. Coders then split up the remaining data. While responses from both reflections were originally coded, the majority of students wrote the same rule for both reflections, or improved their rule after reflection 2. Students who wrote worse rules during reflection 2 demonstrated a distinct change, such that they no longer deemed the deep structure of the task to be important. For these reasons, only responses from the second reflection are used in the analysis below, in an attempt to capture whether or not students noticed the deep structure after having reflected on both sets of cases.

Study 1 results

Most analyses used an ANOVA model. Count data were originally analyzed with a Poisson regression. However, an ANOVA is reported because results from both models showed the same effects, but the ANOVA is easier for readers to interpret.

Exploratory analyses indicated that there were period effects for posttest scores, but period effects did not exist for other outcomes. A period variable was therefore added as a random factor to both learning and transfer models that evaluated posttest performance. There was no effect of gender on any outcome, so it was not added to any analyses.

Given that the contrasting case reflections were done in dyads, outcome measures were not truly independent because students learned together. This issue has been addressed in past work where students learn in dyads but are measured individually on outcomes (Mercier 2017). To test for independence between measures within dyads, intra-class correlation was calculated for dyads on all outcome measures (Table 3). The cutoff for significance was a two-tailed p-value of < .20 (Kenny et al. 2006). The only outcome measures with a significant ICC were learning and transfer subtest scores.

Table 3 ICC between dyads on various outcome measures

To address pair effects for posttest measures, a hierarchical linear mixed-effect model was run with pair as a random effect. Level 1 models individual, fixed effects like task goal, contrasting case use, pretest scores, and period on individual students’ posttest scores. Level 2 models the effects of student pairs on the intercept for the level 1 model. Student pairs were not nested within periods because there were only seven periods, which students were assigned to using convenience sampling, which prevented us from reasonably treating period as a random or level 2 effect (University of Bristol, n.d.).

$${\text{Level }}\;1: Y_{ij} = \beta_{0j} + \beta_{1} GOAL_{ij} + \beta_{2} CASES_{ij} + \beta_{3} PRETEST_{ij} + \beta_{3} PERIOD_{ij} + e_{ij}$$
$${\text{Level}} 2: \beta_{0j} = \gamma_{00} + u_{0j}$$

Task performance

Although students’ cantilevers improved over time, there were no significant effects of either condition on performance (Table 4). A two-way repeated measures ANOVA with goal (learning vs. outcome) and type of reflection (cases vs. no cases) as between-subjects factors, and time as a within-subjects factor, indicated an interaction between goal and reflection type on task performance F(1,82) = 7.40, p = .01. Planned comparisons were conducted using Bonferroni adjusted alpha levels of .025 per test. There were no significant effects of reflection type between conditions with this correction. There was also a significant interaction of case condition over time, F(2,81) = 3.71, p = .03. Planned comparisons were conducted to see the effect of cases on performance during each build period, using Bonferroni adjusted alpha levels of .0167 per test. However, there were no significant differences between cases conditions on build periods at this level. Finally, there was a main effect for time, F(2,81) = 62.95, p < .001, and this pattern was linear, F(1,82) = 126.71, p = .03, indicating that all student structures improved over time. There were no other main effects or interactions p’s > .26.

Table 4 Means (with SD) of student’s structure length (in inches) over time

Posttest outcomes

Learning subtest

Students’ goal did not affect learning, when controlling for period effects and prior knowledge. To evaluate learning, a hierarchical linear model was run with learning pretest score, period, task goal, cases, and a task goal by cases interaction as fixed effects at level one. Pair was modeled as a random effect at level two. There were no significant main effects or interactions, p’s > .10 (Fig. 6).

Fig. 6
figure6

Student posttest scores on the learning subtest. Error bars are ± 1 SE

Transfer subtest

Students’ condition also did not affect transfer (on the posttest), when controlling for period effects and prior knowledge (Fig. 7). To evaluate transfer subtest performance at post, a hierarchical linear model was run with transfer pretest score, period, task goal, cases, and a task goal by cases interaction as fixed effects at level one. Pair was modeled as a random effect at level two. There was a main effect for pretest, t(72.48) = 3.75, p < .01, but there were no other significant effects, p’s > .34.

Fig. 7
figure7

Student posttest scores on the transfer subtest. Error bars are ± 1 SE

Transfer task

Performance on the transfer task indicated that students who had both contrasting cases and a content-focused learning goal transferred more. A two-way ANOVA with goal, cases, and transfer pretest score as between-subjects factors showed a significant interaction between students’ task goal and whether or not they received cases F(1,81) = 8.11, p = .01. Planned comparisons with a Bonferroni alpha correction set at .025 confirmed that of the students who were assigned a content-focused learning goal, those who were given cases performed better than those who were not given cases, F(1,81) = 9.27, p = .03, d = 0.79. Similarly, of students who were given cases, those who had a content-focused learning goal performed better than those who had an outcome goal, F(1,81) = 6.89, p = .01, d = 0.77. Figure 8 shows the pattern of results. All conditions performed similarly on this task, except the content-focused learning goal with cases condition, which outperformed the content-focused learning goal without cases and the outcome goal with cases conditions. There were no main effects of either goal or case manipulations, p’s > .15.

Fig. 8
figure8

Number of paperclips students put on the bottom half of their bird. Brackets with asterisk indicate condition differences at p < .05. Error bars are ± 1 SE

Noticing the deep structure

Within contrasting case reflection groups, students’ assigned goal did not seem to affect whether or not students noticed the deep structure of the problem. Across both goal conditions, students typically failed to notice the deep structure (Table 5). A Chi Square test showed that deep structure noticing did not significantly differ across the two task goal conditions, X2(1, N = 42) = 0.02, p = .89.

Table 5 Number of students who did and did not notice the deep structure by goal condition

Mechanisms of learning and transfer

We explored the relationship between learning process measures and learning and transfer outcomes. For each learning and transfer outcome, a separate hierarchical linear model was run with fixed effects for building performance, noticing the deep structure, period, and relevant pretest scores at level 1, and a random effect for student pairs at level 2 (Table 6). Note that because only the cases conditions had noticing measures, all models shown below depict only the students who were given contrasting cases (n = 48).

Table 6 Hierarchical linear models of learning and transfer outcomes

These models were first run with dummy variables for goal condition and interactions of condition with noticing and build length. However, there were no significant main effects of condition and no interactions, so these variables were excluded from the models.

There was a significant main effect of deep structure noticing on learning, and there was a trend towards deep structure noticing predicting transfer subtest scores. Regardless of condition, students who noticed the deep structure during the cases comparison did better on the learning subtest, controlling for prior knowledge. Additionally, students who built longer structures did better on the transfer subtest, controlling for prior knowledge. The effects of build length on all outcomes are the same when done on the full dataset, including the no cases participants (p = .18 for learning, p = .01 for transfer subtest, p = .09 for transfer task).

Students who noticed the deep structure scored 35% higher on the learning subtest (M = 0.71, SD = 0.27) than students who did not notice the deep structure (M = 0.36, SD = 0.25). Similarly, students who noticed the deep structure scored 19% higher on the transfer subtest (M = 0.59, SD = 0.21) than students who did not notice the deep structure (M = 0.40, SD = 0.19), although this effect was not statistically significant at the p = .05 level.

Study 1 discussion

Overall, results showed that students needed both content-focused learning goals and contrasting cases to have improved transfer task performance. However, neither learning goals nor cases influenced task performance, learning, or transfer subtest outcomes. Results also showed that for students given contrasting cases, deep structure noticing significantly predicted learning, even when controlling for task performance and prior knowledge. Deep structure noticing did not significantly predict transfer task performance. We address our initial hypotheses below.

First, we confirmed the hypothesis that both a content-focused learning goal and contrasting cases were needed for students to transfer center of mass knowledge to a different engineering task. Students who had both a content-focused learning goal and cases did better on the transfer task, but there were no main effects for either factor alone, suggesting that both a content-focused learning goal and contrasting cases are needed for transfer. However, it is unclear exactly how these manipulations affected transfer, given that goals and contrasting cases did not affect task performance, learning, transfer subtest performance, or deep structure noticing during the contrasting cases reflection. It may be that students who were given a content-focused learning goal and contrasting cases were able to build some intuitive knowledge of center of mass which helped them perform on the transfer engineering task, but did not affect their formal understanding of center of mass. This could have also been a preparation for future learning effect (Schwartz and Bransford 1998), where the content-focused learning goal and contrasting cases together helped students more effectively learn from the tell, which taught students that when balancing an object, the center of mass has to be kept low. Maybe students who had a content-focused learning goal paid more attention to the tell to fulfill their learning goal, while the contrasting cases experience helped students develop intuitive knowledge that aided learning from the tell.

While deep structure noticing did not differ between conditions, it was surprisingly associated with higher learning posttest scores, which was not predicted. Note that deep structure noticing was only measured for the students given a contrasting cases reflection, and few students who received contrasting cases noticed the deep structure of the problem. Therefore, the small sample made it difficult to effectively assess the effect of deep structure noticing.

Limitations

This study suffered from a few limitations. First, the sample size for each condition was relatively small (about 20 students). This limited the power available to detect learning or transfer differences between conditions. Before conducting the study, a power analysis indicated that 53 students per condition were needed to see an effect. However due to poor recruitment at the school site, our final sample size fell far below the level needed according to our power analysis. This small sample size especially hurt our ability to effectively detect the relationship between deep structure noticing and other measures, given that noticing behavior could only be measured for students who received contrasting cases. As such, nine students in our sample noticed the deep structure, making it difficult to test our hypothesis of the relationship between deep structure noticing and transfer. Although this relationship was not significant in this study, the effect was in the hypothesized direction, and the significance was not large (p < .10). Some scholars suggest that p-values around p = .05 are still worthy of investigation since getting a statistically significant result across multiple studies is rare, even in the most perfect conditions (Amrhein et al. 2017). For this reason, we continued to measure and analyze this effect in an improved Study 2, which recruited a larger sample size and gave everyone contrasting cases, to address some of the power and study design concerns.

Furthermore, the pre and posttests were not counterbalanced, which could have made it difficult to measure how much students learned, or transferred from the study, especially if the tests were not of equal difficulty. We also recognize that these tests were not very reliable (α < .70), in part because our single test was assessing a number of different transfer skills. This measurement decision could have hurt our ability to find effects that would have affected only one specific type of transfer. Additionally, some items on the transfer subtest seemed to be worded poorly, judging by students’ responses. Test questions were improved for Study 2.

Field observations also indicated that although students understood their unique, assigned task goal at the beginning of the task, during building, students from both goal conditions took on an outcome goal. So, the task goal manipulation did not seem to be as strong as was originally intended.

Study 2 method

Study 1 indicated that the presence of a content-focused learning goal and contrasting cases helped students transfer center of mass knowledge to a different engineering task. After the conclusion of Study 1, a second study was designed in an attempt to replicate some of the learning goal effects with a different student population.

Study 2 focused only on the effects of the goal manipulation. All students were given contrasting cases, and only task goals were manipulated across conditions. This increased the sample size of each condition, which made it easier to detect how task goals affect transfer. Additionally, this should provide a larger sample of students who notice the deep structure, enabling us to better test the hypothesis that noticing the deep structure is important for transfer. Study 2 also investigated a new question—whether resource valuing might explain some of the learning and transfer effects. We hypothesized that content-focused learning goals would improve the perceived value of resources, which in turn should improve learning and transfer.

Study 2 also improved some of the processes and measures of Study 1. The goal manipulation was strengthened, the test measures were enhanced, and the study procedure was simplified. We hypothesized that the strengthened goal manipulation would cause students given a content-focused learning goal to learn and transfer more than students given an outcome goal.

Participants

For Study 2, a new set of 78 students (learning goal n = 39, outcome goal n = 39) participated in the full study. Students were 10th, 11th, and 12th graders in the accelerated science track, at a suburban public high school in the Northeastern United States. The school population was 11% White, 2% Black, 84% Hispanic, and 3% Asian, with 72% of the student body receiving free or reduced-price lunch. The school ranks in the 25th percentile in state test scores. Students participated in the study during their usual science class, which occurred in one of five periods. Students were randomly assigned a task goal within each period.

Procedure

Students were given the same goals, engineering task, and contrasting cases reflection from Study 1. However, to strengthen the goal manipulation, students were given notes sheets to fill out during each build period. These sheets prompted students to track how close they were to reaching their goal. In the outcome goal condition, students were asked to write various ideas for structures, and how far off the table each of those structures stuck. In the learning goal condition, students were asked to list the rules they came up with during the task, and then were asked, “If you apply your rule, does it tell you how to build a structure that can stick 10.5” off the table?” Students were also assigned to work in dyads while building. The intention was that students would hold their partner accountable to work towards their assigned task goal. Class observations indicated that students did remind their partner of their goal throughout the activity.

Unlike Study 1, all students reflected on contrasting cases. Additionally, students had physical copies of the cases in their classroom, which they could look at in real life, in addition to the front and side images of the cases provided as colored printouts for each pair.

Next, several activities were combined into larger blocks to prevent the constant switching between activities, which seemed a bit hectic for students in Study 1. The number of build periods was reduced to two. Both sets of contrasting cases were analyzed in a single reflection period, where students saw each set of cases back-to-back. Instead of giving students a lecture on center of mass, students were given the script for the center of mass lecture as a reading. Finally, students were allowed to access instructional resources, including the reading and images of the cases, at any time during build 2. After both builds, students completed a survey about how valuable the resources were in helping them reach their task goal. Figure 9 shows the study procedure.

Fig. 9
figure9

Study 2 procedure

Measures

Changes made to measures for Study 2 are listed below. Cantilever task performance, transfer task, and deep structure noticing measures were identical to Study 1.

Tests

Students took three paper-and-pencil tests. The pretest was lengthened to eight items: four learning and four transfer items. The learning items were exactly the same on the pretest and posttest. For the transfer tests, isomorphic versions of each of item were created. Tests were counter-balanced such that half of all students were given one set of these questions at pre, and the other set at post.

All test items were enhanced in order to elicit better responses from students. For example the first learning item of the test was changed from “What is center of mass? Give a definition”, to “A new student joins your class, and it is your job to catch them up and explain what center of mass is. How do you explain it to them? What examples might you use?”. For a full list of the learning questions from this study, see Appendix 2.

Two researchers blind-coded 20% of the data and achieved an inter-rater reliability of κ > .70 on all but one item. For items with satisfactory inter-rater reliability, one master coder scored the remaining 80% of data. For the other items, two raters coded all data, discussing and adjudicating all disagreements.

Learning subtest

Two new items were added to the learning test to improve scale reliability. The Study 2 learning subtest contained four questions, which were identical across the pretest (α = .20) and posttest (α = .72). Items evaluated students’ basic, declarative knowledge about center of mass. Learning was measured by averaging across the four learning items on the paper-and-pencil test.

Transfer subtest

Transfer was measured by four transfer questions on the pretest (α = .48), and nine transfer questions on the posttest (α = .64). Several transfer items from Study 1 were replaced to improve construct validity. The posttest originally had ten questions, however one question from the posttest was cut because students could not read the question, due to poor photocopying. Furthermore, when reliability was computed for the transfer scale, this item was negatively correlated with students’ final transfer scores.

The final transfer subtest score averaged the remaining nine questions. As in study 1, reliability between these nine questions was low (α < .70) because this subtest was not intended to measure a single skill. Instead, each question was purposefully designed to test students’ ability to apply center of mass concepts across very different contexts and problem types.

Resource valuing

To measure how much students valued the resources given to them, students were given an 8-item survey at the end of the second build (α = .70). This survey listed resources that students could have used during the build including their partner, their notes, prototypical structures from real life, the center of mass reading, and the contrasting cases. Students were asked to indicate how important each resource was in helping them achieve their task goal on a scale of 1 (not at all important) to 5 (very important), with three indicating neutrality. Student scores were averaged across all items to produce a single resource valuing score.

Study 2 results

Models used for Study 2 were identical to the models used for the same hypotheses in Study 1. For new analyses, the statistical model was chosen based on the nature of the test variables. Preliminary analysis indicated no effects of grade, gender, or period on any outcome variable, so there are no variables accounting for these effects in any model.

Like Study 1, ICC for each outcome variable was calculated (Table 7). The outcome measures with significant ICC values were transfer task scores, noticing the deep structure, and students’ perceived value of the resources. As in Study 1, pair was added as a random effect in the model for each outcome that had a significant ICC value.

Table 7 ICC between dyads on various outcome measures

Task performance

As in Study 1, there was no main effect of task goal on performance. A one-way repeated measures ANOVA with goal as a between-subjects factor indicated a main effect for time F(1,42) = 45.85, p < .001, since student dyad’s structures in both conditions improved over time (Table 8). However there were no other significant main effects or interactions p’s > .15.

Table 8 Average dyad structure length (in inches) by condition

Posttest outcomes

Learning subtest

As in Study 1, students’ goals did not affect learning, when controlling for prior knowledge. To evaluate learning subtest scores at post, an ANCOVA with task goal as a between-subjects factor, and learning pretest score as covariate was run. There was no significant effect of task goal on learning subtest scores, p = .31 (Fig. 10). Students with a content-focused learning goal did just as well as students with an outcome goal. There was a significant main effect for pretest score, F(1,75) = 1.66, p = .02.

Fig. 10
figure10

Student posttest scores on the learning subtest. Error bars are ± 1 SE

Transfer subtest

Content-focused learning goals did improve transfer subtest scores. This is contrary to Study 1, where goals did not affect transfer subtest performance. To evaluate transfer at post, an ANCOVA with task goal as a between-subjects factor, and transfer pretest score as covariate was run. There was a significant effect of task goal on transfer subtest scores F(1,75) = 5.17, p = .03, d = 0.39. Students with a content-focused learning goal performed better on the transfer subtest than students with an outcome goal, controlling for prior knowledge (Fig. 11). There was also a significant main effect for transfer pretest score, F(1,75) = 11.50, p < .01.

Fig. 11
figure11

Student posttest scores on the transfer subtest. Error bars are ± 1 SE

Transfer task

We replicated the direction of the effect in Study 1, where students with a content-focused learning goal did better on the transfer task. In this second study, that effect was not significant at the p = .05 level. To evaluate transfer task performance, a linear mixed-effects model was run with task goal, and transfer pretest score as fixed effects and pair as a random effect. Content-focused learning goals did not significantly affect weight placement on the bird, t(31.2) = 1.88, p = .07, d = 0.45. However descriptively, students who were given a content-focused learning goal put more paper clips in the lower half of their bird than students with an outcome goal. For the pattern of effects, see Fig. 12. There was also a significant effect of transfer pretest score t(69.3) = − 2.43, p = .02.

Fig. 12
figure12

Number of paperclips students put on the bottom half of their bird. Error bars are ± 1 SE

Noticing the deep structure

Similar to Study 1, students with a content-focused learning goal did not notice the deep structure by the end of the contrasting cases reflection more effectively than students with an outcome goal. Across both goal conditions, students typically failed to notice the deep structure (Table 9). A mixed effects logistic regression was run with deep structure noticing as the outcome, task goal as a fixed-effects predictor, and pair as a random intercept. There was no significant effect of content-focused learning goals on deep structure noticing, p = .15.

Table 9 Number of students who did and did not notice the deep structure by goal condition

Resource valuing

Students assigned a content-focused learning goal valued the learning resources they were given more. A hierarchical linear model was run with task goal as a fixed effect and pair as a random effect. There was a significant effect of content-focused learning goals on the perceived value of resources, t(40.88) = 1.98, p = .05, d = 0.48. Students with a content-focused learning goal valued the resources more (M = 3.65, SD = 0.66) than students who were given an outcome goal (M = 3.32, SD = 0.71).

Mechanisms of learning and transfer

To assess relationships between learning processes and outcomes, separate hierarchical regression models were run for each learning and transfer outcome, with fixed effects for building performance, noticing the deep structure, resource valuing, and the relevant pretest scores at level 1. Level 2 modeled a random effect for student pairs on the intercept of the level 1 model. We tested for condition effects and interactions between condition and build length, deep structure noticing, and resource valuing, but none were found. Thus, condition and interaction effects are not included in the models below (see Table 10).

Table 10 Hierarchical linear models of learning and transfer outcomes

There was a significant main effect of resource valuing on learning subtest scores. Students who perceived the resources as more helpful did better on the learning subtest, controlling for prior knowledge. However, unlike Study 1, there was no relationship between deep structure noticing and learning outcomes.

While Study 1 did not find a significant relationship between deep structure noticing and transfer, Study 2 did find a significant effect of deep structure noticing on the transfer subtest. Students who noticed the deep structure scored 12% higher on the transfer subtest (M = 0.57, SD = 0.13) than students who did not notice the deep structure (M = 0.45, SD = 0.15). Nothing was predictive of transfer task performance.

Study 2 discussion

Table 11 summarizes the findings across studies 1 and 2. Note that even under perfect conditions, it is incredibly hard to replicate findings across independent studies, at the p = .05 level (Amrhein et al. 2017). For this reason, many scholars instead consider the p-value to be “graded evidence” against the null hypothesis, whereby an effect is not considered to be null based on the p-value alone. Although some findings did not meet significance thresholds of p ≤ .05, when looking across the two studies there are some consistent trends. Study 2 replicated the direction of the Study 1 finding that content-focused learning goals improve transfer task performance when students are given cases. Study 2 also showed that content-focused learning goals significantly improved performance on a science knowledge transfer test. We attribute this new effect in Study 2 to our fortified transfer test measure and stronger manipulation. Taken together, results across the two studies suggest that when students are given contrasting cases, content-focused learning goals improve transfer of conceptual science knowledge. However, task goals did not influence students’ performance on the engineering task, learning of science concepts, or their noticing of deep scientific structure, in either study.

Table 11 Summary of main findings across Studies 1 and 2

Additionally, Study 2 showed a significant link between deep structure noticing and transfer test performance, when controlling for task performance and prior knowledge. Compare this to our results in Study 1, which showed similar effects but with a small p-value that was > .05 but < .10. Again considering p-values as a graded indication of the likelihood that there was evidence against the null hypothesis, we look at the direction of the findings across the two independent studies, and their implications. In both studies, students who were able to notice the deep structure from the cases performed better on problems that required them to apply those concepts in new contexts. While that trend was not statistically significant in Study 1, the study suffered from a small sample size (n = 48 had noticing data), and only 23% of students noticed the deep structure of the problem. In the second study we suppose that our larger sample (n = 78 had noticing data) allowed us to accurately detect this significant effect between deep structure noticing and transfer ability. This finding supports other literature that links deep structure noticing to transfer (Chi and VanLehn 2012; Loibl et al. 2017; Schwartz et al. 2011).

Surprisingly, there was no relationship between noticing and transfer task performance in either study. Perhaps the transfer task measured a different type of transfer from the transfer test. It is possible that the transfer test measured students' explicit understanding of how center of mass concepts apply in different scenarios, while the transfer task measured a more implicit understanding of the application of center of mass concepts while building. It may be that noticing is important for explicit transfer processes, but less essential for implicit knowledge developed while engineering an artifact.

It surprised our team that even though transfer performance was greater for students with a content-focused learning goal, and deep structure noticing predicted transfer test performance, content-focused learning goals did not affect deep structure noticing. This suggests that there is another mechanism by which content-focused learning goals impacted transfer. It could be that content-focused learning goals were affecting how much students attended to or learned from the tell, which in turn may have affected how much students were able to transfer.

In addition to these replicated findings, a couple of new relationships were found. Content-focused learning goals improved students’ perceived value of the learning resources. Furthermore, students who perceived the learning resources to be more useful performed better on the learning subtest, controlling for prior knowledge. However, students with a content-focused learning goal did not learn more than students with an outcome goal. It may have been that differences in resource valuing were simply not large enough to create significant learning differences between conditions.

General discussion

This work aimed to understand how various factors, including content-focused learning goals, contrasting cases, noticing of deep structures, and perceived value of resources, influenced how students learned and transferred conceptual science knowledge from an engineering task. While there were some inconsistencies in the findings across the two studies and some cases where findings were not significant at the p = .05 level in one study but were in the other, we believe that overall, the pattern of results across the two studies is fairly coherent.

The most significant relationships were between content-focused learning goals and transfer. In Study 1, content-focused learning goals improved transfer task performance, when students analyzed contrasting cases. In Study 2, this finding was not significant at the p = .05 level, although the trend was replicated and the p-value was still fairly small (p < .10). Furthermore, there was a significant effect of content-focused learning goals on transfer subtest scores in Study 2. These results suggest that in general, content-focused learning goals aid transfer of science content from engineering tasks, with medium to large-sized effects. This finding is interesting because although goals consistently improved students’ ability to transfer, there were no effects of goals on engineering task performance, noticing, or learning of scientific concepts, in either study. This suggests that content-focused learning goals have some unique effect on transfer. It may have been that students could perform well on learning and performance measures using other means. For example, students could do well on the science learning measures just by regurgitating information from the lecture, or could do well on the performance task through trial-and-error tinkering or thoughtful iteration (Dow et al. 2009; Marks 2017). However, performance on the transfer test may have required a deeper understanding of the content, which the content-focused learning goals encouraged.

These studies also looked at mechanisms of learning and transfer. In Study 1, deep structure noticing did not significantly predict transfer posttest, though the effect was in the predicted direction (p < .10). In Study 2 this effect was significant, amidst strong controls. In both studies, students who noticed the deep structure during the contrasting cases descriptively performed better on the transfer posttest than those who did not. Taken together, these results suggest that noticing the deep scientific structure of the task may promote transfer of science concepts. However, this finding is tentative, given the small number of students who noticed the deep structure (about 23%). Results also indicated that content-focused learning goals improved the perceived value of learning resources, which in turn was associated with greater science learning. Perhaps valuing learning resources led to greater or more effective use of them, which could promote learning. Given the broad spectrum of literatures covered by this work, there are both theoretical and practical implications for these findings.

Implications for transfer

For the transfer literature, this work provides empirical evidence that content-focused learning goals can affect transfer of science concepts. Transfer is notoriously difficult to obtain even for the simplest of tasks (Detterman 1993). This study indicates that content-focused learning goals are one way to support student’s ability to transfer science concepts from engineering tasks.

Furthermore, this study provides empirical evidence showing that content-focused learning goals may moderate the effect of contrasting cases on conceptual transfer. Many learning activities that use contrasting cases to aid transfer include a goal that focuses students on the deep structure of the task (Kapur 2008; Loibl et al. 2017; Nokes and Belenky 2011; Schwartz et al. 2011). However, we are not aware of any scholarly work that has isolated learning task goals as a critical component of this transfer process. Content-focused learning goals, however, may be necessary for contrasting cases to actually work. The identified interaction between content-focused learning goals and contrasting cases is novel because although past work has suggested the role of contrasting cases and content-focused learning goals on transfer separately, work has not looked at how these two factors interact to affect transfer.

This work is also novel because of the context that it is situated in. Few studies have explored the work of contrasting cases in engineering tasks. There has been some work by Gentner et al. (2016) looking at contrasting case use during a building design challenge in a museum setting. However their population was much younger, and they did not explore how student goals affect resource valuing, noticing, or transfer.

Finally, this work contributes to the transfer and noticing literature. While there has been adequate theoretical and empirical work linking deep structure noticing and transfer (Gick and Holyoak 1983; Greeno et al. 1993; Lobato et al. 2012; Schwartz et al. 2011), there has not been as much empirical work showing a connection between students’ ability to notice deep scientific structure and their transfer performance in engineering contexts (but see Chase et al. 2019). These results echo and extend other research suggesting that noticing the deep structure of the task may be an important transfer process.

Implications for engineering instruction

The beginning of this paper posed how engineering design activities might move beyond “arts and crafts” tasks to actively engage students in learning and transfer processes, with respect to scientific knowledge. This study found that framing engineering tasks with content-focused learning goals enabled students to value provided learning resources and transfer science content. Many engineering tasks use outcome goals, which focus students on producing some intended result. However, this work suggests that engineering tasks which intend to help students learn and transfer science content should be re-framed as learning tasks. Students may also need access to adequate resources, like contrasting cases, to reap the benefits of adopting content-focused learning goals during engineering tasks.

This is especially important because engineering tasks are at risk of turning into fun activities that effectively motivate students, yet fail to teach them science content. Reports of students failing to deeply reflect on science content during engineering tasks may reflect this problem (Berland et al. 2013b; Gertzman and Kolodner 1996; Hmelo et al. 2000; Kanter 2010; Roth et al. 2001; Silk et al. 2009; Vattam and Kolodner 2008). Engineering design curricula with outcome goals often boast that they improve students’ interest in engineering or improve student intrinsic motivation (Brophy et al. 2008; Schunn 2011; Svarovsky and Shaffer 2007). However, this may be happening at the expense of students understanding the scientific content of the task. Engineering tasks should reinforce content-focused learning goals instead of hiding them from students, especially if the intention is for students to transfer science content.

Finally, this work suggests that content-focused learning goals can enhance students’ valuing of learning resources, which students often prefer to ignore (Lee 2017; Mandl et al. 2000). While valuing of resources is not the same as using them effectively, getting students to see their importance has been associated with reports of more frequent resource use (Ryan and Pintrich 1997). We see this as an important achievement for engineering tasks, where students often prefer to focus solely on an engaging engineering task, rather than visit a relevant learning resource.

Limitations

There are some limitations to these studies that warrant the need for further research. First of all, generalizability of findings was hurt by the fact that both studies used science students that were considered advanced within their school. Also, for some of the findings, effects were not statistically significant at the p = .05 level across the two studies; in some cases p was < .05 in one study but < .10 in another. However, we only interpreted these marginal effects when (1) we had a strong hypothesis for the direction of the effect before the study, based on literature suggesting the effect should exist, (2) the direction of the effect found in the study results aligned with our original hypothesis, and (3) descriptive results suggested that the effects were consistently in the same direction across the two independent studies. While some variation in results across studies is expected due to chance, future work should test whether these findings replicate in other settings and with other (non-advanced) student populations, before strong conclusions can be drawn.

Second, many of our test measures had low internal consistency (α’s < .70). A low α is typical on pretest measures when students have low prior knowledge, because they often answer with random guessing, which yields low consistency across test items. Also, many of our pretest measures contained few items, which can also yield low internal consistency (Loibl and Rummel 2014; Tavakol and Dennik 2011). Finally, it can be challenging to achieve a large Cronbach’s alpha, when a measure assesses multiple forms of knowledge (procedural and conceptual) and content (e.g. relationship of center of mass to base, locating center of mass in X and Y directions, equation for center of mass), set in very different contexts (e.g. glasses on a tray, Lego structures, coordinate systems), as was the case on our transfer subtest. While these factors may have influenced the internal consistency of our measures, they did not necessarily influence validity. Indeed, the two transfer posttests with α’s < .70 did correlate with other related measures such as task performance and transfer pretests. Nonetheless, findings related to the transfer subtest should be replicated with more reliable measures before strong conclusions can be drawn.

Third, the resource valuing questionnaire is a self-report measure, which can have flaws because students often fail to effectively evaluate how valuable something actually is. These self-report measures are therefore less reliable than other measures that directly assessed student knowledge or behavior. Therefore, analyses including these measures should be considered with caution. Future work should measure actual student resource use to see if it is affected by the task goal. Qualitative work could also look at how students are using these resources in a way that aids their learning (e.g., Malkiewich and Chase 2019).

Fourth, this study failed to identify a mechanism for how content-focused learning goals improved transfer performance. Although noticing the deep scientific structure of the problem was associated with better transfer (particularly in Study 2), content-focused learning goals did not improve deep structure noticing. Future work could investigate whether a more sensitive measure of deep structure noticing or taking this measure at later time points would detect this effect. In addition, future research could explore other factors, which may have led content-focused learning goals to improve student transfer. For example, other research suggests that it is the ability of students to focus on the deep structure over time, rather than just merely notice it, which may lead to transfer (Malkiewich and Chase 2019). Alternatively, how students integrate the deep structure into their building process may be what aids transfer.

Finally, this study was focused largely on students’ learning and transfer of science content knowledge. However we acknowledge that there are other important outcomes of engineering design tasks such as learning twenty first century skills (Koh et al. 2015), raising interest in engineering careers (Bottoms and Anthony 2005), and teaching valuable engineering practices and habits of mind, such as the importance of iteration and learning from failure (Marks 2017; Lammi et al. 2018). In future research, it would be interesting to explore the impact of content-focused learning versus outcome goals on other desired outcomes of engineering curricula.

Conclusion

In summary, these studies showed that content-focused learning goals improved students’ ability to transfer science concepts from engineering tasks, when they had access to the instructional scaffold of contrasting cases. As engineering tasks become more popular for teaching science content knowledge, it is important to discern the role of goals and contrasting cases in these types of tasks. While researchers have developed engineering design curricula with a wide variety of instructional scaffolds, few studies explore the use of contrasting cases in engineering curricula, and few studies focus on how goals affect students’ ability to transfer science knowledge beyond the engineering task.

We propose that engineering tasks should use content-focused learning goals that focus students on the science content of a problem. While this goal might not affect students’ ability to build more successful constructions or notice the deep structure of the problem, it can direct students to finding more value in learning resources, and ultimately prepare them to transfer their understandings to novel contexts.

We now live in a world where education is not just about students’ ability to learn and regurgitate facts they acquire in the classroom. Engineering design activities are important for student development because they have the potential to engage students in meaningful construction processes that can support the learning and application of science concepts. Engineering tasks must in turn support students so that they may transfer these skills to new problems in new contexts. Otherwise, these activities are not meeting their true potential of delivering value to students. If we as an educational community want students to take the science they learn from engineering activities and adapt it for new contexts, then students need to be given goals that will direct them through learning processes that will best prepare them to apply and extend what they learn.

References

  1. Ahmed, S., Wallace, K. M., & Blessing, L. T. (2003). Understanding the differences between how novice and experienced designers approach design tasks. Research in Engineering Design, 14(1), 1–11.

    Article  Google Scholar 

  2. Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205–223.

    Article  Google Scholar 

  3. Aleven, V., Stahl, E., Schworm, S., Fischer, F., & Wallace, R. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research, 73(3), 277–320.

    Article  Google Scholar 

  4. Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students’ learning strategies and motivation processes. Journal of Educational Psychology, 80(3), 260–267.

    Article  Google Scholar 

  5. Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017). The earth is flat (p > 0.05): Significance thresholds and the crisis of unreplicable research. PeerJ, 5, e3544. https://doi.org/10.7717/peerj.3544.

    Article  Google Scholar 

  6. Apedoe, X. S., & Schunn, C. D. (2012). Strategies for success: Uncovering what makes students successful in design and learning. Instructional Science, 41(4), 773–791.

    Article  Google Scholar 

  7. Arbreton, A. (1998). Student goal orientation and help seeking strategy use. In S. A. Karabenick (Ed.), Strategic help seeking: Implications for learning and teaching (pp. 95–116). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  8. Barnett, S. M., & Ceci, S. J. (2002). When and where do I apply what I learn?: A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637.

    Article  Google Scholar 

  9. Barron, B. J., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., et al. (1998). Doing with understanding: Lessons from research on problem-and project-based learning. Journal of the Learning Sciences, 7(3–4), 271–311.

    Google Scholar 

  10. Berland, M., Martin, T., Benton, T., Petrick Smith, C., & Davis, D. (2013a). Using learning analytics to understand the learning pathways of novice programmers. Journal of the Learning Sciences, 22(4), 564–599.

    Article  Google Scholar 

  11. Berland, L., Martin, T., Ko, P., Baker Peacock, S., Rudolph, J., & Golubski, C. (2013b). Student learning in challenge-based engineering curricula. Journal of Pre-College Engineering Education Research, 3(1), 53–64.

    Google Scholar 

  12. Bottoms, G., & Anthony, K. (2005). Project lead the way: A pre-engineering curriculum that works. Southern Regional Education Board. Atlanta, GA. Retrieved June 27, 2014 from http://www.sreb.org/programs/hstw/publications/briefs/05V08_Research_PLTW.pdf.

  13. Bransford, J. D., Brown, A., & Cocking, R. (1999). How people learn: Mind, brain, experience, and school. Washington, DC: National Research Council.

    Google Scholar 

  14. Bransford, J. D., Franks, J. J., Vye, N. J., & Sherwood, R. D. (1989). New approaches to instruction: Because wisdom can’t be told. In S. Vosniadou & A. Ortony (Eds.), Similarity and analogical reasoning (pp. 470–497). Cambridge, MA: Cambridge University Press.

    Google Scholar 

  15. Bransford, J. D., & Schwartz, D. L. (1999). Chapter 3: Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24(1), 61–100.

    Article  Google Scholar 

  16. Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P-12 classrooms. Journal of Engineering Education, 97(3), 369–387.

    Article  Google Scholar 

  17. Bundy, B. M. (2013). AP physics 1: Algebra-based course planning and pacing guide. New York: The College Board.

    Google Scholar 

  18. Chase, C. C., Harpstead, E., & Aleven, V. (2017). Inciting out-of-game transfer: Adapting contrast-based instruction for educational games. In Paper presented at Games+Learning+Society, Madison, WI.

  19. Chase, C. C., Malkiewich, L. J., & Kumar, A. (2019). Learning to notice science concepts in engineering activities and transfer situations. Science Education, 103(2), 440–471. https://doi.org/10.1002/sce.21496.

    Article  Google Scholar 

  20. Chi, M. T., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152.

    Article  Google Scholar 

  21. Chi, M. T., & VanLehn, K. A. (2012). Seeing deep structure from the interactions of surface features. Educational Psychologist, 47(3), 177–188.

    Article  Google Scholar 

  22. Day, S. B., & Goldstone, R. L. (2012). The import of knowledge export: Connecting findings and theories of transfer of learning. Educational Psychologist, 47(3), 153–176.

    Article  Google Scholar 

  23. Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1–24). Westport, CT: Able Publishing.

    Google Scholar 

  24. Dow, S. P., Heddleston, K., & Klemmer, S. R. (2009, October). The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on creativity and cognition (pp. 165–174). ACM.

  25. Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10), 1040–1048.

    Article  Google Scholar 

  26. Elliot, A. J., & Harackiewicz, J. M. (1994). Goal setting, achievement orientation, and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 66(5), 968–980.

    Article  Google Scholar 

  27. Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., & Mamlok-Naaman, R. (2004). Design-based science and student learning. Journal of Research in Science Teaching, 41(10), 1081–1110.

    Article  Google Scholar 

  28. Gardner, A. K., Diesen, D. L., Hogg, D., & Huerta, S. (2016). The impact of goal setting and goal orientation on performance during a clerkship surgical skills training program. The American Journal of Surgery, 211(2), 321–325.

    Article  Google Scholar 

  29. Gentner, D., Levine, S. C., Ping, R., Isaia, A., Dhillon, S., Bradley, C., et al. (2016). Rapid learning in a children’s museum via analogical comparison. Cognitive Science, 40(1), 224–240.

    Article  Google Scholar 

  30. Gero, J. S., Jiang, H., & Williams, C. (2013). Design cognition differences when using unstructured, partially structured and structured concept generation creativity techniques. International Journal of Design Creativity and Innovation, 1(4), 196–214.

    Article  Google Scholar 

  31. Gertzman, A. D., & Kolodner, J. L. (1996, July). A case study of problem-based learning in a middle school science classroom: Lessons learned. In Proceedings of the 1996 international conference on learning sciences (pp. 91–98). International Society of the Learning Sciences.

  32. Gibson, E. J. (1988). Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge. Annual Review of Psychology, 39(1), 1–41.

    Article  Google Scholar 

  33. Gibson, J. J., & Gibson, E. J. (1955). Perceptual learning: Differentiation or enrichment? Psychological Review, 62(1), 32–41.

    Article  Google Scholar 

  34. Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15(1), 1–38.

    Article  Google Scholar 

  35. Greeno, J. G., Moore, J. L., & Smith, D. R. (1993). Transfer of situated learning. Westport, CT: Ablex Publishing.

    Google Scholar 

  36. Hmelo, C. E., Holton, D. L., & Kolodner, J. L. (2000). Designing to learning about complex systems. Journal of the Learning Sciences, 9(3), 247–298.

    Article  Google Scholar 

  37. Holbrook, J., & Kolodner, J. L. (2000). Scaffolding the development of an inquiry-based (science) classroom. In B. Fishman & S. O’Connor-Divelbiss (Eds.), Proceedings of international conference of the learning sciences ICLS 2000 (pp. 221–227). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

  38. Kanter, D. E. (2010). Doing the project and learning the content: Designing project-based science curricula for meaningful understanding. Science Education, 94(3), 525–551.

    Google Scholar 

  39. Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424.

    Article  Google Scholar 

  40. Kenny, D. A., Kashy, D. A., Cook, W. L., & Simpson, J. A. (2006). Dyadic data analysis. New York, NY: The Guilford Press.

    Google Scholar 

  41. Klahr, D., Triona, L. M., & Williams, C. (2007). Hands on what? The relative effectiveness of physical versus virtual materials in an engineering design project by middle school children. Journal of Research in Science Teaching, 44(1), 183–203.

    Article  Google Scholar 

  42. Koh, J. H. L., Chai, C. S., Wong, B., & Hong, H.-Y. (2015). Design thinking for education. Singapore: Springer.

    Google Scholar 

  43. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., … & Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design (tm) into practice. The Journal of the Learning Sciences, 12(4), 495–547.

  44. Kolodner, J. L., Gray, J., & Fasse, B. (2003b). Promoting transfer through case-based reasoning: Rituals and practices in learning by design™ classrooms. Cognitive Science Quarterly, 3(2), 183–232.

    Google Scholar 

  45. Kurtz, K. J., Miao, C. H., & Gentner, D. (2001). Learning by analogical bootstrapping. The Journal of the Learning Sciences, 10(4), 417–446.

    Article  Google Scholar 

  46. Lachapelle, C. P., & Cunningham, C. M. (2007, June). Engineering is elementary: Children’s changing understandings of science and engineering. In Paper presented at American Society for Engineering Education Annual Conference & Exposition, Honolulu, HI.

  47. Lammi, M., Denson, C., & Asunda, P. (2018). Search and review of the literature on engineering design challenges in secondary school settings. Journal of Pre-College Engineering Education Research, 8(2), 49–66.

    Article  Google Scholar 

  48. Latham, G. P., & Brown, T. C. (2006). The effect of learning vs. outcome goals on self-efficacy, satisfaction and performance in an MBA program. Applied Psychology, 55(4), 606–623.

    Article  Google Scholar 

  49. Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday life. Cambridge: Cambridge University Press.

    Google Scholar 

  50. Lee, A. (2017). Productive responses to failure for future learning. Doctoral dissertation, Columbia University.

  51. Lobato, J., Rhodehamel, B., & Hohensee, C. (2012). “Noticing” as an alternative transfer of learning process. Journal of the Learning Sciences, 21(3), 433–482.

    Article  Google Scholar 

  52. Locke, E. A., & Bryan, J. (1969). The directing function of goals in task performance. Organizational Behavior and Human Performance, 4(1), 35–42.

    Article  Google Scholar 

  53. Locke, E. A., & Latham, G. P. (1990). A theory of goal setting & task performance. Englewood Cliffs, NJ: Prentice-Hall Inc.

    Google Scholar 

  54. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717.

    Article  Google Scholar 

  55. Locke, E. A., Shaw, K. N., Saari, L. M., & Latham, G. P. (1981). Goal setting and task performance: 1969–1980. Psychological Bulletin, 90(1), 125–152.

    Article  Google Scholar 

  56. Loibl, K., Roll, I., & Rummel, N. (2017). Towards a theory of when and how problem solving followed by instruction supports learning. Educational Psychology Review, 29(4), 693–715.

    Article  Google Scholar 

  57. Loibl, K., & Rummel, N. (2014). Knowing what you don’t know makes failure productive. Learning and Instruction, 34, 74–85. https://doi.org/10.1016/j.learninstruc.2014.08.004.

    Article  Google Scholar 

  58. Malkiewich, L. J., & Chase, C. C. (2019). Focusing processes: Potential pathways for transfer of science concepts from an engineering task. International Journal of Science Education. https://doi.org/10.1080/09500693.2019.1613583.

    Article  Google Scholar 

  59. Mandl, H., Gräsel, C., & Fischer, F. (2000). Problem-oriented learning: Facilitating the use of domain-specific and control strategies through modeling by an expert. In W. J. Perrig & A. Grob (Eds.), Control of human behavior, mental processes and consciousness (pp. 165–182). Mahwah, NJ: Erlbaum.

    Google Scholar 

  60. Marks, J. (2017). The impact of a brief design thinking intervention on students’ design knowledge, iterative dispositions, and attitudes towards failure. Doctoral dissertation, Columbia University.

  61. Mercier, E. M. (2017). The influence of achievement goals on collaborative interactions and knowledge convergence. Learning and Instruction, 50, 31–43.

    Article  Google Scholar 

  62. Miller, C. S., Lehman, J. F., & Koedinger, K. R. (1999). Goals and learning in microworlds. Cognitive Science, 23(3), 305–336.

    Article  Google Scholar 

  63. National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: The National Academies Press.

    Google Scholar 

  64. Nokes, T. J., & Belenky, D. M. (2011). Incorporating motivation into a theoretical framework for knowledge transfer. In J. Mestre & B. H. Ross (Eds.), Psychology of learning and motivation (Vol. 55, pp. 109–135)., Cognition in Education San Diego, CA: Academic Press.

    Google Scholar 

  65. Perkins, D. N., & Salomon, G. (1988). Teaching for transfer. Educational Leadership, 46(1), 22–32.

    Google Scholar 

  66. Petrosino, A. J. (1998). The use of reflection and revision in hands-on experimental activities by at-risk children. Unpublished doctoral dissertation, Vanderbilt University, Nashville, TN.

  67. Pick, H. L. (1992). Eleanor J. Gibson: Learning to perceive and perceiving to learn. Developmental Psychology, 28(5), 787–794.

    Article  Google Scholar 

  68. Renkl, A. (2002). Learning from worked-out examples: Instructional explanations supplement self-explanations. Learning & Instruction, 12, 529–556.

    Article  Google Scholar 

  69. Riskowski, J. L., Todd, C. D., Wee, B., Dark, M., & Harbor, J. (2009). Exploring the effectiveness of an interdisciplinary water resources engineering module in an Either Grade Science Course. International Journal of Engineering Education, 25(1), 181–195.

    Google Scholar 

  70. Roll, I., Aleven, V., & Koedinger, K. (2011, January). Outcomes and mechanisms of transfer in invention activities. In Proceedings of the Annual Meeting of the Cognitive Science Society, 33(33), 2824–2829.

  71. Roth, W. M., Tobin, K., & Ritchie, S. (2001). Re/constructing elementary science. New York: Peter Lang Publishing.

    Google Scholar 

  72. Rothkopf, E. Z., & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive processing model from inspection times and eye movements. Journal of Educational Psychology, 71(3), 310–327.

    Article  Google Scholar 

  73. Ryan, A. M., & Pintrich, P. R. (1997). “ Should I ask for help?” The role of motivation and attitudes in adolescents’ help seeking in math class. Journal of Educational Psychology, 89(2), 329–341.

    Article  Google Scholar 

  74. Schauble, L., Klopfer, L. E., & Raghavan, K. (1991). Students’ transition from an engineering model to a science model of experimentation. Journal of Research in Science Teaching, 28(9), 859–882.

    Article  Google Scholar 

  75. Schnittka, C., & Bell, R. (2011). Engineering design and conceptual change in science: Addressing thermal energy and heat transfer in eighth grade. International Journal of Science Education, 33(13), 1861–1887.

    Article  Google Scholar 

  76. Schunk, D. H., & Swartz, C. W. (1993). Goals and progress feedback: Effects on self-efficacy and writing achievement. Contemporary Educational Psychology, 18(3), 337–354.

    Article  Google Scholar 

  77. Schunn, C. (2011). Design principles for high school engineering design challenges: Experiences from high school science classrooms. National Center for Engineering and Technology Education. Retrieved July 18, 2019 from http://eric.ed.gov/?id=ED537383.

  78. Schwartz, D. L., & Arena, D. (2013). Measuring what matters most: Choice-based assessments for the digital age. Cambridge: MIT Press.

    Google Scholar 

  79. Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–5223.

    Article  Google Scholar 

  80. Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103(4), 759–775. https://doi.org/10.1037/a0025140.

    Article  Google Scholar 

  81. Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129–184.

    Article  Google Scholar 

  82. Shemwell, J. T., Chase, C. C., & Schwartz, D. L. (2015). Seeking the general explanation: A test of inductive activities for learning and transfer. Journal of Research in Science Teaching, 52(1), 58–83. https://doi.org/10.1002/tea.21185.

    Article  Google Scholar 

  83. Shute, V. J., & Gluck, K. A. (1996). Individual differences in patterns of spontaneous online tool use. Journal of the Learning Sciences, 5, 329–355.

    Article  Google Scholar 

  84. Silk, E. M., & Schunn, C. D. (2008, January). Utilizing contrasting cases to target science reasoning and content in a design-for-science unit. In Annual Meeting of the National Association for Research in Science Teaching (NARST), Baltimore, MD.

  85. Silk, E. M., Schunn, C. D., & Cary, M. S. (2009). The impact of an engineering design curriculum on science reasoning in an urban setting. Journal of Science Education and Technology, 18(3), 209–223.

    Article  Google Scholar 

  86. Svarovsky, G. N., & Shaffer, D. W. (2007). Soda constructing knowledge through exploratoids. Journal of Research in Science Teaching, 44(1), 133–153.

    Article  Google Scholar 

  87. Tavakol, M., & Dennik, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd.

    Article  Google Scholar 

  88. University of Bristol (n.d.). Sample sizes for multilevel modeling. Retrieved January 5, 2019 from http://www.bristol.ac.uk/cmm/learning/multilevel-models/samples.html.

  89. Vattam, S., & Kolodner, J. L. (2008). On foundations of technological support for addressing challenges facing design-based science learning. Pragmatics & Cognition, 16(2), 406–437.

    Article  Google Scholar 

  90. Wood, D. (2001). Scaffolding, contingent tutoring, and computer-supported learning. International Journal of Artificial Intelligence in Education, 12, 280–292.

    Google Scholar 

  91. Worsley, M., & Blikstein, P. (2014). Assessing the Makers: The Impact of Principle-Based Reasoning on Hands-on, Project-Based Learning. In J. L. Polman, E. A. Kyza, D. K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, … L. D’Amico (Eds.) Proceedings of the 2014 International Conference of the Learning Sciences (pp. 1147–1151). Boulder, CO.

Download references

Acknowledgements

This work was supported by two grants from Teachers College Columbia University (Research Dissertation Fellowship, and the Dean’s Grant for Student Research). We thank the following colleagues for their contributions to various parts of the project, including data collection and general advice: Aakash Kumar, Bryan Keller, Deanna Kuhn, Naomi Choodnovski, Matthew Zellman, Vivian Chang, Li Jiang, Xinxu Shen, Kimberly Zambrano, and Elisabeth Hartman.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Catherine C. Chase.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

Studies presented in this paper were vetted and approved by the Institutional Review Board at Teachers College, Columbia University, and the school board for the school where the study was performed.

Informed consent

Informed assent was obtained from all student participants, and informed consent was obtained from their parents.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Sample transfer posttest questions

(Study 1 question) A waiter is carrying a tray with five water glasses on it, as pictured below. The first glass contains 4 oz. of water. The second glass is empty. The third glass contains 4 oz. of water. The fourth glass contains 8 oz of water. The fifth glass is empty. Circle the arrow that points to the center of mass of this tray of water glasses. Explain how you got your answer.

figurea

(Question used in both studies) The two buildings below are plans for skyscrapers in San Francisco, and have been built to withstand an earthquake. Both buildings have their weight evenly distributed throughout; no part of the building is heavier or lighter than any other part. Given what you know about Center of Mass, circle which building do you think will be more likely to withstand an earthquake. Why did you choose that one?

figureb

(Study 2 question) Below is an unbalanced structure built from Legos. If I wanted to balance it, which Lego would I remove? Explain your choice.

figurec

Appendix 2: Study 2 learning test questions

  1. 1.

    A new student joins your class, and it is your job to catch them up and explain what center of mass is. How do you explain it to them? What examples might you use?

  2. 2.

    Explain what features of a structure affect the location of its center of mass. Describe precisely HOW these features impact the center of mass.

  3. 3.

    What is the equation for center of mass?

  4. 4.

    The late student from problem 1 is back in class. How would you explain the equation to them?

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Malkiewich, L.J., Chase, C.C. What’s your goal? The importance of shaping the goals of engineering tasks to focus learners on the underlying science. Instr Sci 47, 551–588 (2019). https://doi.org/10.1007/s11251-019-09493-2

Download citation

Keywords

  • Transfer
  • Engineering education
  • Contrasting cases
  • Learning goals
  • Physics learning