Introduction

The increasing accessibility of online resources and artificial intelligence tools such as ChatGPT has necessitated science, technology, engineering, and mathematics (STEM) faculty to reassess what it means for students to know and understand science topics. One way to view this situation is to pivot from examining how students gain factual knowledge to how they develop systems thinking (Ewing & Sadler, 2020) or sensemaking (Kaldaras & Wieman, 2023; Odden & Russ, 2018; York et al., 2019; Zhao & Schuchardt, 2021) to integrate mathematical reasoning with scientific ideas. As mathematics educators teaching preparatory classes, we want to take a closer look at the sensemaking experiences students are expected to engage in once they enroll in a typical large-lecture STEM class. In this study, we examine which tools and habits of mind ostensibly learned in prior mathematics courses support students’ sensemaking in a physics course and which of these tools and habits are found lacking. In some sense, this is a wake-up call. If mathematics courses are not serving the needs of partner disciplines to promote sensemaking and integrated thinking, why would these majors continue to require mathematics courses at all?

The basic idea of promoting sensemaking skills in a world where answers appear on demand is to have students use all tools available to determine the veracity and accuracy of their results and for them to use these results to lead to more complex scientific thinking and modeling. As Sirnoorkar et al. (2023) note, “despite being two distinct processes, modeling is often described as sensemaking of the physical world” (p. 010118–1). The framework proposed by Kaldaras and Wieman (2023) goes further to propose a cognitive framework to map out sensemaking processes via three levels of cognition: qualitative, quantitative, and conceptual, with a specific emphasis on how students reach the conceptual level.

The question of how to promote these ideas in classrooms was discussed in a recent editorial by Li and Schoenfeld (2019), who consider the idea of sensemaking as an active, self-derived set of understandings by contrasting it with the traditional method of presenting mathematics as a set of rules. They argue that mathematics education needs to be transformed into a “codification of experiences of both making sense and sensemaking through various practices including problem-solving, reasoning, communicating, and mathematical modeling” (Li & Schoenfeld, 2019, p. 1). Their view of how this could be enacted is presented as a framework for examining mathematics opportunities in classrooms that begin with experiences rather than content, that include cognitively demanding activities and discussions, that provide equitable access to learning opportunities and technologies, and that include ongoing formative assessment. Their overall thesis is that students need to be mathematically proficient and confident in order to engage in sensemaking once they head into STEM courses at the college level. The irony of the situation is that, while there have been efforts to address these needs for years, the advent of artificial intelligence may be the catalyst that finally raises awareness for the need to work with partner disciplines to rethink what it means to make sense of scientific information. In what follows, we provide a brief history of efforts from within the mathematics community to reach out to partner disciplines. We build on these efforts to create a list of practical considerations that emerged in one classroom as students attempted to make sense of physics problems using effective mathematical practices to continue the discussion of how to promote sensemaking.

Background: Mathematics Curricula and Partner Discipline Engagement

The Mathematics Association of America (MAA) has been actively focused on revising and updating collegiate mathematics curricula since 1953. The Committee on the Undergraduate Program in Mathematics (CUPM) has supported efforts to study and update curricula and teaching to address the needs for teacher training (e.g., COMET, 1992; Macduffee, 1953), the importance of statistics (e.g., CUPM Panel on Statistics, 1971; Hogg, 1992), and the inclusion of computer science (CUPM Subpanel on Computer Science, 1981; Hohn, 1955).

Starting in the 1990s, the MAA’s Curriculum Foundation Project began publishing curriculum guides (e.g., Schumacher et al., 2015) to inform college curricula planning. These guides have been informed by several subgroups including Curriculum Renewal Across the First Two Years (CRAFTY) or its alternate form of Calculus Reform and the First Two Years, which were established to examine how mathematics courses could be more responsive to the wide variety of client disciplines that they support (including both STEM and non-STEM partners). The first step that the CRAFTY group took was to convene several department-specific working groups to ask partner faculty a relatively straightforward question: What mathematical topics do you expect your incoming students to understand and what skills do you expect them to have as they begin your course? The results were published in two volumes containing chapters written by faculty from each of the disciplines engaged in the various workshops. The first volume contains 19 chapters focusing on how calculus courses could be reformed to better support business and management, teacher education, and each of the STEM fields (Ganter & Barker, 2004). The second volume addresses additional partner disciplines (agriculture, arts, economics, meteorology, and social sciences) and focuses on changes that could be implemented in introductory college mathematics courses (Ganter & Haver, 2011).

Despite the efforts of the CRAFTY reports and the CUPM guides, sustained collaboration between mathematics instructors and partners in other disciplines has been scant. One effort to support more robust collaborations was recently led by the editors of the original CRAFTY reports. The National Science Foundation (NSF)–funded National Consortium for Synergistic Undergraduate Mathematics via Multi-Institutional Interdisciplinary Teaching Partnerships (SUMMIT-P) project involved forming a consortium of 12 universities who partnered mathematics and other discipline faculty to encourage collaboration efforts that were context-appropriate. This led to a variety of outcomes including interdisciplinary faculty learning communities (Poole et al., 2022); methods for knowledge transfer between mathematics and engineering (Ellwein Fix et al., 2022); explicit collaboration protocols (cf. Hofrenning et al., 2020); cross-curricular projects (Wood & Bourdeau, 2022); the inclusion of student partners from different majors (Bowers et al., 2020a); and partner discipline teaching reforms (cf. Luque et al., 2022; Bowers et al., 2020b). The group has recently completed a compendium of classroom-ready mathematics activities (Ganter et al., 2021) that was recently published by the MAA.

These collaborations, with a focus on even greater and more sustained engagement between mathematics and partner disciplines, inspired the work described in this report. Our approach involved going into a physics classroom (virtually) to catalog the extent to which students engaged in various math practices that CRAFTY faculty expect their students to use. Thus, we hope to add to the conversation spawned by the CRAFTY reports so that mathematics and science educators can emphasize these important math practices to enhance instruction in their service courses.

Mathematical Recommendations from the Physics Discipline

The first CRAFTY workshop to include physics faculty took place at Bowdoin College in 1999. The meeting included a variety of physics professors and instructors. In order for the focus to remain on the mathematical skills needed for physics content rather than on conflicting demands within the mathematics curriculum itself, mathematicians, while present, did not enter the conversation. The resounding takeaway from the meeting was highlighted in the opening statement of their subsequent report: “Conceptual understanding of basic mathematical principles is very important for success in introductory physics. It is more important than esoteric computational skill. However, basic computational skill is crucial” (Cummings & Emery, 2004, p. 115). Other statements within the document reiterated this idea in different ways, suggesting that physics instructors were looking for mathematically confident students who could think through a problem, rather than seeking students who could, for example, compute advanced integrals while lacking the ability or predilection to make sense of the answer once it was computed.

Of course, the panel also recommended specific mathematical topics that they expected students to know. These were presented in three rungs labeled high, higher, and highest. For the calculus-based introductory physics course that we studied, the instructor (and second author of this paper) highlighted the following topics within each rung of that list. At the high rung, he identified “polar and other coordinate systems.” At the higher and highest rungs, he identified “limiting cases” and “behavior of simple functions, derivatives of simple functions, and integrals of simple functions,” respectively (Cummings & Emery, 2004, p. 116). However, the most important skill he required of students was described in the report as “…a well-placed confidence in their trigonometric and algebraic skills” (Anderson, personal communication, June 2022; Cummings & Emery, 2004, p. 116).

Mathematical Recommendations Based on High School Mathematics Practices

Where, then, do incoming students develop conceptual understanding and mathematical confidence? On average, only roughly 10% of students who take introductory physics have met the mathematical prerequisites (including calculus I) in high school, while the remainder take at least one mathematics course in college. Interestingly, as Burkholder et al. (2021) reported, prior college math coursework taught in a traditional manner is not always a good predictor of success in an introductory physics course. In contrast, the authors found that those students in a second physics course (mechanics) did benefit from taking at least one course in vector calculus.

Regardless of the mathematical pathway taken, most American physics students’ mathematical backgrounds were likely affected by the recommendations from the Common Core State Standards for Mathematics (National Governors Association Center for Best Practices, Council of Chief State School Officers, 2010). At the time of the standards’ publication, most of the students in our study were in high school and, therefore, most likely encountered some aspects of the standards movement. Moreover, the Standards for Mathematical Practice (a component of the Common Core State Standards) has influenced our teaching of the service courses that many students in the physics class were required to take (Bowers et al., 2019).

Given this potentially high level of exposure, we turned to the list of eight Standards for Mathematical Practice to begin chronicling general “habits of mind” that have been shown to be correlated with student success (Kilpatrick et al., 2003; National Research Council, 1996; Stylianides & Stylianides, 2007). Although under continuous critique, the Common Core Standards for Mathematical Practice offers the most current and concise descriptions of mathematical practices in which proficient math students engage. Even critics agree that the list should be acknowledged for promoting, in equal part, conceptual understanding, fluency, and application (Pondiscio & Mahnken, 2014). While some deem the list “elitist” in the sense that it envisions all students as being college- and career-ready (e.g., Wood, 2014), we argue that our audience of students are in college and, therefore, most likely did benefit from being exposed to the ideas and goals incorporated into these standards.

In summary, we argue a strong rationale for using selected Standards for Mathematical Practice—augmented by the Next Generation Science Standards (National Research Council, 2013), the CRAFTY workshop notes (Cummings & Emery, 2004), and other educational research—as the basis of our framework for evaluating students’ videos. In the following sections, we describe a framework of considerations we used to analyze the videos in the order in which they appeared as the speakers presented their solutions. The first three considerations relate to the introduction of the problem (problem setup, modeling with variables, and modeling with functions). The next two considerations relate to the actual problem solution process (conceptual orientation and use of precise terminology). The final two relate to the conclusion of the presentations (conducting a unit analysis and obtaining a correct answer).

Consideration 1: Problem Setup

The first Standard for Mathematical Practice (the standards will hereafter be referred to as MP 1, MP 2, etc.) states, “Mathematically proficient students start by explaining to themselves the meaning of a problem and looking for entry points to its solution” (Common Core Standards Initiative, 2010, p. 6). This succinct explanation can be used as a strong first guidepost when analyzing how students perceive their task. When solving physics problems, most tasks require identifying known and unknown quantities, variables, and relationships. Other aspects within this consideration include how the speaker(s) introduces the problem, brings in background knowledge, and demonstrates how to think through ways to solve the problem. Some students might even go so far as to anticipate a reasonable range or magnitude for the answer.

Consideration 2: Modeling with Variables

A second aspect of successful problem-solving is developing models. This process involves conceptualizing—often with diagrams in the case of physics—how the given quantities interact with each other and with other constraints of the problem. MP 4 suggests that problem solvers “… are able to identify important quantities in a practical situation and map their relationships using such tools as diagrams, two-way tables, graphs, flowcharts, and formulas” (Common Core Standards Initiative, 2010, p. 7).

For the purpose of this study, the goal for considering modeling with variables is to distinguish between students thinking about the variables and how they are related to the quantities and units versus students appearing to follow an algorithm without thinking about its relation to the values given. One indication of not satisfying this consideration would be if a presenter does not include a negative sign in a model for a variable that represents an opposite force. Another aspect of this consideration relates to ratios used in functions. Do students understand how proportional relationships—measured as intensive quantities—are used to measure a relationship within a given problem?

Consideration 3: Modeling with Functions

The practice of modeling is complex, with the “black box” analogy often used to describe how students come up with drawings and mathematical equations to solve physics problems. In an attempt to open up this box, Sirnoorkar et al. (2023) have used think-aloud protocols to examine the steps within a “sensemaking epistemic game” (Odden & Russ, 2018). Although our methodology (examining extant videos) did not allow us to observe the specific arguments that students used to justify their modeling, we can view their choice of functions as a stage within their sensemaking process. We considered, for example, whether the students had generated an explanation for their function (analogous to the third stage in the sensemaking epistemic game) as opposed to making an implicit assumption that because they were studying Chapter X, they must use the formula introduced therein.

While related to MP 4 (modeling), the modeling with functions consideration also draws from MP 7, which focuses on how problem solvers choose formulae that suit their situation. This also dovetails with the “Structure and Function” standard described in the Next Generation Science Standards, which cites one example from the Science and Engineering Practices: “Constructing explanations and designing solutions in 9–12 builds on K–8 experiences and progresses to explanations and designs that are supported by multiple and independent student-generated sources of evidence consistent with scientific ideas, principles, and theories” (National Research Council, 2013).

Consideration 4: Assuming a Conceptual Orientation

The hypothesis that underlies the fourth consideration is that the current ideas of sensemaking and math-science integration can be evidenced through the lens of students’ development of a conceptual orientation for mathematics. The idea of conceptual orientation evidenced through language was first discussed by Thompson et al. (1994) as a method for thinking about how teachers’ use of language influences students’ engagement in mathematically successful problem-solving paradigms. The authors establish a contrast between calculational orientations and conceptual orientations by analyzing the way that teachers and students describe their work. Discussants who have conceptual orientations focus on ideas and ways of referring to variables in context. In contrast, those with calculational orientations tend to focus on algorithmic procedures. Their language involves describing steps to complete a computation, but does not explain the reasoning behind those steps. When video explanations focus only on procedures, it creates a problem for the viewer and can also signal a problem for the presenter. It is unlikely that a viewer who does not already understand the landscape or the domain would be able to create an understanding based solely on trying to make sense of a series of computations (cf. Erlwanger, 1975). Moreover, with regard to the presenters’ own learning, Thompson et al. (1994) note, “It is important that students appreciate that the most powerful approach to solving problems is to understand them deeply and proceed from the basis of understanding and that a weak approach is to search one’s memory for the ‘right’ procedure” (p. 90). This idea of a conceptual orientation to problem-solving is also reflected in Practice 6 of the Science Practices published within the Next Generation Science Standards (National Research Council, 2013), which states that students should be able to “construct an explanation that includes qualitative or quantitative relationships between variables that predict phenomena” (p. 60) and “apply scientific ideas to construct or revise an explanation” (p. 63).

A second indication of a conceptual orientation can be seen in students who can nimbly move between abstraction and context, as described in MP 2. In physics, students who demonstrate this ability can give a conceptual definition of the topic being studied. Another example would be students offering a real-life example, which was also mentioned in the CRAFTY document. In particular, the CRAFTY physicists called for students to get “…extensive practice [in] writing reasons for their answers, [in]communicating their thoughts on procedures, [in] solving real problems (where the path to the answer is not known by the student at the beginning), [in] applying their knowledge in a context meaningful to them, [and in] making connections to other domains of their knowledge” (Cummings & Emery, 2004, p. 120). Note that there is a distinction between identifying a conceptual orientation toward mathematics (or physics) and a conceptual understanding of mathematics (or physics). The former is based on listening to intention and process, while the latter is focused on appropriate concept application. Both conceptual orientation and conceptual understanding involve moving between abstract and context. (The physics instructors’ view of conceptual understanding is discussed in the section on obtaining a correct solution.)

A third indication of a conceptual orientation might be gleaned by examining what Odden and Russ (2018) define as the “resolution” step in the sensemaking epistemic game. This occurs when a group of problem solvers build an explanation that resolves an inconsistency among ideas. Because we were not privy to the sensemaking discussions of our student presenters, we cannot determine if the groups were engaging in the full sensemaking process. However, we can assume that if the presenters explained how the answer made sense within the context of the problem, they may have been representing this resolution step of the sensemaking epistemic game.

In summary, identifying a conceptual orientation is challenging, and there is no one way to identify someone’s view of a process based on a short video. Given that, we attempted to operationalize the construct as broadly as possible by looking for any indication that students attempted to do one or more of the following: (a) describe their mathematical steps using meaningful referents instead of speaking only in procedural terms, (b) give a conceptual definition of a topic, (c) offer a real-life example, or (d) explain the answer within the context of the problem.

Consideration 5: Using Appropriate Terminology

According to MP 6, “[m]athematically proficient students try to communicate precisely to others. They try to use clear definitions in discussion with others and in their own reasoning. They state the meaning of the symbols they choose. They are careful about specifying units of measure, and labeling axes to clarify the correspondence with quantities in a problem” (Common Core Standards Initiative, 2010). Educational researchers often emphasize that engagement in the practice of trying out new terms is critical for learning when and how use of the terms is most appropriate (cf. Sfard, 2008).

A second aspect of this consideration is that communication supports the listener as well. Students watching quality educative videos may note that presenters model the use of appropriate physics language. In contrast, videos posted by those who want to help people just “get the answer” offer explanations using pronouns—for example, “it moves up,” “this over this,” or “move this guy over here”—when solving equations.

Consideration 6: Conducting a Unit Analysis

As noted above, specifying units is mentioned in MP 6. However, we felt that this practice, which is critical in physics problem-solving, should be observed and counted explicitly. Considering units is useful when setting up a problem because it forms an expectation for the goal of the task and is important in assessing the reasonableness of a derived solution. When students view discussion of units as a means to make sense of quantities as opposed to a necessary action of “tagging on labels” at the end of their answers, they begin to think about how the units can be used to determine if their final answers make sense. Units are mentioned in MP 2, which states, “Quantitative reasoning entails habits of creating a coherent representation of the problem at hand; considering the units involved; attending to the meaning of quantities, not just how to compute them; and knowing and flexibly using different properties of operations and objects” (Common Core Standards Initiative, 2010). This idea of unit analysis also appears in the National Research Council’s description of how a student’s performance for evaluation should include the criterion that “[m]easurements are reasonably accurate and include correct units” (National Research Council, 1996, p. 43). Use of unit analysis can foster the vision that Wieman and Perkins (2005) describe when they encourage pedagogy in which students engage in thinking like a practicing physicist.

Consideration 7: Obtaining a Correct Answer

The preceding six considerations are not mutually exclusive, nor are they necessary or sufficient to compute a correct answer. For example, correct answers could be obtained by simply following a template. However, over the long run, these considerations clearly characterize more successful students who can apply prior learning to more complex applications in the future.

When examining videos that gave an incorrect answer, we found it useful to distinguish between errors in conceptual understanding (of either physics or mathematics) and other errors. From either point of view, we were particularly interested in the types of errors. In the case of mathematics, conceptual errors include incorrect use or interpretation of functions, while computational errors might involve incorrect multiplication or even using an incorrect unit. In the case of physics, Burkholder et al. (2021) created five categories of errors beginning with conceptual errors (e.g., not understanding how a circuit element works) and then further breaking down algorithmic calculus errors, geometric reasoning errors, vector notation errors, and miscellaneous other errors.

Summary of Framework

Table 1 summarizes the considerations for examining the types of mathematical skills students bring to and use in the physics classroom.

Table 1 Analytic framework for evaluating student videos
Full size table

This framework was used to examine the general question of which practices were observed in the students’ videos. While we acknowledge that these videos are not a perfect representation of how students engage in problem-solving—in fact it could be argued that they are a “best case” example—we found the framework as applied to the videos to be a useful method for bridging discussions between the math and physics faculty and for answering the question of what math skills and habits of mind the students use in physics.

Methods

The methods used in this examination were post hoc; the instructor had implemented this assignment prior to talking with the mathematics education coauthors. Hence, the analysis focuses solely on the video data that was collected after the course had ended.

Setting

The setting for this study was a calculus-based introductory physics class taught at a large university in the southwestern part of the USA. The class, which enrolled 523 students, was taught in 50-min lectures held three times per week. A lab class was associated with the lecture, but students were not required to take the lecture and lab course simultaneously; and many did not.

Learning Glass Technology

To facilitate the teaching of a large-lecture class, the instructor for the class had created an innovative technology called the Learning Glass™ that enables an instructor at the front of an auditorium to write on a 32-inch by 57-inch glass screen that projects his expressions, gestures, and inscriptions onto a larger screen as he writes in real time (Anderson et al., 2018). During lectures, the image is projected onto two of three huge screens mounted at the front of the auditorium, while clicker questions or other computer-based physics simulations are shown on the third screen. This setup is a vast improvement from previous arrangements that do not enable students to see an instructor’s face or gestures—such as writing on a whiteboard (which is difficult for students to read) or on an overhead projector (which provides very little writing space).

The students in the class also used the Learning Glass system to create their videos (see Fig. 1). This practice aligns with the group-wise benefits Liljedahl (2021) described regarding the use of vertical nonpermanent surfaces. To ease scheduling for the recordings, each student group was asked to sign up for a time to meet in a Learning Glass studio housed in the digital humanities section of the campus library. The studio contains a 4-foot by 5-foot screen that provided sufficient space to solve entire problems without having to erase. This setup also contains a “one-button recording” feature that enabled students to save their recorded videos onto a USB stick and subsequently upload them to a secure website visible only to the class (and to the research team).

Fig. 1
figure 1

Professor Anderson teaching with the Learning Glass technology

Full size image

The Assignment

Data were collected during the spring semester of 2019, which was the first time the assignment was given. The goal was to have each student work with two other peers to create one video that explained the solution to a problem related to a current physics topic that was covered in the class. The instructor formed random groups of three students, resulting in 172 groups. Each week, 17 groups posted their videos.

It is important to note that each group member was given full homework credit for completing the assignment, but students did not receive a specific grade based on content. All students were asked to view others’ videos and leave comments and critiques in the appropriate discussion thread (which was located on the class learning management system, accessible only to students in the class). Students were given two incentives for watching at least two videos each week: (a) a maximum of two homework points per week for posting a critique of at least two videos, and (b) the assurance that some of the problems assigned would appear on upcoming tests.

Analysis

The two mathematics educators began by watching several videos together to determine what variables should be coded. The coding followed the general outline of a thematic analysis (Peel, 2020). Our final list of variables included length of video, correctness of video, gender of speakers, number of speakers, use of diagrams, use of language (e.g., precision versus using articles such as “it”), and the presence or absence of introductory or concluding remarks indicative of sensemaking (i.e., what a reasonable answer might look like, whether the final answer was reasonable, etc.). Once an exhaustive list of variables was created, the remaining videos were scored by one rater. Consistency was assured by having a second rater discuss any videos that were in question along with a subset of randomly chosen videos coded by the first author until agreement was reached.

The second step in the thematic analysis involved creating the framework as described in the literature review. We knew the general themes we wanted to observe, but some variables, such as gender of presenters and length of video, were ultimately not used. Other variables were included to develop the most complicated theme—that of conceptual orientation (including the idea of sensemaking). We modified our classification system until we achieved what, in our opinion, was the most comprehensive approach to framing the themes without attributing any erroneous intentions to the speakers. For example, the original coding involved a three-level scoring system wherein variables were scored as “not visible,” “visible,” or “partially visible” (an example of the last would be a mention of units, but not a consistent focus on their use). After discussions to obtain inter-rater reliability, we decided to adopt a two-tiered system so that all scores would be counted as either “observed consistently” or “not observed consistently.”

Results

The goal of our analysis was to determine which aspects of the effective practices framework were most and least often observed in the videos created. Results of the video coding are shown in Fig. 2. In the following sections, we elaborate on each of the considerations from most to least consistently observed in the data.

Fig. 2
figure 2

Distribution of practice engagement across videos. Note. This stacked bar graph depicts the percentage of groups observed engaging in each component of the considerations, ordered from most to least frequently observed in the data

Full size image

Practices That Were Most Prevalent in Videos

  1. 1.

    Presenting a correct answer

The most frequently observed result was that student groups presented a correct answer. Videos with correct answers and videos with incorrect answers were evenly distributed across the semester and across all of the various physics problems modeled, which indicates that there was no one question that was disproportionately challenging.

Table 2 contains a summary of the types of errors observed in the 36 videos that presented an incorrect solution (21% of all videos). Seventeen of the videos included mathematical errors and 19 included physics errors. Errors within these groups were further divided into conceptual or more calculational. Within the mathematical error group, 12 videos were considered having conceptual errors, such as difficulty interpreting the result of the arctan(x) function correctly, canceling out a variable in a fraction that ended up changing the value of the ratio, or using degrees in an equation that called for radians. The remaining calculational errors related to unit conversions, such as using the wrong constant to convert kcal to joules or not converting rpm to m/s, which resulted in incorrect values used in center of mass equations. These unit errors were explicitly identified because they are tangible and relatively straightforward errors that instructors can focus on in their teaching.

Table 2 Error types in videos with incorrect final answers
Full size table

Within the physics error group, 19 videos demonstrated errors related to conceptual understanding such as not using the sum of forces in either the x or y direction, assuming that an angle was 90° because that is what it was on the free body diagram, or not understanding that the angle of a frictionless incline does not affect final speed. The remaining eight videos were coded as demonstrating a “variable” mistake. While this type of error could be related to a lack of conceptual understanding, it was given its own special code because, like the mathematical unit errors, it is something that physics instructors can become more attuned to addressing in class discussions. Examples of variable errors include using mass instead of distance when solving for velocity or conflating weight and mass.

  1. 2.

    Using precise terminology

Presenters who used appropriate terminology and precise language consistently throughout their videos avoided the use of pronouns such as “this quantity” or “x,” referring instead, for example, to “the constant for gravity.” One excerpt from a group that tried to use precise terminology is shown in Fig. 3. The students are seen explicitly describing how they labeled their axes to call attention to the vertical and horizontal components of velocity.

Fig. 3
figure 3

Example of good use of terminology

Full size image

In contrast to the above example, some videos did not show students engaging in precise communication efforts. One indication of a lack of precision was the overuse of pronouns. For example, one presenter concluded his computations by stating, “We’re trying to find out the angle of it. We’ll just split that by half. We know that gravity is 9.8, so we just times that together and get 29.4.” The pronouns that the presenter used (and the fact that he did not include any units or a negative sign to indicate the direction of gravity) may indicate either that he was not comfortable referring to the values as quantities or that perhaps he felt as though the viewers already understood the values and was trying to get through the explanation quickly. In any case, no member of this particular team told the presenter that he needed to use more precise language when referring to various quantities, their measures, and their units. In general, however, the large majority (78%) of the groups did seem to demonstrate the practice of fully defining quantities and being able to move fluently between the concrete context of the problem and the abstract mathematics needed to solve it.

  1. 3.

    Setting up the problem

The third most frequently observed practice in which students participated involved taking care to set up a problem. We found several good examples of presenters laying out the givens as physical quantities (i.e., quantities with numeric values) and situating them within the problem using a drawing. The example in Fig. 4 shows a student who began by clearly defining all of the values and formulas needed along with appropriate units. He also leveraged the utility of the Learning Glass by writing all of the physical quantities in one color and then filling in their given values using a different color while he spoke.

  1. 4.

    Formula selection and description

Fig. 4
figure 4

Example of good use of color, variable description, and formula selection

Full size image

The goal of this consideration was to determine if students were able to think through and explain why they modeled a problem as they did. In particular, we were looking to determine how they selected the formulas they chose by observing the following: if they showed the derivation of their modeling, if they explained their choice, or if they did not explain their choice but instead appeared to rely on the current chapter of study. As can be seen in Table 3, 56% of the students featured in a video showing a correct answer presented an explanation for why they selected a particular formula, which contrasts with only 22% of the videos that showed an incorrect answer including such an explanation. This observation was particularly differentiated when listening to how students described ratios used in functions. Those who understood proportional relationships were able to better explain why, for example, division or unit conversions were needed.

Table 3 Classification of ways in which students described their choice of functions for their model
Full size table

Practices That Were Least Prevalent in Videos

The three considerations that were observed the least often among the videos were (a) describing the variables and their associated values, (b) assuming a conceptual orientation in describing the solution process, and (c) conducting a unit analysis.

  1. 5.

    Describing variables and values

The students in all of the videos demonstrated the general pattern of creating a model, illustrating with a diagram, and then substituting values for the variables used in the model. The goal of the fifth consideration was to determine the degree to which students were able to describe and explain the variables and values used in their models. In total, about half of the video groups did include some type of explanation regarding the variables and values they were using in their models. Examples include distinguishing between the values of weight and mass and using correct values and units for quantities such as the radius of the Earth. Table 4 displays the percentage of groups who explained their use of variables disaggregated by videos with correct answers versus those with incorrect answers.

Table 4 Classification of which students described their variable choices
Full size table

It is noteworthy that while 57% of the groups that presented a correct answer explained the values that they were plugging in, only 22% of the student groups that presented incorrect answers attempted to do so. For example, one group who presented an incorrect answer used Kepler’s Third Law but plugged in incorrect values for the variables. Thus, it seems that the practice of identifying variables and explaining their role within a model might be helpful in enhancing students’ problem-solving efforts.

  1. 6.

    Assuming a conceptual orientation

Given that this consideration is multifaceted, we continually added and combined observations to create a thematic analysis. The following four themes emerged as components of a conceptual orientation in the video presentations:

  1. (a)

    Presenters gave a conceptual definition of the topic. This meant going beyond reciting a definition from class notes or the textbook. Examples included rephrasing the question or referencing prior knowledge and conceptions. This was evident in 43% of the videos overall (in 49% of the videos that showed a correct answer and in 22% of the videos that had an incorrect answer).

  2. (b)

    Presenters used conceptual language to explain the steps of the problem. As noted in the work of Thompson et al. (1994), this could include referring to the algebraic terms by their real-world referents—for instance, “the mass of the rock” rather than just “m.” Students could also draw connections between the question assigned and a real-life scenario that seems to model the same physics that are occurring. Use of conceptual language was evident in 49% of the videos overall (in 57% of the videos that showed a correct answer and in 19% of the videos that had an incorrect answer).

  3. (c)

    Presenters discussed the reasonableness of their answers. This was evidenced in only four videos.

  4. (d)

    Presenters framed their answer in terms of the problem’s context (i.e., by placing their answer in a complete sentence that explained how their answer related to the solution of the problem). Even if students had incorrectly answered the question, it was still important to recognize their efforts in working toward a conceptual understanding of the question’s topic. This framing was seen in 45% of all videos (in 49% of those showing a correct answer and in 31% of those showing an incorrect answer).

In keeping with the binary observed/not observed categorization, we scored one point for the observation of any of the above four themes and determined that a group was demonstrating a conceptual orientation if they demonstrated two or more themes (components). The results of this coding are shown in Fig. 5.

Fig. 5
figure 5

Percentage of conceptual orientation across videos. Note. This figure demonstrates that the groups presenting a correct answer displayed more components comprising a conceptual orientation than those who presented an incorrect answer

Full size image

The presenter shown in Fig. 4 provides a vivid example of a speaker assuming a conceptual orientation. He explained the topic in context and discussed not only the givens and the formula to be used but also why the formula was appropriate, stating, “The force of gravity and the normal force must add together to equal zero.”

In contrast to those that demonstrated several components of a conceptual orientation, we found that many videos either lacked any effort to describe the work conceptually or else attempted to label a variable but still described the work in calculational rather than conceptual terms. Consider the example in Fig. 6.

Fig. 6
figure 6

Student engaging in calculational rather than conceptual explanation

Full size image

The first part of this excerpt indicates that the student is able to move between the context and the mathematical model (e.g., “1.5 m per second…is the speed of the current”). The latter part, during which the speaker describes “finding velocity,” focuses much more on discussing the calculations than on explaining why the calculations were done. In particular, she does not clarify why she is taking the “square root of 3.0 minus 1.5 m per second.” This is also an example of a student who does not appear to be thinking in terms of ratios as comparisons of lengths. For this reason, we might conjecture that a viewer who did not understand the overall point of the problem would most likely be unable to understand or replicate the solution process. This group earned only one out of four points because while they did explain the context for each of the variables, they did not explain why the calculations were being performed or the context or reasonability of their final answer.

  1. 7.

    Conducting unit analysis

The consideration that was observed least frequently among the videos involved focusing on the units of the variables throughout the presentation. This practice was seen fully demonstrated in only 18% of the videos, but it was partially realized in another 64%. The example shown in Fig. 7 earned full points for effective unit analysis. The student began by describing the procedure, but then used a unit-canceling practice as a means to verify that the operations made sense and to explain how the resultant quantity could be measured in newtons.

Fig. 7
figure 7

Student engaging in unit analysis. Note. This image contains a text overlay because the writing was difficult to read given the quality of the image

Full size image

In contrast to this example of a thoughtful inclusion of unit analysis, presenters in 18% of the groups did not write or speak about the units when announcing the final answer. For example, one speaker simply concluded, “So the square root of 39.2 equals velocity.” This conclusion indicates that the student did not attempt to equate velocity as measured in meters per second and would, therefore, be less likely to link the operation (taking the square root) as a means for reaching the correct unit. To him, 1/39.2 might have been equally acceptable, even though velocity is not measured as seconds per meter.

In summary, most videos demonstrated appropriate use of terminology (78%) and set up variables and models in coherent ways (66%). However, the majority of groups did not appear to leverage the value of units or assume a conceptual orientation. These differences become more pronounced when comparing the practices of groups presenting correct versus incorrect answers. While 54% of the videos showing a correct answer displayed two or more of the components of a conceptual orientation, only 19% of the videos showing incorrect answers demonstrated two or more conceptual components.

Discussion

The goal of this study has been to provide grounded examples of sensemaking experiences that students are expected to engage in once they enroll in science courses for which precalculus and calculus are required. In our work, we aimed to identify specific tools and habits of mind, ostensibly learned in mathematics courses, that were observed (or found lacking) in students enrolled in a physics course. The study was based on an analysis of 172 videos that students created to solve assigned problems. The results and analytic framework were designed to complement the work of the CRAFTY researchers with the aim of further defining the ways that mathematics instructors can enhance the skills that their students need and are expected to use in subsequent major classes.

The study is complementary to other CRAFTY research in that we shifted from asking discipline partners what math skills and practices they expect their incoming students to have (cf. Hofrenning et al., 2020) to observing what mathematical skills students demonstrated in self-created videos in a physics class. Because these were rehearsed, prepared video presentations rather than impromptu responses to a query, the students had plenty of time to check their answers and eliminate mistakes. Thus, it is not surprising that only 17 videos (10% of all submissions) contained a mathematical error. Based on this finding, one might conclude that the students’ mathematical backgrounds served them well and that no changes to preliminary mathematics curricula are needed. However, one must also consider the finding that less than half of the video presenters discussed the variables and values they were using, paid attention to unit analysis during or after the problem-solving process, or demonstrated a conceptual orientation toward problem-solving. These latter results align with the distribution of grades within the class (average grade of B-) and highlight where more support might help to prepare students for this class as well as for future classes where students’ ability to go beyond template-based problem-solving into sensemaking will be important.

We take these findings to indicate that learning mathematics and physics can be improved by focusing on the process and communication of problem-solving (including back-and-forth discussions of variables, values, and model assumptions) rather than on the simple computation of an answer. We operationalized skill achievement as the ways in which students engaged in various problem-solving practices and identified how various practices, as identified by partner disciplines and in mathematics and science education research, could be emphasized to support deeper learning of the material.

Practices with Strong Engagement

Results revealed that 79% of the video submissions featured correct answers. Following Greene and Crespi (2012), we predicted that students would dedicate great effort to solving the assigned problems because they knew that their peers were incentivized to watch and critique their videos. When asked in an end-of-semester survey if they had consulted the instructor or teaching assistant to verify their solution approaches, only 6% of the students said that they had. Consistent with this seeming lack of concern about correctness, only 26% of the respondents strongly agreed that they should have been provided the answer to verify that their solution was correct. The question of whether incorrect answers can be useful may be informed by literature on students’ conceptions that are not congruent with the consensus of the scientific community’s current understanding (e.g., Cook et al., 2014; Tippett, 2010). Our reading of this literature suggests that having students create videos in which they describe their first attempts and then how their thinking changed (after being given the correct answer) could be even more educative for both the creators and the viewers than creating videos without knowing if the answers shown are correct or not. Without such commentary, the videos showing incorrect answers were counterproductive.

The second and third most frequently observed practices were the use of precise language when describing or referring to variable quantities and the development of a strong problem setup, respectively. With regard to precision, 78% of the speakers avoided using pronouns such as “this” or “the value” and chose instead to use specific terms such as “the constant of gravity.” These groups aligned with the groups that produced videos featuring well-described problem setups (including a discussion of the question and the drawing and labeling of a diagram model).

Practices with Weak Engagement

One of the most revealing discoveries of the analysis was the limited use of conceptual explanations. Results indicate that only 75 of 172 videos (44%) provided conceptual explanations; only 85 (50%) explained how the values in the problem related to the variables in the formulas used; and only 30 (17%) used units throughout their descriptions to explain the problem’s relation to the formula and the reasonableness of the answer.

The fact that many students in the study produced calculation-based videos must itself be placed in context. First, they were given no particular directions or criteria for their presentations (which makes the argument stronger that what we were seeing does more accurately portray their actual problem-solving routines); second, they were taking their first physics class; and third, most of the free, online videos that students access on YouTube are often calculation-oriented (cf. Genota, 2018). However, this finding is important because the vast majority of video groups who presented an incorrect answer portrayed a calculational rather than conceptual orientation.

Implications for Teaching Mathematics

The study suggests several ways in which assignments such as creating videos can be used to improve student learning. These include having instructors serve as role models, monitoring students’ ways of engaging in their problem-solving practices, and considering various student group compositions. Each of these is briefly discussed below.

Instructors as Role Models

When students see instructors consistently modeling good problem-solving practices, they are more likely to vicariously engage in them. For instance, if instructors model proper use of terminology, students, in turn, will appropriate the meanings of the terms and how they are used in the wider scientific community. We suggest that teachers (1) model the use of appropriate terminology when referring to formulas, variables, or values; (2) model the use of unit analysis both during and at the conclusion of a derivation such that if a step is skipped or an answer does not reflect the current units, then an error flag is built into their practices; and (3) model the practice of assessing the correctness of an answer both before the solution process begins (i.e., “a reasonable answer would have to be between 90° and 180°”) and at the end (i.e., “this makes sense because the units are correct and the answer is within the range we anticipated”).

Monitoring Students’ Engagement

As Wang (2020) concluded, having teachers listen to students’ language helps build a community with shared meanings. Having video records from an activity such as ours can support community building if the material is used by instructors to illuminate the practices that are taken as shared as well as those that have yet to be negotiated more widely among the larger group of students. These records can also enable teachers to see how their students are engaging in problem-solving practices. For example, teachers could encourage active engagement and conceptual—rather than procedural—language by highlighting videos that feature dialogic formats in which two presenters talk about a solution.

Unsurprisingly, the majority of videos (63%) in this preliminary study featured only one presenter, perhaps because this format is consistent with the instructional videos that students consult on the internet and the way they are taught in a large-lecture class. We were encouraged to see that 19% of the videos did feature two presenters, but most of these were not considered to be dialogic in the sense that advocates such as Lobato et al. (2019) promote, wherein the two talk together and, ideally, debate different paths forward. Instead, these teams featured either one speaker at a time or one person drawing and writing while the other spoke.

Supporting Diversity

One of our initial goals for this project was to leverage the transparency of the Learning Glass to implicitly emphasize that all students can be featured as strong physics problem solvers. We hoped that this would support feelings of inclusion within the class community. We were pleased to see that 23% of the videos included women presenters (15% as sole presenters, 8% as part of a pair), which was proportional to the number of women enrolled in the course. One teaching implication from this result is that students may feel more comfortable presenting with their peers when they are allowed to choose their own video groups rather than being randomly assigned to other students.

Limitations and Future Direction for Research

This report does not provide any data linking students’ engagement in the practices discussed to course or test success. In fact, such a conclusion was beyond the scope of this paper for several reasons. First, although we do have data relating to target test items that aligned with the videos produced, the results showed that less than half of the groups all answered their target item in the same way. That is, even though three people worked on one problem together to create each video, at least one of the group members answered the exact same question template differently than how it was answered on the video. Thus, using group video scores as an independent variable would not have been a valid measure of participants’ engagement. Therefore, we rely on the large body of research supporting the practices to maintain the effectiveness of the considerations.

A second reason that we cannot claim causality is that this post hoc analysis was not designed as a randomly controlled trial. One could argue that the students who produced the highest-rated videos were the strongest in the class and therefore conclude that their success was tied to their expertise rather than the quality of the video produced. As a modification to this research, this point needs to be tested by rerunning the experiment while sharing the scoring rubric as described above and using students’ prior grades or other information as covariates. If the rubric motivates more students to engage in the practices that the strongest students demonstrated, then the point will be more strongly supported.

Conclusions

The swift proliferation of artificial intelligence and other online sources has catalyzed STEM faculty to rethink how students are prepared to make sense of scientific data. While many authors have called for high-level changes to mathematics instruction, our goal has been to provide a grounded example of how students are enacting various sensemaking practices in a physics course. The goal is to provide concrete examples of how instructors can encourage the steps that transcend simple problem solving. These include encouraging students to use unit analyses and to develop habits of mind that look at models as two sides of a coin: the mathematical illustration of relationships among variables and the physical manifestations of these relationships.

Although the context for this study was a physics classroom, we believe that the framework for analyzing student-generated videos could apply to just about any STEM discipline for which mathematics is a prerequisite. By framing learning as engaging in communally negotiated practices, we argue that videos, which serve as records of practice (cf. Ball et al., 2014), are an excellent vehicle for allowing outsiders to identify the types of practices student groups perceive to be contextually appropriate. Moreover, they also provide feedback to the instructor regarding the types of practices that are negotiated in the classroom.