Drawing on Experience: How Domain Knowledge Is Reflected in Sketches of Scientific Structures and Processes
- First Online:
- Cite this article as:
- Jee, B.D., Gentner, D., Uttal, D.H. et al. Res Sci Educ (2014) 44: 859. doi:10.1007/s11165-014-9405-2
- 619 Downloads
Capturing the nature of students’ mental representations and how they change with learning is a primary goal in science education research. This can be challenging in spatially intense domains, such as geoscience, architecture, and engineering. In this research, we test whether sketching can be used to gauge level of expertise in geoscience, using new technology designed to facilitate this process. We asked participants with differing levels of geoscience experience to copy two kinds of geoscience images—photographs of rock formations and causal diagrams. To permit studying the process of sketching as well as the structure and content of the sketches, we used the CogSketch system (Forbus et al. 2011, Topics in Cognitive Science 3:648–666) to record the time course of sketching and analyze the sketches themselves. Relative to novices, geoscience students included more geological structures and relational symbols in their sketches of geoscience materials and were more likely to construct their sketches in a sequence consistent with the order of causal events. These differences appear to stem from differences in domain knowledge, because they did not show up in participants’ sketches of materials from other fields. The findings and methods of this research suggest new ways to promote and assess science learning, which are well suited to the visual–spatial demands of many domains.
KeywordsSpatial knowledge Causal knowledge Science education Sketching Sketching software STEM
A crucial concern for researchers and educators alike is how to study the development of people’s mental representations as they acquire domain knowledge. This challenge can be particularly great for domains that are intensely spatial, such as geoscience, engineering, and other STEM disciplines. Much of the content in these domains is expressed through diagrams, maps, and other visual representations. Traditional written assessments, including multiple-choice tests, are of limited value for highly spatial topics because of the difficulty of conveying complex spatial relations through speech or writing alone (e.g., Larkin and Simon 1987). Sketches, by contrast, readily capture spatial information (Goel 1995; Heiser and Tversky 2002; Suwa and Tversky 1997; Vosniadou and Brewer 1992). Sketches can also reveal students’ conceptual representations, a point that is amplified below. All this suggests that sketching is crucial to capturing and assessing students’ knowledge in STEM disciplines, such as geoscience.
Indeed, in a recent Science article, Ainsworth et al. (2011) go further, stating that sketching is vital to science learning. They maintain that sketching helps students learn to represent and reason about scientific phenomena, and that “Unlike other constructive strategies, such as writing summaries or providing oral self-explanations, visual representations have distinct attributes that match the visual-spatial demands of much of science learning” (p. 1097). Based on a survey of the literature on drawing during learning, Van Meter and Garner (2005) argued that learner-generated drawing shows particularly strong effects in higher-order assessments, such as transfer. Schwanborn et al. (2010), for example, provide evidence that drawing during learning leads 9th-grade students to improve on tests of transfer in chemistry. Recent work by Gagnier et al. (2014, accepted for publication) suggests that sketching improves a learner’s comprehension of diagrams, including inferences about the internal structure of a 3D object. These findings tell us that sketching can be important in learning and assessment, but they do not focus on what properties of sketches might be measured to aid in assessment. This paper provides evidence relevant to that question.
Ainsworth et al. (2011) also point out that sketching is currently underutilized in science instruction. It is not hard to see why—grading sketches from a large class can be extremely time-consuming. It is also a challenge to score drawings reliably, because they are often quite variable (e.g., White and Gunstone 1992). This not only places a burden on the instructor but also results in delayed (and therefore less useful) feedback to the student (Dihoff et al. 2004; Kulik and Kulik 1988). Recent innovations in computer-based sketching have made the task of collecting and assessing sketches substantially easier (de Silva et al. 2007; Do 2005; Forbus et al. 2011; Pargas et al. 2007; Valentine et al. 2012). These developments could have great impact on science education research and practice. In this paper, we describe findings from research using CogSketch, a computer-based sketching platform (Forbus et al. 2008, 2011), to investigate the effects of spatial scientific knowledge on sketching. Our focus here is on geoscience—an intensely spatial domain in which visualization, modeling, and sketching are fundamental to instruction (Jee et al. 2010; Kastens et al. 2009; Libarkin and Brick 2002; Liben and Titus 2012; Piburn et al. 2011). However, we believe our findings are relevant to spatially intensive fields in general.
This research has two linked goals. First, we aim to discover whether and how geoscience understanding is reflected in sketches and in the process through which sketches are constructed. This is especially relevant to theories that aim to capture and explain changes in knowledge that result from science learning. To this end, we elicited students’ sketches of two distinct kinds of geoscience materials—photographs of geological formations, and causal diagrams. We collected sketches from both novices (psychology students) and geoscience students. Our second goal is methodological: to explore the utility of CogSketch software (Forbus et al. 2008, 2011) for collecting and analyzing people’s sketches and the time course of their construction. These two goals are interconnected. At present, evaluating sketches is very laborious and often inaccurate, and thus their use in classroom evaluation as well as in research is limited. If we find that certain properties of a sketch reveal a person’s level of domain knowledge, then having a less effortful, more reliable, and more sensitive way to analyze sketches will be of great value both to instructors and to researchers in science learning. In this paper, our focus is on possible differences between novice and advanced geoscience students that may be revealed through analyzing their sketches with the CogSketch system. We will discuss CogSketch’s methodological utility more broadly in the “General Discussion.”
Spatial and Causal Knowledge in Geoscience
How might the content and active construction of a sketch change as students gain knowledge of geologic structures and the causal forces underlying them? We consider this question with regard to two common types of images that geoscience students encounter: photographs of geological structures and causal diagrams of geologic processes.
Sketching from Photographs
An important aspect of geological expertise is the ability to accurately characterize geological formations, such as outcrops of rock, and to be able to sketch such formations (either from photographs or from real-world scenes). Therefore, in experiment 1, we asked our participants to sketch rock outcrops. To ensure that participants had adequate opportunity to view the materials, the photographs remained present until they finished sketching.
What differences might we expect between novices and geoscience students? A number of studies suggest that expertise involves conceptualizing domain-related objects in terms of their deeper causal and goal-relevant structure, deemphasizing superficial details (e.g., Chi et al. 1981; Gentner and Rattermann 1991; Hay et al. 2013; Jee and Wiley 2007; Lowe 1993). As students acquire domain knowledge, their drawings emphasize domain-relevant causal structure (Gobert 2000; Gobert and Clement 1999). Kindfield (1992), for example, found that more expert biology students tended to create more schematized representations of the entities involved in mitosis. Sketches by less expert participants were more detailed, often including irrelevant physical dimensions and omitting relevant structural characteristics. This result is consistent with recent research by Hay et al. (2013), which explored expertise-related differences in drawings of brain cells. Experts in neuroscience created schematized sketches, biased toward the conventions and features relevant to their subfield. Undergraduate neuroscience students made drawings that were similar to textbook images of neurons. Similar patterns have been found in studies that ask participants to sort visual depictions. For example, in a well-known study, Chi et al. (1981) found that physics experts sorted diagrams of physics problems on the basis of common underlying principles, whereas novices relied on superficial commonalities, such as the presence of common objects such as ramps and pulleys. Likewise, when asked to group together elements of a diagram, experts tend to group together objects that are causally related, even if they are visually dissimilar and spatially distant (Lowe 1993). Based on findings from expertise studies, we could expect people with a background in geoscience to emphasize features related to domain-relevant spatial structures, such as faults and folds, in their sketches of geological photos. By contrast, novices are likely to include geologically irrelevant but perceptually salient details, such as vegetation.
Sketching Diagrams of Causal Processes
We hypothesized that as students gain knowledge of spatially complex scientific material, such as plate tectonics, their diagrammatic sketches will become more focused on domain-relevant structures and more similar to standard geoscience depictions (Gobert and Clement 1999). The inclusion of relational and causal symbols in a sketch (e.g., directed arrows and other connectors) also reflects a person’s causal understanding of the system they are sketching. When people are provided with a functional description of a system, they tend to include more of these symbols in their drawings (Heiser and Tversky 2006). As functional and causal knowledge increases with expertise (e.g., Chi et al. 1988; Hmelo-Silver and Pfeffer 2004; Proffitt et al. 2000), we predict that knowledgeable participants will include more relational symbols in their sketches than novices will.
An advantage of using a computer-based sketching program is that it can record the real-time process by which a sketch is drawn. Such timing information could reveal knowledge of key causal and spatial relations (Chase and Simon 1973; Gobet et al. 2001; Moss et al. 2006; Taylor and Tversky 1992; Tversky 2002). Taylor and Tversky (1992) found that people tend to sketch large features of a scene, such as mountains, before smaller-scale features, such as houses, suggesting that people organize their knowledge of the scene in terms of a hierarchical structure. A different kind of organization may be revealed when people sketch diagrams that depict a set of interrelated causal processes. Moss et al. (2006) found that senior engineering students recreated a diagram of a device starting with its input and moving to its output. Knowledge of how the device works influenced how they assembled it later. If people with geoscience knowledge actively interpret a causal diagram in terms of the sequence of causal events, they may construct their sketch in line with this sequence. They may draw an antecedent causal event before the resulting event, and a primary causal event before secondary causes.
Overview of the Experiments
In summary, our goals are, first, to discover whether and how patterns of sketching reveal differences in geoscience expertise, and, second, to test the efficacy of a computer-based sketching platform, CogSketch, for collecting and analyzing such sketches.
To ensure that the novices had access to the same visual information as the geoscience students, including causal symbols that could support diagram interpretation, each source image was available for the entire time that the participant sketched. The participants’ task was simply to copy the image onto the computer screen. The question was whether differences in spatial and causal knowledge would affect the relational content and temporal construction of a sketch even in this copying task, where there was no inherent need to go beyond the information in the images. We tested this possibility across three experiments. Experiment 1 provides an initial test of the effects of geoscience knowledge on sketching. Experiments 2 and 3 further explore sketching of geological formations and causal diagrams, respectively, and explore whether differences between novices and geoscience students are specific to geoscience-related materials (suggesting an effect of domain-specific knowledge) or if they generalize to nongeoscience materials. In each experiment, we collected sketches on a tablet computer running CogSketch software, which tracks the spatial and temporal properties of the digital ink strokes that a person uses to create a sketch (using a stylus, mouse, or other input device). The software also records the conceptual labels and descriptions that the participant assigns to different parts within a sketch, as described below. By using Cogsketch as a research tool, we can examine not only the drawings and conceptual labels that comprise a sketch, but also the order in which different parts were created.
Participants with differing levels of geoscience experience used CogSketch to sketch two photographs containing geological formations, and a diagram of a geoscience-related causal process. The participants copied a photograph or diagram onto the screen of a tablet computer using a stylus. For the geologic formations, the main question was whether geoscience students included more features related to geologically relevant structures, such as rock layers and folds. For the causal diagram, the main questions were whether geoscience students included more causal and relational information in their sketches, and whether the ordering of their sketches reflects the sequence of causal events depicted in the diagram.
Twenty students from Northwestern University participated: ten geoscience students (six graduate students and four advanced undergraduates) and ten undergraduate psychology students who had no prior course experience in geoscience. The geoscience students received $12 for their participation, and the psychology students received course credit. All students participated individually.
We asked participants to sketch three geoscience-related images: two photographs of geological formations—one that showed a number of horizontal rock layers and one that showed folded rock—and one causal diagram illustrating subduction. The images are listed in the Appendix.
An HP tablet computer with a 12-in. screen and a stylus was used for data collection.
CogSketch is an open-domain, general-purpose sketch understanding system (see Forbus et al. 2011 for a more detailed description). CogSketch allows users to tell it what entities they are sketching via a specialized interface. Every sketch drawn in CogSketch is made up of glyphs. A glyph is a shape, an object, or a relationship that has been drawn by the user. The user presses a button to indicate that they are about to start drawing a glyph; they then draw as many ink strokes as they want, and end the glyph by pressing the button again1. As a separate step, users describe what they intend a glyph to represent by labeling it with either a concept from the knowledge base associated with CogSketch or with a word or phrase. Each glyph is represented internally as digital ink—essentially the lines drawn by the user when creating the glyph—and content—a conceptual entity representing what the glyph depicts. A video demonstration of CogSketch can be viewed at http://www.youtube.com/watch?v=o6f1udbIkrI.
CogSketch offers several features relevant to data gathering and interpretation. First, CogSketch records a time stamp along with every ink point (sampled at a rate of 4 points/s). These data can be exported to common formats, such as Microsoft Excel. Second, because the user has manually segmented the digital ink into glyphs, there is no ambiguity about which parts belong to which depicted entity. Third, there is no need to analyze video and audio transcripts to ascertain what labels should be used for the parts of a sketch, making CogSketch data simpler and cheaper to analyze.
The tablet computer was folded flat and placed on a desktop in front of the participant. The experimenter first gave the participant a brief tutorial and practice using the software and hardware, followed by an overview of the task and procedure. Participants were asked to try to copy the experimental images to the best of their ability, to use the glyph function in CogSketch to divide their sketch into the parts that seemed natural to them, and to label the parts as they saw fit, using words or phrases. They were given up to 3 min/sketch, with a warning given 1 min before the time limit. For the duration of each trial, a color printout of the image was presented next to the computer screen. The participant sketched directly on the computer screen using the stylus. The images were sketched in the same order for all participants: subduction 1, rock layers, and then folded rock. Each session took under 30 min to complete.
Two raters coded each sketch independently with high overall inter-rater reliability (Cohen’s Kappa = 0.89). Sketches were scored blind to any participant identification information. Coding disagreements were settled through discussion.
Geological Formation Sketches
Participants’ sketches were coded for the presence of the key features from the two photos. We computed the mean proportion of key features in each sketch. Overall, geoscience students’ sketches contained a higher proportion of the geological features (M = 0.78, SD = 0.26) than novices’ sketches (M = 0.49, SD = 0.19), t(18) = 2.95, p < .05, d = 1.27.
Causal Diagram Sketches
This analysis investigated whether the sketch displayed particular geologically significant events, and in what order these events were drawn. A geoscience expert identified a set of geologically significant events in the diagram—e.g., seafloor spreading and plate convergence—and specified their causal order. These events were conveyed by one or more directed arrows in the diagram, which were a subset of the total number of arrows in the diagram. The presence of these arrows in a sketch was taken as evidence that the participant represented the corresponding causal event. We scored the participants’ sketches for the presence of these causal events, and used the timing data recorded by CogSketch to determine the order in which they were sketched.
On average, the geoscience students (M = 0.72, SD = 0.23) included a higher proportion of the causal events than the novices (M = 0.15, SD = 0.32), t(18) = 4.45, p < .05, d = 2.05. As the novices rarely included causal events, we were unable to analyze the ordering of events in their sketches. For the geoscience student sketches, we computed the mean ordering for each of the four events, including only the sketches that contained at least two causal events. This left us with useable data from seven of the geoscience student sketches. Mean ordering for the geoscience students was compared with the ordering of the expert by computing Kendall’s coefficient of concordance (Kendall’s W). This statistic provides an index of the similarity between distinct orderings—ranging from 0 (minimum similarity) to 1 (maximum similarity)—and can be tested for statistical significance using a chi-square distribution (see Taylor and Tverky (1992) for a similar analysis). The value of Kendall’s W was .70, suggesting a relation between the mean3 geoscience student ordering and the expert ordering; however, this statistic did not approach significance, χ2(3) = 4.20, p = .24.
The results of experiment 1 support the hypothesis that differences in geoscience knowledge lead to systematically different patterns of sketching. The results also are encouraging for our goal of capturing differences in scientific knowledge using CogSketch. The results revealed that even in a simple copying task, geoscience students and novices differed both in terms of what information they included and the order in which they drew the elements in the sketch. When sketching photographs of geological formations, geoscience students were more likely than novices to include spatial features that reflected key geologic structures. When sketching the causal diagram, geoscience students emphasized causal relations and processes, and their sketches often reflected the causal ordering of geologic events. Novices sketched fewer relations overall and often omitted the events that a geoscience expert considered important.
The finding that geoscience students differ systematically from novices in sketching geoscience materials supports the hypothesis that sketching reveals domain understanding. The results are consistent with the previous findings that knowledgeable individuals emphasize domain-relevant structures in their sketches (Gobert 2000; Hay et al. 2013; Kindfield 1992). Yet, the generality of this result is unclear. We do not know whether geoscience students would show similar effects when drawing images from different domains, because only geoscience-related materials were included in experiment 1. If our findings are attributable to differences in domain knowledge—and not some other factor, such as drawing ability or spatial skill—then the between-group differences we observed in experiment 1 should not carry over to nongeoscience materials, even those that overlap in their features and causal complexity. This is a critical issue. Prior work has found that experts often rely on domain-general knowledge for thinking and reasoning (e.g., Schunn and Anderson 1999).
To determine whether the differences between geoscience students and novices are specific to geoscience, we asked participants in experiments 2 and 3 to sketch control images in addition to geoscience images. We chose spatially intricate pictures from domains unrelated to geoscience, such as a cross-section of a fruit and an artificial heart valve. The goal was to find images that were about equally familiar to both groups. For the control photographs, we sought images that contained similar features (folds, layers, etc.) to the geoscience images; for the control causal diagrams, we sought images similar in causal complexity. If geoscience students construct sketches differently to novices regardless of the domain, then we should find similar patterns for both the geoscience and nongeoscience materials. However, if geoscience students are utilizing domain-specific knowledge during sketching, then we should observe differences for the geoscience-related images but not for the control images.
A further goal of experiments 2 and 3 was to generalize our findings to a broader range of materials. Experiment 2 tested additional geoscience formations and experiment 3 tested additional causal diagrams. Both of these experiments included a greater number of images than did experiment 1 to gain experimental power. Finally, to explore how sketching effects relate to prior knowledge of the particular topics shown in the images, at the end of the study we asked people to rate their prior knowledge of the content of each image.
This experiment focused on sketches of geological formations. As in experiment 1, participants were given the simple task of copying photographs: three photographs of geological formations (not used in experiment 1) and three control (non-geological) photographs. The geological and control photos contained similar surface features, such as curves, layers, and complex lines, and were comparably intricate. If geoscience students are simply better at detecting and depicting the spatial features of objects in a sketch, then they should sketch both the geoscience images and the control images differently than novices do. However, if the differences between novices and geoscience students reflect domain understanding, then differences should emerge only for the geoscience-related sketches. Furthermore, such expertise-related differences should be most pronounced for images on which the geoscience students rate themselves as having greater prior knowledge.
Sixteen participants from Northwestern University took part: eight geoscience students (five graduate students and there advanced undergraduates) who were paid for participating, and eight undergraduates with no prior course experience in geosciences, who received course credit in an introductory psychology course. None of the participants took part in experiment 1. All participated individually.
There were three geoscience images and three nongeoscience images, all listed in the Appendix. The geoscience images were selected to be novel, unfamiliar examples of the kinds of structures that geoscience students would be likely to encounter in a geoscience course. Sketches were again collected on a tablet PC running CogSketch software.
The procedures for the pre-task tutorial on CogSketch and the sketching task were the same as in experiment 1. Each participant received the images in a different random order. At the end of the study, we asked participants to rate their level of prior knowledge of the object or structure depicted in each photo on a scale from 1 (poor prior knowledge) to 7 (excellent prior knowledge). Each session lasted less than 45 min.
Two raters coded each sketch independently with high overall inter-rater reliability (Cohen’s Kappa = 0.88). Coders were blind to any participant identification information. Coding disagreements were settled through discussion.
Self-ratings of Prior Knowledge for the Objects and Structures in the Images
Mean ratings of prior knowledge (out of seven) and mean proportion of key features in participants’ sketches for experiment 2 photographs
Mean ratings of prior knowledge
Mean proportion of key features
Presence of Key Features
As in experiment 1, a geoscience expert analyzed the geoscience photos for their key features. He identified an average of about five features (M = 4.67) in each photo. For the control images, we asked two judges who were unaffiliated with the research to identify the key features in each photograph—i.e., those they considered most central to the overall structure. The judges agreed entirely for two of the photos, and reached agreement on the third after a short discussion. They identified an average of about four features (M = 3.67) in each control photo. Participants’ sketches were coded for the presence of these key features. Table 1 shows the mean proportion of key features for novices and geoscience students for both kinds of images.
Figure 6 also reveals another interesting trend. Several geoscience students added relational symbols (which were not present in the original photos) to their sketches. The half arrows in the geoscience student’s sketch are conventionally used to indicate movement on a fault plane, as determined using “offset markers” (evidence that the previously adjacent beds of rock have been displaced). These arrows were not present in the original image (and in our scoring, were not counted as key features). Nevertheless, several geoscience students drew them, usually after the key features had been sketched. In a technical sense, adding these arrows is an error, as subjects were instructed to copy the photographs. This result strongly demonstrates that the geoscience students brought domain knowledge to bear on their construal of the materials.
Knowledge Ratings and Sketching Performance
If the differences between novices and geoscience students are driven by domain knowledge, then the differences should be greatest for those items on which geoscience students rated themselves as having high prior knowledge. We therefore conducted a regression analysis to predict the proportion of key features in a sketch from three factors: the participant’s rating of knowledge for the content the image, the type of image (geoscience vs. control), and the interaction between these variables. Geoscience experience level (experienced vs. novice) was entered as a covariate and the variables were centered for the analysis. The model accounted for a significant amount of the variance in the proportion of key features, F(4, 89) = 7.68, p < .05, R2 = .26; however, only the prior knowledge rating x image type interaction term was a significant predictor, β = .40, t(93) = 2.92, p < .05, partial r2 = .27. This interaction reveals that prior knowledge ratings were a better predictor of the proportion of key features for the geoscience-related sketches than for the control sketches. Thus, a participant’s self-rating of understanding of a particular geological structure was an important predictor of whether they included key geologic features in their sketch. By contrast, the key features of the control photos were sketched by nearly all of the participants, regardless of their self-assessed prior knowledge of the material in those images.
The results of experiment 2 further bear out the predicted difference between novices and experienced students. As in experiment 1, geoscience students sketched a higher proportion of features that reflected geologic activity, such as fault lines and folds, for photos of geologic formations. When given photos of non-geologic structures, however, geoscience students drew sketches that were highly similar to those of novices (geoscience students included .91 key features vs. .86 for novices, a nonsignificant difference). Only when the images were related to geoscience was there a significant difference between geoscience students and novices. In fact, participants’ self-assessed understanding predicted the presence of geologic features in their sketches of geological photos. These findings suggest that the difference between the groups stemmed chiefly from differences in domain knowledge, as opposed to some more general factor. Geoscience students appear to rely on domain-specific spatial and conceptual knowledge in order to capture the geologically relevant features of an image. Indeed, geoscience students occasionally added causal symbols to their drawings, a strong indication that causal knowledge was activated during the sketching process.
This study focuses on sketching of causal diagrams. We sought to generalize the pattern found in experiment 1 (in which one causal diagram was drawn): that geoscience students differ from novices both in what they include in their sketches, and in the order in which they construct each sketch. As in experiment 2, we included control (nongeoscience) diagrams as well as geoscience diagrams to assess whether any differences between groups could be attributed to domain-general knowledge or skill vs. domain-specific knowledge of the causal processes depicted. Our intention was to select control diagrams that were similar in causal complexity to the geoscience diagrams, and about equally familiar to the geoscience students and novices. Also as in experiment 2, we asked participants to rate their prior knowledge of the system or process displayed in each diagram at the end of the session. If geoscience students invoke knowledge of the causal processes depicted in the geoscience diagrams, then relative to novices (1) they should emphasize more relational information in their sketches of geoscience diagrams, and (2) the order in which they draw parts of the sketch should more closely reflect the sequence of causal events depicted in the diagrams. Further, if this predicted pattern stems from understanding of geoscience, then it should hold only for the geoscience diagrams, not for the control diagrams; and it should be most pronounced for diagrams on which geoscience students rate themselves as having most knowledge.
Sixteen participants from Northwestern University took part: eight geoscience students (five graduate students and three advanced undergraduates) who were paid and eight undergraduates with no prior course experience in geosciences, who received course credit in introductory psychology. None of these participants took part in either experiment 1 or 2. All participated individually.
We used four geoscience-related diagrams and four nongeoscience (control) diagrams. The images are listed in the Appendix.
The procedure for the sketching task was as in the other experiments, except that the participants were given 4 min per sketch. Each participant received the images in a different random order. At the end of the study we asked participants to rate their level of prior knowledge about the system or process displayed in each diagram on a scale from 1 (poor prior knowledge) to 7 (excellent prior knowledge).
For the relational content and ordering analyses below, two raters coded each sketch independently with high overall inter-rater reliability (Cohen’s Kappa = 0.92). Sketches were scored blind to any participant identification information. Coding disagreements were resolved through discussion between the raters.
Mean ratings of prior knowledge (out of seven) and mean proportion of relations in participants’ sketches for experiment 3 diagrams
Mean ratings of prior knowledge
Mean proportion of relations
As our coding took into account the written labels as well as the nature of the glyphs (relational symbols, such as arrows), the difference in relational content for the geoscience-related images could be due to only one of these components or both. There was inadequate statistical power to test the experience level x image type interaction using either relational labels or relational symbols alone as a dependent variable, but trends in the data suggest that the geoscience students used more of each in their sketches of the geoscience-related images.
Geoscience students included a higher proportion of relational labels in their geoscience-related sketches (M = .23, SD = .18) than novices (M = .09, SD = .13), as well as a higher proportion of relational symbols (M = .38, SD = .14 vs. M = .23, SD = .15). (Note that the sum of the relational labels and symbols does not equal the number of relational parts in Table 2, because several parts included both a relational label and symbol, and these counted as a single relation in our coding.) For the control sketches, which did not show a difference in overall relational content between geoscience students and novices, the proportion of relational labels was similar for the geoscience students (M = .08, SD = .14) and novices (M = .04, SD = .05), as was the proportion of relational symbols (M = .33, SD = .14 and M = .30, SD = .15, respectively). In summary, relational labels were relatively uncommon, except for the geoscience students’ use of them in their geoscience-related sketches. The effect of experience level on the overall number of relational parts in the geoscience-related sketches appears to be due to an increased use of both relational labels and symbols by the geoscience students.
Self-ratings of Prior Knowledge of the Systems and Processes in the Images
Table 2 shows each group’s mean prior knowledge ratings for the system or process in the geoscience and control causal diagrams. A 2 (experience level) × 2 (image type) ANOVA revealed a significant experience level x image type interaction, F(1,14) = 8.28, p < .05, ηp2 = .37. As expected, the geoscience students rated their prior knowledge higher on the geoscience images than did the novices, t(14) = 6.14, p < .05, d = 1.86. By contrast, both groups rated their prior knowledge for the control images as about the same, t < 1, ns. Two main effects are subsumed by this interaction: geoscience students rated their knowledge higher than novices, F(1, 14) = 10.19, p < .05, ηp2 = .42, and the geoscience diagrams were given higher ratings than the control diagrams, F(1,14) = 20.68, p < .05, ηp2 = .60. We also note that the prior knowledge ratings were lower for the control causal diagrams than the control photos from experiment 2. One reason is that participants rated their knowledge for the systems displayed in the diagrams, not the objects that were present. The diagram on water conservation in the home included many objects that were likely familiar, such as a shower, toilet, and garden hose. The system that connected these different objects was probably less familiar. Most important for our purposes is that the control images were about equally familiar to both groups.
Knowledge Ratings and Sketching Performance
As in experiment 2, we conducted a multiple regression analysis to predict the mean proportion of relations in a sketch from the participant’s rating of prior knowledge, the type of image, and the interaction between these variables. Geoscience experience level (experienced vs. novice) was entered as a covariate and the variables were centered for the analysis. This model failed to account for a significant proportion of the variance in the proportion of relations, F(4,117) = 1.24, ns, R2 = .04, and no variable accounted for more than 1 % of the unique variance (βs < .16, ts ≤ 1.0). Yet, as is evident from the results in Table 2, there was a strong relation between mean prior knowledge ratings for the geoscience images and the mean proportion of relations in sketches. Participants who gave higher mean prior knowledge ratings for the geoscience images had a higher mean proportion of relations in their sketches, r(16) = .58, p < .05. However, there was no such relation for the control images, r(16) = .08, ns.
For each diagram, a geoscience expert identified two to four key causal events (depicted by one or more directed arrows) and specified their causal order. These were a subset of the total number of arrows in the diagram. We scored the participants’ sketches for the presence of these causal events, and found that geoscience students (M = 0.51, SD = 0.17) drew a higher proportion of key causal events than novices (M = 0.26, SD = 0.22), t(14) = 2.63, p < .05, d = 1.27. When key events were sketched, we used the timing data from CogSketch to determine their ordering. We compared the order in which geoscience students sketched the causal events to those of other participants and to the causal ordering identified by the expert.
Agreement in mean event ordering (Kendall’s coefficient of concordance) for sketches of experiment 3 diagrams
The geoscience students’ ordering was highly similar to the expert’s ordering, Kendall’s W = 0.98, χ2(11) = 21.64, p < .05, implying that geoscience students sketched the events in causal order. In contrast, neither the similarity between the novices’ ordering and the expert ordering, Kendall’s W = 0.39, χ2(11) = 7.72, ns, nor the similarity between the geoscience students’ and novices’ orderings, Kendall’s W = 0.43, χ2(11) = 8.51, ns, reached significance. Mean ordering was representative of ordering at the individual level. The median value of Kendall’s W was 1.0 between the expert and the geoscience students and 0.38 between the expert and the novices.
The control diagrams were from different domains, so two experienced individuals were recruited to identify the causal events. An engineer with over 20 years of experience identified four key causal events in each of the mechanical diagrams (carburetor, blood pump, and water conservation system). An advanced biology student identified five key causal events in the photosynthesis diagram. Each causal event was represented by one or more directed arrows in the diagram, and the experienced individuals specified the order of these events. We scored participants’ sketches for the presence of the causal events. Although there was a suggestion of a higher proportion of key causal events for geoscience students, M = 0.55 (SD = 0.12), than for novices, M = 0.44 (SD = 0.15), this difference was not statistically significant, t(14) = 1.34, p = .20.
We followed the procedure described above for analyzing the control diagrams, using the timing data from CogSketch to analyze the order in which participants sketched the causal events. We again included only the sketches that contained at least two causal events for a given diagram, leaving us with useable data from 100 % of the geoscience student sketches and 79 % of the novice sketches. As shown in Table 3, none of the similarity comparisons reached statistical significance, Kendall’s Ws < 0.58, ps > .05. Thus, in contrast to the finding for the geoscience diagrams, we saw no evidence that either group followed causal order when sketching the control diagrams.
These findings again bear out the prediction that geoscience expertise would be reflected in participants’ sketches of geoscience materials. As in experiment 1, geoscience students included more relational content in their sketches of geoscience-related causal diagrams. Additional analyses suggest that this difference was due to increased use of both relational labels and relational symbols, such as arrows indicating movement or causality, by the geoscience students. The difference was found only for geoscience diagrams. For causal diagrams from other fields, there was no significant difference in the amount of relational information included by the two groups. This is consistent with the idea that the differences in geoscience sketching stem from greater domain knowledge on the part of the geoscience students. Further evidence for the role of domain knowledge comes from the finding that participants’ self-ratings of knowledge of the systems and processes in the geoscience diagrams correlated with their sketching performance. However, unlike our finding for sketches of geological formations (experiment 2), knowledge ratings for individual geoscience diagrams were not a significant predictor.
Geoscience students also differed from novices in terms of how they sketched, and again, this difference was specific to geoscience materials. Geoscience students sketched causal events in the geoscience diagrams in an order consistent with the causal order identified by an expert. Novices, when they did sketch causal events from the geoscience diagrams, did not order them in terms of their causal sequence. Likewise, neither geoscience students nor novices tended to draw the causal events from the control diagrams in sequence. Again, this suggests an effect of domain-specific knowledge.
These studies show that people’s sketching patterns can reveal differences in their understanding of geoscience. When sketching photographs of geological formations, geoscience students were more likely than novices to include spatial structures that reflected geologic activity, such as faults and folds. When sketching diagrams of geologic processes, geoscience students included more relational information in their sketches, and often drew causal events in sequential order. Strikingly, these effects of domain knowledge occurred despite the fact that the task was simply to copy the photograph or diagram. Our participants could view the photos and diagrams for the entire time that they sketched; yet there were still sizable differences in what and how they sketched.
We also asked whether these group differences could have stemmed from some general skill or knowledge that the geoscience students possessed, rather than from greater domain understanding. The pattern of evidence suggests that the effects were due largely to domain-specific knowledge. For both photographs and causal diagrams, the differences between geoscience students and novices in what was sketched, and in the order in which it was sketched, held only for geoscience materials, not for control materials. Further, these patterns aligned with participants’ self-ratings of knowledge for the content of the images. Of course, it is possible that effects of general knowledge or skill will emerge in studies involving different sketching tasks and materials. Further research is needed to address this issue, and to explicitly test the influence of general skills—such as spatial thinking and visualization—that have been associated with attainment in science (Black 2005; Kali, and Orion 1996; Kastens et al. 2009; Shea et al. 2001; Uttal and Cohen 2012; Wai et al. 2009).
Effects of Conceptual Knowledge on Sketching
Our results suggest that the geoscience students relied on conceptual knowledge even when simply copying photographs of geological formations. Their construals were not solely based on the visual information present; they were influenced by participants’ mental models, including knowledge of the forces that could produce a given structure (Forbus 2011; Gentner and Stevens 1983; Hay et al. 2013; Tversky 1991). This is underscored by the finding that geoscience students often added arrows depicting the direction of forces to their sketches of geological formations. Expertise-related exaggerations in spatial depiction have also been found in other arenas. For example, Landy and Goldstone (2007) found that when knowledgeable students copy math equations, they tend to group the elements of operations that must be done earlier in the equation more closely in space. Likewise, acquired categorical knowledge can lead to underestimating distances between same-category items (Holden et al. 2010; Tversky 1981; Uttal et al. 2010). Knowledge of forces and causal relations might lead to similar distortions in sketches of scientific phenomena. For example, understanding of the forces that caused displacement may lead the sketcher to exaggerate this displacement in the direction of the forces. Such findings would shed further light onto the ways that learning affects the organization of domain knowledge (Forbus 2011).
Effects of domain knowledge were also evident for causal diagrams. Geoscience students were more likely than novices to include causally relevant parts for diagrams depicting geoscience processes, but not for control diagrams. This fits with evidence that knowledge of the identity and causal role of the objects in a diagram affects diagram construction. For example, Heiser and Tversky (2006) found that novices who are provided with a functional description of the parts of a diagram were more likely to include relational symbols when sketching the diagram than those provided with a merely physical description.
A particularly intriguing finding was that when copying causal diagrams, geoscience students tended to sketch following the causal order of events. These results align with findings on expertise in engineering and design6. Moss et al. (2006) had freshmen and senior engineering students recreate diagrams of devices using a specialized computer interface that allowed them to drag and drop component parts into a diagram space. Moss et al. found that seniors tended to assemble the parts based on functionality, whereas freshman tended to chunk together the same types of components. More to the point, seniors were also more likely to follow the flow of energy as they recreated the diagrams, as with the geoscience students in the present research. If this pattern of causal ordering holds up in other domains, then students’ order of sketching may provide an unobtrusive measure of their domain understanding.
In sum, this work advances our understanding of science learning by identifying markers of domain understanding that appear in students’ sketches. The findings suggest that, whereas novices tended to focus on depicting the objects in the scene, experienced students activated causal knowledge as they copied images, affecting both what and how they sketched. It is intriguing that similar signs of domain knowledge have been found in other tasks, such as recreating a machine diagram, and with materials that are less intensely spatial, such as math equations. This raises the possibility that identifying certain properties of a sketch—such as its relational content, spatial layout, and temporal construction—may be generally indicative of a student’s level of understanding.
Using Sketching Software to Assess and Enhance Science Learning
We also aimed to make a methodological contribution, by testing whether data gathered by CogSketch can be used to detect markers of domain understanding. We found that it can. Two features were particularly useful. First, CogSketch automatically records the order in which objects are drawn. As our results show, such ordering information can be diagnostic of differences in knowledge. In the past, such timing data has been onerous to obtain. For example, in the Rey–Osterreith Complex Figure test, a widely used assessment of neurological disorder (e.g., Duley et al. 1993; Shin et al. 2006), timing data for participants’ sketches are obtained by giving participants colored pencils. After each time segment, they are asked to switch to a new color. Clearly, this kind of task would be prohibitive in classroom settings. Timing data can also be obtained by analyzing audio and video transcripts, but this is time-consuming and unsuitable for most instructional purposes. Although more analysis are needed to discover when time-course data is most useful, our findings suggest that automatic tracking of the time course of drawing can be highly informative.
A second useful feature of CogSketch is that it requires participants to segment their drawing into objects and label them with words or phrases7. While people could be instructed to label the parts of their drawing in a paper-and-pencil drawing task, it is likely that without close supervision they would sometimes fail to do so, complicating the analysis. Furthermore, the fact that participants distinguished objects from relations using labels (in addition to symbols) was extremely useful in our analyses. In this study, some aspects of the coding—including some of the identification of key events—were done by human coders, but this work was facilitated by the fact that segmentation, labeling, and ordering of events was provided by CogSketch.
CogSketch’s use of participant labeling distinguishes it from most other sketch understanding systems, which are recognition-based (e.g., de Silva et al. 2007; Pargas et al. 2007; Plimmer and Freeman 2007; Plimmer and Hammond 2008; Valentine et al. 2012). Recognition-based systems require that there is a one-to-one mapping between a shape and its intended meaning. This works well when a domain has a graphical symbol vocabulary (e.g., electronics, organic chemistry, and free body diagrams). However, this requirement does not fit many kinds of scientific diagrams. For example, in geoscience, a line can represent a fault, a bedding layer, a horizon line, and many other objects. Moreover, many entities in scientific diagrams are not graphical symbols. When annotating a photograph to show faults and marker beds, the shape of these entities is determined by the structure shown in the photograph, not by a set of domain conventions, as they are in electronics, for instance. Labeling removes any ambiguity about the meaning of a sketch object and dispenses with the need for perfect depiction of the objects in a scene. Although providing labels could be seen as burdensome compared with free sketching, it has the advantage of making the drawing task less forbidding for participants, since they do not need to rely on their drawing skills alone to communicate.
Sketch understanding software could eventually have a profound impact on educational assessment and instruction. Automated recording and analysis of sketches could facilitate the assessment of spatial scientific knowledge and the scoring of standardized tests that involve drawing, greatly reducing the burden on instructors and allowing more rapid feedback to the students. Sketch understanding software could also help improve student learning directly. CogSketch has been used to develop worksheets that are being used in classroom experiments (Yin et al. 2010) to attempt to provide such benefits. Furthermore, these benefits extend beyond geoscience. CogSketch has been used to support learning in fifth-grade biology classrooms (Miller et al. 2014) and in college engineering classes (Wetzel and Forbus 2012). Additional research is needed to explore the connections between sketching and learning, and new technologies are sure to play an important role in this endeavor (Ainsworth et al. 2011).
Our findings suggest that differences in domain-specific knowledge are reflected in both the products and the process of sketching. These differences were apparent even when participants were simply asked to copy the materials in front of them. Our results further suggest that the differences between novices and geoscience students were largely domain-specific. We suggest that the techniques used here could be applied in other spatially intense domains to explore students’ knowledge and to support their learning. Ultimately, work on sketching will not only inform our understanding of STEM learning but will contribute to our understanding of the nature of scientific expertise and how it develops.
This accurately describes the interface used in these experiments; since then, the interface has been streamlined so that pressing a “start glyph” button is unnecessary, and users can merge or split their ink as they desire.
In our coding scheme, each part of the sketch that contained an arrow was coded as a relational part. Although there are five arrows in the geoscience student’s sketch, the participant grouped the arrows into three spatially separated parts; thus, only three relational parts were counted. (Coding also took into account the labels given to the parts, so additional relational parts may be present in Fig. 5 in addition to those indicated in red.)
The coefficient of agreement for mean ordering was lower than the median and modal values because one student sketched the events in perfect reverse order (Kendall’s W = 0). The median value of Kendall’s W was .75 and the mode was 1.0 for geoscience students who drew at least two causal events.
Experiments 2 and 3 both showed a similar statistical interaction, i.e., an advantage for geoscience students on the geoscience images, but no difference between the groups on the control images. However, whereas in experiment 2 both groups produced a high proportion of relations on the control images, in experiment 3 both groups produced a low proportion of relations on the control images.
We used CogSketch’s ordering of events but omitted events not considered to be key events.
This pattern has also been observed in biology. Louis Gomez (personal communication, May 2012) reports that students who understand the photosynthesis process tend to copy diagrams of photosynthesis in causal order.
CogSketch normally requires the user to categorize their glyphs as objects or relations, but to allow for more spontaneity, this feature was not used in this research.