Introduction

When hearing the term “realism,” common associations are computer graphics and virtual reality. This association is usually a positive one—photo-realistic graphics are an important selling point of current entertainment systems and computer technology. Three-dimensional (3D) visualizations are popular tools in the field of education as they allow digitally recreating real-world objects and presenting them to learners. However, there is a longstanding dispute over the influence of realism on cognitive processing, in particular concerning the suitability of realistic visualizations for learning (e.g., Lin et al., 2017). An often-cited theory arguing against the use of realistic graphics is called naive realism (Smallman & St. John, 2005) and emphasizes that studies found negative effects of realistic visualizations on different types of performance measures. Several studies have resulted in evidence that abstract visualizations can be more effective in terms of learning than realistic versions (e.g., Menendez et al., 2020; Scheiter et al., 2009), while other authors argue that realistic details can have beneficial effects, such as helping learners to retrieve information (e.g., Skulmowski & Rey, 2021a). Some of the earliest investigations into the effects of realistic visualizations in the context of anatomy learning by Dwyer showed that realism can be detrimental to learning, but also that the performance on visual test types can benefit from viewing more detailed material (e.g., Dwyer, 1969; see Nebel et al., 2020, for a discussion). Another conflicting result pattern can be found in the literature on mathematics education. Realistic, and thus, “perceptually rich” learning materials were found to be detrimental in a number of mathematical learning tasks (e.g., McNeil et al., 2009; Sloutsky et al., 2005), but it was also shown that such effects are of a weaker magnitude for older students (Kaminski & Sloutsky, 2013).

This contradictory nature of previous results concerning the effects of realism is a major issue for the design of emerging instructional technologies such as virtual reality. Importantly, the creation of photo-realistic computer-generated imagery is usually more costly than creating simple line drawings, necessitating a careful analysis concerning the value of realism for a particular setting. Recent studies have examined the instructional efficacy of realism for different types of learning tasks. One series of studies revealed that realistic details can interfere with learners’ ability to imagine a physical change in organisms (Menendez et al., 2020). More specifically, realism made the transformation of bugs harder to imagine, potentially by offering a too concrete representation that is harder to mentally rearrange into a different shape (Menendez et al., 2020). On the other hand, retention performance in an anatomy learning task was not lowered but raised for those visualizations that included realistic details (Skulmowski & Rey, 2021a). As can be seen from these few examples, the effects of realism appear to require a comprehensive theoretical basis that includes contextual variables.

In this article, we want to lay open the underlying causes of some of these results and demonstrate that realism in itself could in many cases not be the actual culprit for negative effects but rather the vagueness of currently employed definitions of realism as well as the incompatibility of realism with certain learning tasks. Besides, we review recent studies that revealed findings contradicting an absolute interpretation of the naive realism perspective. Crucially, we also address the technological aspect of realism by introducing a fine-grained terminology into the debate that is grounded in the actual creation process of computer-generated imagery. Based on the components of realism from a technical standpoint, namely, the resolution of the geometry, the details involved in the shading, and rendering effects, we present a more accurate system of categorizing realism for empirical studies in order to strengthen their comparability. In addition, we devise a cognitive model of learning with realistic visualizations that traces learning outcomes back to the initial perceptual stage and the ensuing schema construction. This model integrates relevant contextual effects, such as learner characteristics. After discussing these theoretical, technical, and cognitive aspects of realism, we describe how new systematic educational research using a more elaborate framework of realism may contribute towards more comparable research. Furthermore, the model could be used as a guide for investigations into the instructional design of emerging technologies such as instruction using virtual reality.

Theoretical Models of Realism

Definitions of Realism

Realism in visualizations is commonly measured by how accurately a visual representation matches reality (e.g., Rieber, 1994). Schematic visualizations (usually in the form of line drawings) are considered to be the lower end of the realism spectrum. These schematic representations commonly simplify complex shapes and omit some information that may be unnecessary (or too detailed) for specific tasks (e.g., Goldstone & Son, 2005; Scheiter et al., 2009). A typical comparison between a realistic and a schematic visualization from the literature can be seen in Fig. 1. Table 1 lists several conceptualizations of realism that all share an emphasis on how precisely an image resembles a real object (or makes one think of the real counterpart).

Fig. 1
figure 1

taken from the literature. The realistic version of the side view of the knee features surface detail, physically correct surface shading and glossiness. The schematic version is rendered in a toon mode that adds contour lines similar to a drawing and omits any surface detail. The original images including additional text labels and lines were published in Skulmowski and Rey (2020) under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), © 2020 Skulmowski and Rey

A typical comparison between a realistic (left panel) and a schematic (right panel) visualization

Full size image
Table.1 Definitions of realism
Full size table

Dwyer’s Research on the Realism Continuum

The effects of realism have been a major interest for the field of learning in the past decades and found a renewed relevance due to current issues in the design of virtual reality in educational contexts. In older studies, Dwyer used the idea of a realism continuum to compare the effects of different types of visualizations on learning (Dwyer, 1967). Dwyer used heart anatomy as learning content and presented his participants with one of the multiple versions of these presentations (e.g., Dwyer, 1967, 1971). As an example, Dwyer (1967) contrasted learning materials without visuals, versions that included simplified line drawings, materials featuring more detailed drawings, and learning contents containing photographs in their efficacy for learning heart anatomy. Other studies by Dwyer extended the realism continuum by including photographs of heart models as one of the more realistic points of the continuum and comparing the use of colored representations with grayscale images (Dwyer, 1971). As noted by Skulmowski and Rey (2018; see also Castro-Alonso et al., 2016), it may be difficult to trace differences in learning performance back to the level of realism if the different visualizations cannot be considered to be equivalent in the quantity of information they convey (although Dwyer, 1967, acknowledges this issue).

One important insight reported by Dwyer is that a greater level of realism is not necessarily linked to better learning performance (e.g., Dwyer, 1971). He emphasized that the specific task demands and goals are an important aspect to consider when choosing the degree of realism (Dwyer, 1967), as different visualization types in some cases lead to fundamentally dissimilar performances on different learning measures (Dwyer, 1967, 1971). However, we argue that investigating the effects of realism using a continuum ranging from more abstract to more faithful representations should be considered an inappropriately vague approximation. A unidimensional continuum ignores the degrees of freedom and range of choices available in styling a visualization using modern 3D tools. In the next section, we review current, more precise ways of classifying realism.

Recent Classifications of Realism

Since Dwyer’s studies on the realism continuum, several related classifications have been put forth. For instance, Höffler (2010) used four realism levels varying from abstract to realistic in a meta-analysis that did not reveal realism as a significant moderator of learning. These four levels are primarily distinguished by their level of detail (Höffler, 2010), and other authors have similarly linked realism with the amount of detail present in a visualization (e.g., Brucker et al., 2014; Skulmowski & Rey, 2018). Höffler’s (2010) classification resembles Dwyer’s realism continuum in that it is a one-dimensional, unidirectional ordinal scale. However, multi-dimensional descriptions of visualizations have also been proposed. Çöltekin et al. (2016) argue that there needs to be a distinction between the three-dimensionality of graphics and their realism. They argue that realistic visualizations do not necessarily need to be 3D graphics and that 3D graphics do not necessarily have to look realistic (Çöltekin et al., 2016).

Current Issues of Realism Research

As discussed in the previous section, there exist several models of realism, but two fundamental problems facing research on realism in educational settings remain: a vague terminology and the lacking comparability of realistic visualizations. Current realism studies usually compare two to four conditions in which different versions of visual learning materials are used (Skulmowski & Rey, 2021a; e.g., Brucker et al., 2014; Lin et al., 2017; Scheiter et al., 2009; Skulmowski & Rey, 2018). Across most of these studies, the terms “realistic” and “schematic” (and their synonyms) can have a vastly different meaning (Skulmowski & Rey, 2018). For example, Brucker et al. (2014) compared live-action videos of fish with animations featuring contour drawings. The videos show the fish in their natural surroundings while the animations only feature the contours of the fish against an all-white backdrop (Brucker et al., 2014). In two studies by Scheiter et al. (2009), microscopic videos of cell replication featuring the typical grainy look of such photographs were contrasted with strongly simplified drawings. Lin et al. (2017) used shaded and colored anatomical drawings as the “detailed” version and a “simplified” version that relied on grayscale contour drawings. In most of these examples, the visualizations used as learning materials were primarily described to be exemplars of different levels of realism (or having different amounts of detail) while actually differing in one or more additional aspects, such as the use of color (see also Castro-Alonso et al., 2016). Furthermore, the different forms of schematization and realism would certainly not fall on the same two points of Dwyer’s (1971) realism continuum (see Brucker et al., 2014, for a discussion of this aspect). While the schematic version used by Scheiter et al. (2009) features a high level of abstraction consisting of basic two-dimensional shapes, the schematic learning materials presented in the study by Brucker et al. (2014) still contain enough visual information to be considered simplified drawings with some amount of detail. As can be seen from these examples, the effects of realism are hard to compare across studies (see Skulmowski & Rey, 2018, for a discussion).

Theoretical Approaches to Realism: Naive Realism and Cognitive Load Theory

Before we turn to the technical implementation of realism, we first take a look at some relevant theoretical models that can further our understanding of the effects of realistic graphics on learners. One of the more prevalent theoretical frameworks used to discuss the effects of realism is cognitive load theory (CLT; Sweller et al., 1998, 2019). Essentially, cognitive load theory suggests that unnecessary cognitive load (extraneous cognitive load; ECL) may be created by inadequate instructional design and thus precludes learners from using their maximum cognitive capacity for the proper learning contents (intrinsic cognitive load; ICL; Sweller et al., 2019). The less extraneous load, the more cognitive capacity is available to process intrinsic load (Sweller et al., 2019). Using the CLT perspective, realism has been considered to potentially endanger learning by introducing an unwanted complexity (e.g., Scheiter et al., 2009; see also Castro-Alonso et al., 2019). However, given the problem of comparability described above, such broad statements currently need to be made with caution (for discussions of this issue, see Brucker et al., 2014, and Nebel et al., 2020). Thus, CLT can be considered an important theoretical basis for instructional realism research.

Similarly to CLT, a theory focused solely on realistic visualizations recommends the use of simplified rather than realistic imagery. This perspective is called naive realism and states that laypeople generally prefer realistic graphics due to naive beliefs concerning the cognitive system (Smallman & St. John, 2005). Smallman and St. John (2005) emphasize three mistakes that laypeople make when thinking about visual perception. According to Smallman and St. John, people assume that seeing is a simple, precise, and exhaustive capability. They summarize the naive idea of visual perception as the conviction that human eyes work like cameras which accurately transform our surroundings into high-fidelity “mental photographs.” But actually, visual perception is a complex and error-prone process (Smallman & St. John, 2005). They provide the example of change blindness, the psychological phenomenon that people fail to notice even major changes in a scene if their attention is directed towards a specific goal (Smallman & St. John cite Simons & Levin, 1998). These fallacies have been described as contributing to naive beliefs regarding the usefulness of realistic imagery (Smallman & St. John, 2005). Importantly, they give several examples in which realism actually decreased performance, mostly as an effect of being overwhelmed by details or a lack of clarity. Based on these assumptions, Smallman and St. John (2005) warn of an unjustified belief that using technologically advanced visualization techniques automatically lead to better (cognitive) performance. Since they are primarily focused on the field of ergonomics and human factors, we now turn to an overview of theoretical approaches from educational contexts.

The Technical Dimensions of Realism: Geometry, Shading, and Rendering

In order to investigate more nuanced effects of realism on learning, controlled comparisons between few, but well-defined exemplars of learning materials differing in their level of realism should be the most informative. We argue that a well-defined degree of realism is achieved if researchers and practitioners have enough information concerning the learning materials available to recreate that particular level of realism with the same or other learning contents–shortly put, to create reproducible comparisons. In order to do so, research papers should provide more standardized information concerning the learning materials used in their studies (although most papers already include excerpts of the learning materials). Also, it will be useful to include the name of the software package(s) used and whether special plug-ins or unusual settings have been utilized. But more importantly, authors need to specify their design choices made during the production of their learning materials. For example, is the 3D model used for “realistic” and “schematic” renderings the same? Does the realistic version use texture maps? These and other questions are important to thoroughly understand how the learning materials were created and what their level of realism is. In the following, we present a model that offers a starting point for more specific categorizations of computer-generated instructional visualizations.

In order to accurately capture the finer nuances of comparisons between realistic and abstract visualizations, we propose to assess visualizations concerning their geometry, shading, and rendering. This model of realism is grounded in the actual choices that have to be made when creating visualizations using 3D applications. Thus, our model is closely aligned with the different methods a visualization can be created using current 3D software packages. Furthermore, this model can be used both by educators, designers, and researchers to find a common terminology to more adequately describe various visual styles. In order to do so, we propose a multi-dimensional model consisting of the three dimensions of geometry, shading, and rendering.

In our geometry, shading, and rendering (GSR) model, we distinguish between the extent of realism present within the three dimensions that can be individually manipulated. As can be seen in Fig. 2, geometry, shading, and rendering options can individually range from simplified to highly realistic. Throughout the sections on geometry, shading, and rendering, we will describe basic concepts of 3D visualizations as they are implemented current 3D software packages. Figure 2 was created using Blender (Versions 2.79b and 2.91.2; Blender Foundation, Amsterdam, the Netherlands). It depicts examples for the different combinations arising from the dimensions of our model. Most importantly, these renderings underline how the different dimensions individually contribute to realism.

Fig. 2
figure 2

The geometry, shading, and rendering (GSR) model exemplified using renderings of the patella (modeled after Gray, 1918). The first row presents three levels of detail for the dimension of geometry: a low resolution approximation is shown in the left panel, the middle panel contains a rendering in which the basic shape of the model is accurately represented, and the model featured in the right panel contains even small details such as ridges. The second row gives an impression of different shading options: while the left panel shows the model shaded using a solid white color, the middle panel adds a color texture. In the right panel, shading is enhanced using specular shading. The bottom row presents different rendering styles: the left panel uses toon shading with a contour line, the middle panel utilizes realistic rendering, but with unnaturally hard highlights and cast shadows. The right panel shows the patella rendered using softer, more natural lighting

Full size image

Geometry

Digital models usually start out as a geometric primitive, such as a box or cylinder (Paquette, 2013). As polygonal models used for 3D rendering can consist of varying counts of polygons (i.e., solid faces in 3D space), details can be added to a model by increasing the number of polygons and using the new geometry to sculpt details (see Paquette, 2013). Besides adding geometry and modeling more details, it is possible to create stylized shapes, thereby emphasizing certain aspects of the model or simplifying the model (see Paquette, 2013). Generally speaking, the range of realism within the dimension of geometry ranges from flat or smooth, even surfaces to highly detailed 3D scan data.

Based on several results, less detail may in some cases facilitate visual processing and thus benefit learning (e.g., Scheiter et al., 2009). From a cognitive load theory perspective, added details may be considered as a source of extraneous load (e.g., Scheiter et al., 2009) as long as these details are not essential to accomplish a learning task (see Castro-Alonso et al., 2019). However, it should be noted that in some instances, detailed visualizations were found to receive more attention than simplified versions (Lin et al., 2017). Given that the geometry of a model determines the overall shape as well as details, it is highly important to consider this dimension when creating models for instructional visualizations. We will discuss ways on how to take advantage of this component in later sections.

Shading

In addition to the complexity of the model, shading options can dramatically increase realism. By default, many software packages assign single-colored, plain surfaces to newly generated 3D models. Using shaders and texture maps, a considerable amount of visual fidelity to real-world objects can be achieved. Furthermore, material properties can be set to make the model appear shiny, glossy, or transparent. Realism may be enhanced using texture maps (i.e., images that are “wrapped” around the 3D geometry, see Paquette, 2013) that can fulfill different purposes beyond mapping a color map (i.e., a photograph or painted texture depicting a real object’s surface).Footnote 1 Even a very simple geometry can be transformed into a realistic visualization that resembles a photograph through the use of texture maps. Thus, the problem of cognitive overload through details discussed above may apply to the use of shading. As an example of how shading can affect learning, one study examined the effects of detail realized through shading and found that the cognitive burden resulting from detailed shading can be compensated using a colored texture map that helped learners to segment the parts of the object (Skulmowski & Rey, 2018). Results such as this one may be used to arrive at more specific guidelines on how to employ realistic shading in a way that does not overburden learners.

Rendering

The last dimension of the GSR model is rendering. This entails setting up the virtual lights and camera using which the models are lit and “photographed.” The rendering stage offers several options to minimize or maximize realism. By default, modern software such as Blender generally uses rendering algorithms that closely mirror how light travels through space in the real world. This includes indirect lighting and related forms of physically-correct lighting that keep renderings from appearing unnatural, such as when cast shadows lack a variation in value and are simply rendered as a solid dark color. The similarity between computer-generated imagery and photographs can additionally be enhanced through more specialized rendering options, such as motion blur or setting the depth-of-field to mimic the focus of a real camera. However, there are also options to decrease the degree of realism, for instance by creating “toon” renderings using specific shading and rendering options. To achieve the look of drawings or cartoons, rendering software can simplify a surface to consist of only a light and a dark area instead of a realistically rendered image of the surface in question. The resemblance to drawings can be further enhanced by letting the software superimpose contour lines onto the rendering (see Fig. 2). It needs to be considered that the toon rendering style is not a distinct rendering mode but merely an extreme simplification of the usual realistic rendering mode. While a realistic rendering is created by generating fine-grained calculations of how much light a specific point on a surface is reached by, a toon shader simplifies these gradations of light and shadow into just a few values (usually a half-tone and a shadow color). The outline is a separate effect that can be applied at the rendering stage to further strengthen the appearance of a drawing.

In many studies on instructional visualizations, the rendering mode is changed alongside other parameters such as shading, thereby transforming a realistic model featuring believable shading and accurate rendering into a single colored, “toon” rendered model (e.g., Skulmowski & Rey, 2020, 2021a, b). Two studies specifically compared 3D-rendered visualizations with a 2D variant (Huk et al., 2010). These studies featured the same objects with the same shapes and coloring in both conditions, thereby allowing to trace back any differences in learning performance directly to the dimension of rendering (although a slight difference regarding the camera angle was also present). In the first study, the comparison between the 2D and 3D variant did not result in a significant difference regarding retention performance, but the use of a 3D presentation did increase comprehension test scores. Experiment 2 did not confirm this positive effect.

Towards More Fine-Grained Hypotheses Using the GSR Model

Shortly summarized, the complexity and styling of the geometry underlying a visual rendering determines the amount of detail and the similarity to the overall shape of a real object (Paquette, 2013); shading techniques such as image textures or parameters such as the specularity of an object can further contribute to the realism of a rendered object (Paquette, 2013); and rendering options such as cartoon shading (e.g., Todo et al., 2009) can be used to give a visualization a hand-drawn look, while current ray-tracing technologies can be utilized to create visualizations with physically correct lighting (Paquette, 2013).

Given our three-component model, it needs to be noted that studies on realism in instruction usually compare only a few possible combinations of the dimensions we just presented. Nevertheless, the implicit (and sometimes explicit) assumption is that these studies deliver results that can be used as arguments in favor or against realistic learning materials in general (for a discussion, see Skulmowski & Rey, 2018). Instead, we argue that most previous studies on this issue offer helpful data concerning how different GSR combinations fare when pitted against each other, but more specific comparisons are needed to attain more precise guidelines.

The application of the GSR model can enhance the understanding of the particular realism levels investigated in studies and lead to more nuanced comparisons. Hence, arriving at generalizable claims concerning realism can be much more straightforward and fine-grained than relying on vague descriptions such as “realistic” and “schematic.” Acknowledging that realism in computer-generated imagery is the result of the combination of different design choices that can be varied more or less independently puts research on realism on a more solid theoretical ground and facilitates replication.

A Cognitive Model of Learning with Realistic Visualizations

After having defined a technical model of 3D visualizations, we will now turn to the cognitive processing of realism. Considering that realism can have a negative toll on cognitive processes, as emphasized by the perspectives of naive realism and cognitive load theory, the question arises how we can use realism while avoiding the potential drawbacks. Judging from the body of research presented in this paper, using realistic visualizations could in many cases be regarded as an ill-guided way of designing learning materials that risks overburdening learners with information that is unnecessary for their particular learning objective. However, we will extend our GSR model of realism to incorporate the cognitive functions of perception, schema generation, and learning performance in the following sections. At the same time, we highlight how to avoid the potential disadvantages of realism through appropriate design choices. In addition, we discuss whether the naive realism perspective is sufficient to assess the usefulness of realism.

Our cognitive model of learning with realistic visualizations (see Fig. 3) includes three stages: (1) in a perceptual stage, learners visually process a visualization by segmenting the contents and forming a 3D representation. (2) Then, learners need to form a schema in long-term memory based on the 3D representation and additional information that may be presented alongside the visualization. (3) At the testing stage, learners’ performance can be assessed using different assessment types. A number of factors can affect learning outcomes at the various stages of this model. It needs to be noted that these stages may be completed in a different order as well as in a continuous loop, for instance, if a virtual reality learning environment is being used and several objects are perceived and learned in succession. We will now discuss the different stages in more detail.

Fig. 3
figure 3

The cognitive model of learning with realistic visualizations. The level of realism created using the components of geometry, shading, and rendering is perceived by learners. The degree of realism can contribute towards a perceptual load that can be lightened using measures such as cues. From this mental representation of the visualization, a schema needs to be generated by learners. The perceptual load is now processed in the form of different elements in the learners’ minds that is integrated into long-term memory. The number of elements and the complexity of their relationships increase the cognitive load, which some learners can tolerate better than others due to differences in learner characteristics. In the testing stage, the resulting schemas may affect performance in different tests based on the level of realism of the visualization. In addition, the schema constructed during learning may affect their perceptual load and further attention-allocation (see Hegarty, 2011)

Full size image

Perception

Based on the degree of realism introduced using the individual dimensions of the GSR model, learners must first generate a 3D mental representation of the shapes presented in a visualization. Berney et al. (2015) provide a cognitive task analysis of the first perceptual stages of learning anatomy using visualizations. In this analysis, the first step consists of extracting a form from the surroundings. This step is supposed to lead to a mental representation of the isolated relevant features of that visualization. In a second step, multiple views are being used to memorize the form as a flat plane. From these two-dimensional views, learners need to generate a mental 3D model in the third step.

An important mechanism during the formation of a 3D mental representation of objects is thought to use geometric primitives, so-called geons (Biederman, 1985, 1987), as components to represent complex real-world objects (Nebel et al., 2020; Skulmowski & Rey, 2018). For instance, a table can consist of a flattened box with four elongated cylinders, and more complex structures such as a hand can similarly be approximated using geometrical primitives (such as a wedge shape for the palm and a series of cylinders for the finger segments). By adding information to the GSR dimensions of a visualization, this segmentation process may be made more difficult. By adding more complex geometry and greater detail, a segmentation into simple primitives can become harder, as more mental effort must be put into this process. Learners are then required to use more of their working memory capacity to mentally hold all the different primitives of a complex shape. Similarly, different colors, patterns, and shaders can make it harder to detect the underlying shapes. Furthermore, all of these different information communicated using shaders become elements to be learned on their own. As an example, bones in a realistic anatomical rendering will likely have a matte appearance, while fat pads will have a light glossiness to them. Muscles feature a high number of small fibers that may be visible in a realistic visualization. All of these shading properties, such as colors and material qualities, generate additional elements of information that need to be kept in working memory. Simplified rendering options can further remove information from a visualization by removing shadows and introducing contour lines that help in segmenting the components of a visualization. These examples show that a more complex geometry, intricate shading, and realistic rendering can induce a certain perceptual load. Perceptual load is commonly understood as the demands that visual presentations place on their observers (Lavie, 1995; Lavie & Tsal, 1994). These demands can include the number of elements that need to be attended to (Lavie, 1995). For our purposes, we define perceptual load in realistic visualizations as the result of complex shapes that need to be mentally disassembled, different materials that need to be distinguished, and rendering effects that may distract from the essential information. Additional geometry can be utilized to add further details, texture maps are a means to depict material properties of an object, and rendering effects such as depth of field can contribute towards realism, but may make a rendering harder to visually process. For instance, the middle image in the top row of Fig. 2 depicts the general shape of the patella, but the image in the top right corner additionally features ridges on the model. Observers need to process this new visual element of the model having ridges, involving considerations whether the exact shape of these ridges may have a relevance as well as other processing steps. This perceptual load can be an obstacle in processing a realistic visualization and may produce more visual elements, such as geometry parts and materials that must be kept in working memory.

Fortunately, there are strategies to minimize the perceptual load of realistic visualizations. An effective principle in the design of (digital) learning is signaling (for a meta-analysis, see Richter et al., 2016). There are numerous ways in which the importance of contents or their relation can be communicated to learners, including color highlighting (e.g., Ozcelik et al., 2010; see also Castro-Alonso et al., 2021) as well as the use of matching colors for visualizations and accompanying texts (Scheiter & Eitel, 2015). An eye tracking study found that expert learners focus less on task-irrelevant parts of visualizations while novices make more fixations in those unimportant sections (Gegenfurtner et al., 2011). Thus, it becomes understandable why a meta-analysis revealed that signaling is of particular use for novice learners rather than experts (Richter et al., 2016). Considering that signaling can be accomplished with various types of differences in a visual presentation that draw attention, it is important to note that detailed visualizations have been found to elicit more visual attention in an eye tracking study than schematic drawings (Lin et al., 2017). In line with this finding, several recent studies used the realism of visualizations to signal important information (e.g., Lokka & Çöltekin, 2019; Skulmowski & Rey, 2020). In a route learning task, Lokka and Çöltekin (2019) were able to show that realism can be used to highlight important landmarks. In their study, three variants of a virtual world were presented to participants: a highly realistic street featuring texture-mapped buildings, a simplified version in which the houses on the street were presented as untextured, white blocks, and a presentation style that combined the simplified version with only few realistically rendered buildings. These realistic buildings served as landmarks. The latter mode of rendering that uses realism to communicate the importance of certain buildings facilitated route learning and elicited the highest recall scores. This memory effect was adapted to anatomy learning by Skulmowski and Rey (2020) by combining one realistic (i.e., featuring surface detail with detailed textures) and one schematic visualization (i.e., a toon rendering) of the human knee joint into one presentation. Retention scores were higher for the respective realistic visualization previously presented to participants in that study. Hence, Skulmowski and Rey (2020) presented their results as a realism-based form of signaling. It needs to be noted that there are several ways that signaling can interact with perceptual and cognitive load. Color cues can be used as part of realistic visualizations as a form of signaling to reduce cognitive load (Skulmowski & Rey, 2018), while differences in realism within a visualization can direct learners’ attention to relevant parts (Skulmowski & Rey, 2020).

Considering the studies we just summarized, the differences in realism between the signaled and non-signaled components of a visualization might need to be substantial to achieve the desired effect. In terms of the GSR model, it is unlikely that a small difference in the number of details generated through geometry will suffice. Instead, both studies used extreme differences in the approach to shading and compared untextured objects with models to which highly detailed textures had been applied. Also, Skulmowski and Rey (2020) changed the rendering mode by using a toon-shaded model for the non-signaled components. In sum, signaling using realism may be best achieved by implementing a drastic contrast in terms of shading and rendering.

Schema Construction

With a 3D representation in their mind, learners must now generate a schema in long-term memory. As already mentioned, the GSR components may lead to a higher density of information that learners must keep in mind as different elements. These elements also have specific visual relationships (such as arrangements) that may need to memorized. As such, the elements and their relationships can be considered a primarily visual form of element interactivity (Sweller, 2010; for a related argument, see Frederiksen et al., 2020). Sweller defines an element as a basic building block of learning materials and gives the examples of concepts and procedures. He further specifies low element interactivity to occur when the elements in learning materials can be learned in isolation, without links between these elements that need to be considered. Conversely, a high element activity demands that learners not only learn the individual elements but also their relationships (Sweller, 2010). This notion of element interactivity can be applied to learning using realistic visualizations. For example, in the visualization shown in Fig. 2, the realistic version with surface details and naturalistic rendering could be considered to contain additional elements when compared with the toon rendering. These elements include the color variation on the surfaces, the distinction between matte and shiny surfaces, and information regarding the smoothness and roughness of the surfaces. Thus, learner representations of the realistic version will likely contain the gist of these surface properties that may then be beneficial for later recall (Skulmowski & Rey, 2021a).

However, learners need an appropriate mental capacity to deal with this additional information. In numerous studies, it was demonstrated that the effects of realism on learning performance depend on the learners’ spatial ability (e.g., Brucker et al., 2014; Huk, 2006; see Castro-Alonso et al., 2019). In a comparison between a digital learning environment on the topic of biological cells, learners with a higher spatial ability achieved higher scores after having learned with 3D models (Huk, 2006). Learners with a lower spatial ability, however, performed better without the realistic presentation in that study. This was explained using the assumption that students high in spatial ability were not cognitively overtaxed by the 3D models and thus had cognitive capacity left to benefit from the 3D view (Huk, 2006). Students who had a low spatial ability appear to have been cognitively overburdened by the 3D models (Huk, 2006). A study by Brucker et al. (2014) compared dynamic visualizations of fish that were either presented as contour drawings of the fish or through actual film footage of these fish. One of the main results was that participants with lower visuospatial abilities achieved higher scores when learning with the simplified drawings, while participants who had higher visuospatial abilities performed better after having learned with the real fish footage (Brucker et al., 2014).

Results such as these are hard to integrate into the idea of naive realism. While it may be true that a layperson might assume an overall advantage for learning with realistic visualization regardless of spatial ability, following the conclusion of avoiding realism would rob more capable learners of an opportunity to gain in-depth knowledge. Identifying individual components of the GSR model that are relevant for ability-based effects of realism is not as straightforward as for the previously discussed effects. Advantages based on a higher spatial ability may depend on the geometry and shading of 3D models (e.g., Huk, 2006) or on the rendering style (e.g., Brucker et al., 2014). More research is needed to understand why and how learners with a high spatial ability profit from realism.

Getting back to the notion of element interactivity, this concept has been an important foundation of the expertise reversal effect (Kalyuga, 2007; see Chen et al., 2017). This effect occurs when learners with a high prior knowledge achieve a worse learning performance than novices using the same learning material (Kalyuga, 2007). One explanation for this effect is that information already known to experts hinders them during learning and thus becomes a form of extraneous load (Sweller, 2010). Novices, however, profit from the information and thus process this information as a form of intrinsic load (Sweller, 2010). A similar mechanism may be behind the results showing that high spatial ability learners appear to be able to benefit from realistic visualizations, while low spatial ability learners do not (for an alternative explanation arguing that learners’ abilities can make up for a demanding instructional design, see Mayer & Sims, 1994). Consequently, spatial ability is one learner characteristic (potentially among many more) that can affect schema construction. It needs to be noted that information gained from learning with a visualization may affect the subsequent observation pattern (see Hegarty, 2011).

Testing

Finally, learning tasks usually include some form of assessment in which learners need to retrieve the learned information and possibly apply them to new tasks. The nature of these tests is emerging as a major factor that determines whether realism is helpful. Older studies by Dwyer already indicated that in some cases, more realistic visualizations can have an advantage over simplified or schematic versions. For instance, in learning tests that require in-depth visual knowledge of an organ, having learned with a realistic visualization including details enables learners to more easily identify relevant structures (e.g., Dwyer, 1969). Schematic visualizations such as line drawings, on the other hand, are particularly useful for learning visual information that needs to be transformed or transferred to novel problems by learners (e.g., Menendez et al., 2020; see Fyfe et al., 2014). An explanation of such negative effects of realism is that having a very concrete mental representation may be an obstacle to overcome when being required to mentally transform this information (Menendez et al., 2020; see Fyfe et al., 2014). As a result, such tasks will be easier to solve with a more simplified mental representation gained using schematic visualizations.

The feature of delivering a higher amount of concrete information that enables learners to form mental connections was recently linked to the notion of retrieval cues by Skulmowski and Rey (2021a). In that study, surface detail has been shown to act as a retrieval cue in an anatomy learning task, an effect that is particularly strong for learners who were already familiarized with the realistic learning materials before the test (Skulmowski & Rey, 2021a). It needs to be emphasized that this study only used a retention test. Future research is necessary to determine the cognitive mechanism behind this effect. It may be the case that realistic details that are present in the learning phase as well as in the testing phase allow learners to use these details as cues to mentally retrieve information (Skulmowski & Rey, 2021a). Another explanation may be that this match between two detailed visualizations could facilitate the identification of relevant parts of the visualization—a feature that schematic visualizations do not have due to their lack of details. Furthermore, it should be investigated whether this benefit of realism regarding retention performance is robustly diminished by a lower transfer performance (as suggested by Menendez et al., 2020, and Scheiter et al., 2009).

Following our GSR model, we assume that the aspects of geometry and shading may be the most important factors to consider when utilizing realism as a retrieval cue. In order to be able to act as a cue, models used for such visualizations might need to have recognizable details that will usually need to be shaped using geometry or be “painted” on the surface using a variety of textures. Global rendering parameters such as lighting settings will unlikely create memorable cues, although lighting effects that highlight some details over others could be considered when trying to use this effect.

The examples discussed in this section confirm the assumption that realistic visualizations lead to the creation of more concrete and detailed representations (Goldstone & Son, 2005), while schematic visualizations appear to foster the generation of more flexible representations that lend themselves particularly well to transfer tasks. Again, these results underline that blanket statements in favor or against the use of realism might not be appropriate.

Summary of the Model and a Categorization of Previous Research

Based on the GSR model, our cognitive model of learning with realistic visualizations can be used to formulate more specific recommendations on the use of realism in instructional visualizations. Realistic details created using shading and texture maps can induce a perceptual load that can, in turn, be lowered using cues and, potentially, by other means of instructional design. At the stage of schema creation, learner characteristics such as their spatial ability affect how well relevant information is extracted and converted to a stable representation in long-term memory. In addition, using the CLT terminology, we can assume that different learning goals can transform realistic details from being a form of extraneous load towards contributing towards ICL (and vice versa) as recently discussed by Skulmowski and Xu (2021). The level of realism in terms of the global shape (usually generated using geometry) needs to enable the generation of a stable mental representation for retention tasks while not “locking” learners into a specific visual format in case they need to transfer this information. Depending on the learning task, one should carefully choose which facet of the GSR model should feature a higher level of realism (and thus, potential costs in cognitive load) and find ways to balance the total mental load.

Table 2 lists a number of examples from the literature and categorizes the comparisons made in these studies according to the GSR dimensions. In addition, the effects found on different dependent variables, ranging from retention and transfer to more abstract comprehension and problem-solving tasks, are included. The table can be used as a guide on how to plan, categorize, and describe previous and future research. However, the categorization of existing research is often impossible to perform due to missing information, lacking figures, or both. From the limited literature that can be appropriately described using the model, several trends and avenues for future research become apparent.

Table.2 Empirical results concerning different types of knowledge tested using different test types
Full size table

Studies in which surfaces and shapes are the main learning content tend to use retention tests to measure performance, often finding an advantage for more realistic visualizations. Realism is usually increased using shading and rendering options in these studies, suggesting that there is a gap concerning the effects of the complexity of the geometry shown in visualizations. Furthermore, this type of study rarely includes transfer and problem-solving tasks that challenge learners to apply their knowledge regarding surfaces to novel, but related stimuli. Regarding learners’ performance with realistic visualizations when learning about processes, most of the studies in this category either found null effects or disadvantages of realism on learning, usually measured using transfer and comprehension tests. Since the learning of processes usually involves both the learning of concrete shapes in conjunction with the more abstract changes they are involved in (or go through), future research should aim to disentangle and consider all these components of the learning task for a more comprehensive look at this type of task. In particular, it should be assessed whether learning about concrete shapes and abstract processes has a cost–benefit aspect to it, so that a stronger focus on concrete shapes hampers with the comprehension of abstract aspects (as suggested by Menendez et al., 2020). Finally, learning to perform abstract calculations or comprehending abstract content appears to be hindered by concrete details, at least judging from the literature summarized in Table 2. The studies summarized in the table mostly use very drastic differences in realism according to the GSR model, thereby showing a research gap for more subtle variations in realism and their effects on abstract cognitive processes.

Realism as a Condition for Cognitive and Emotional Effects

In addition to the immediate effects that realism has, it is important to consider a number of aspects that are indirectly enhanced or enabled by the use of realistic visualizations. One important advantage of realistic visualizations is that they can be used in stereoscopic displays. Virtual reality headsets and other devices that present viewers with adjusted views for each eye induce the effect of depth perception that occurs during natural binocular viewing (Urey et al., 2011). Across several disciplines, it has been found that stereoscopic viewing has a major advantage over flat, non-stereoscopic presentations for tasks involving spatial perception (e.g. Hackett & Proctor, 2016; McIntire et al., 2012, 2014). This is especially true for the presentation of complex scenes such as visualizations with a high level of detail (van Beurden et al., 2010). For instance, with a focus both on static and dynamic visualizations, research revealed an advantage of stereoscopic visualizations for estimating spatial relationships and for recognizing structural details (Remmele & Martens, 2019; Remmele et al., 2015, 2018). Since science education aims to convey authentic information about topics that often can hardly be demonstrated to students in their original form, such as human anatomy, one desideratum is to find a digital mode of presentation that enables a perceptual experience that is as faithful to our perception of the real world as possible. Due to the more authentic experience offered by stereoscopic visualizations compared with their non-stereoscopic counterparts, stereoscopy can be regarded as a potential contributor to a more authentic learning experience (see Schwan & Dutz, 2020, and Schwan et al., 2014, for overviews of authentic learning). Regarding the learning of complex physiological processes related to anatomical structures, recent studies indicate that stereoscopic visual perception might offer learning enhancements when combined with interaction possibilities with 3D objects (Bogomolova et al., 2021). Realistic visualizations presented using stereoscopic devices may be more appropriate for transfer-oriented learning compared to non-stereoscopic imagery (Bogomolova et al., 2021). However, to what extent stereoscopic imagery could contribute to counter the restricted suitability of realistic visualizations for transfer tasks described in previous sections remains an open question. One explanation for the advantages of stereoscopic viewing found in the literature is that stereoscopy is a way to lower extraneous load by sidestepping certain mental transformations usually needed to build a 3D mental representation from flat two-dimensional images (Bogomolova et al., 2020).

In addition, certain emotional and perceptual effects also depend at least in part on the degree of realism. In two studies, Kwon et al. (2013) investigated how realism affects the feeling of being present in a virtual world and the level of anxiety felt during an emotionally taxing situation. Realism was varied by comparing the use of photographs of people, a virtual agent moderately high on all GSR dimensions, and a 3D cartoon character as virtual agents involved in the task. These studies revealed that the induced anxiety did not directly depend on the level of realism, but that the impression of being present in the virtual world was affected by the degree of realism. Furthermore, the emotional response of anxiety depended on this feeling of actually being present in the virtual world. Based on these results, we can assume that certain emotional and immersive features relevant for digital learning (see Makransky & Petersen, 2021, for an overview) require at least some degree of realism. Another example that illustrates how realism can be a necessary condition for cognitive effects comes from a study by Assländer and Streuber (2020). In their study, participants took part in a balance behavior task using a virtual reality headset in a realistic (i.e., detailed, textured, and physically-correctly rendered) or a simplified (i.e., less accurate, largely untextured, and more primitively rendered) version of a real environment. Furthermore, the study also compared participants’ sway behavior with the real environment and a condition in which they kept their eyes closed. The study revealed that participants’ bodily behavior was similar in the real environment and the realistic virtual reality reconstruction but differed in the schematic version and when participants kept their eyes closed. This result suggests that a realistic (in terms of all the dimensions of the GSR model) virtual reality scene elicits comparable bodily behavior to a real environment, while a simplified virtual environment does not have a comparable effect (Assländer & Streuber, 2020).

Besides the cognitive aspects that make up our cognitive model of learning with realistic visualizations, realism can have a range of attitudinal, social, and emotional effects (for an overview, see Nebel et al., 2020). For instance, the credibility of learning content is a highly relevant factor for educational contexts (Chinn et al., 2020; Scharrer et al., 2019). The stronger resemblance between realistic visualizations and the real world is a major advantage over schematic visualizations that has been demonstrated in empirical studies (Skulmowski & Rey, 2021b; Zanola et al., 2009). Zanola et al. (2009) compared three visualization styles regarding their credibility ratings: a sketch style mimicking hand-drawn lines, a clean technical style lacking details, and a realistic style featuring even small details. In that study, a higher level of realism was associated with higher credibility ratings as indicated by differences between the ratings for these visualizations. In another study, Skulmowski and Rey (2021b) demonstrated that pictures of pathogens (bacteria) were rated as more credible than “toon” renderings of the same bacteria. Results such as these could be considered as evidence for a naive trust in realistic graphics in line with the assumptions of naive realism. However, instead of merely dismissing this feature of human cognition, this aspect should be noted and could be of critical importance in situations in which a learning task also has a persuasive component. However, it is plausible that these emotional effects can also bias other judgments concerning learning materials. For instance, different types of visual representations have resulted in illusions of understanding (Wiley, 2019), an effect that could potentially be influenced by the realism of visualizations. As can be seen from the studies summarized in this section, the credibility of visualizations critically depends on believable shading and physically correct rendering. Naturally, the depicted models also need to have correct proportions and plausible shapes, but appropriate textures and rendering styles are a must in order to create a credible visualization.

In sum, the realism of a visualization or virtual environment can have profound effects on learners’ responses. Furthermore, these effects suggest that at least for some psychological effects to occur, a certain level of realism is required.

Discussion

As we have discussed in this paper, a preference for realism does not necessarily need to be naive. Realistic visualizations offer a multitude of desirable effects on learning that can be harnessed using emerging instructional technologies. However, our paper clearly shows that a blanket assumption of a superiority of realism indeed cannot be confirmed. In order to fully comprehend how and when realism can be helpful, we argue that a fine-grained analysis of the components of realism (geometry, shading, and rendering) and an understanding of how each of these components can contribute towards specific learning tasks is important. As we have analyzed in this paper, several recently discovered functions of realistic visualizations such as signaling, credibility enhancements, and the potential to act as retrieval cues, are closely linked to specific properties of these three realism components. Hence, our GSR model can be a useful starting point when determining the functions that a visualization should have during learning and how to implement them in the design.

Building upon this rather technical classification of realism components, we have presented a cognitive model of learning with realistic visualizations. In this model, the information conveyed through the GSR facets is perceived by learners. This perceptual step is affected by the perceptual load generated through the GSR components. More details, surface details, and realistic rendering options can manifest themselves as more elements that need to be kept in working memory. Depending on learner characteristics such as spatial ability, these elements can either be integrated into a schema in long-term memory, or they end up as distracting cognitive load that overburdens learners. Finally, different learning tests extract different information from these schemas. While retention tests have been shown to rely on details as cues, transfer tests benefit from more flexible and less concrete mental schemas. It needs to be noted that the presentation of realistic visualizations works in tandem with additional processes, such as generating the feeling of immersion and stereoscopy; two aspects that can contribute towards learning.

As Smallman and St. John (2005) summarize, people do not form exact mental representations of every minute detail that a realistic visualization may contain. Various areas of psychology have shown that the brain operates using a parsimonious approach that is aimed at only storing just enough information to solve a given task (e.g., De Pretto & James, 2015; Glassman, 1985). This aspect is critical for the understanding of how and when realistic visualizations can help learners. As has been discussed in previous papers (e.g., Skulmowski & Rey, 2021a, b), for some learning tasks, detailed knowledge of important surface shapes, their material properties, or both may be essential. Despite the plausibility of the warning of naive realism to acknowledge that mental representations are rather loose approximations of selective excerpts of our surroundings (Smallman & St. John, 2005), we argue that in many circumstances, even parsimonious and incomplete mental representations gained from realistic visualizations may contain crucial information needed to recall or identify an object or scene. Put simply, even a parsimonious representation gained from a realistic visualization might still contain useful information that will be helpful for learners. However, as we have seen in this review, this helpfulness of a potentially richer representation hinges on the learning task. To sum up, Smallman and St. John (2005) provide a compelling case for the claim that mental representations are only parsimonious approximations of reality. However, we emphasize that even when only simplified representations are being formed, the information contained in realistic visualizations can be of critical value depending on the learning task.

Future research should focus on more fine-grained comparisons of realism levels using a wide array of different tests to more precisely capture the advantages and disadvantages that realism can have on learning. From the literature reviewed in this paper, it becomes apparent that shading and rendering are most often used to vary realism in studies. The effects of the geometry dimension (including variations in the overall shapes as well as smaller details) need to be studied more closely. However, details generated through shading and realistic rendering have been found to help learners in retention tasks. The effects of the GSR dimensions on transfer tasks and more abstract understanding in the learning of surfaces still needs to be examined in more detail. Learning tasks that involve processes, including concrete shapes and their abstract transformation or dynamics show a mixed result pattern. While some studies revealed null results of realism on transfer and comprehension tasks, other studies on processes resulted both in positive and negative effects. An explanation for these contradicting effects may be a mismatch between the cognitive load induced by realism, the cognitive processes it triggers, and the assessment method (Skulmowski & Xu, 2021). While a realistic rendering may help learners in their construction of a mental representation of the surface of objects, it may entail a cognitive cost that reduces learners cognitive capacity for understanding the abstract processes depicted in these visualizations. If the ensuing test focuses on the process rather than surface details, this cognitive misalignment results in a disadvantage of realistic visualizations (see Skulmowski & Xu, 2021). Therefore, future research that considers the fine-grained aspects of realism and testing methods needs to be conducted. Lastly, learning tasks that involve abstract knowledge or computations do not seem to benefit from realism. It should be investigated whether subtle forms of realistic details harm learning as much as the more drastic comparisons usually found in the literature reviewed in this paper. In addition, points of reference for the GSR dimensions could be established that would enable a more precise quantification for comparisons between studies.

Two general guidelines can be condensed from the reviewed literature: (1) avoid realistic visualizations when cognitive load is already very high and when a task is primarily transfer-oriented. (2) Consider using realism for learning tasks with a strong visual focus, in cases in which schematic visualizations offer too little guidance, for learners with high spatial abilities, and in situations in which credibility is critical. If more specific guidance is needed, the GSR dimensions and their effect as presented in the cognitive model of learning with realistic visualizations need to be considered.

Most of the recommendations on the use of realism discussed in this paper have been derived from studies using static or animated visualizations that the study participants viewed on standard computer screens. However, a very important question is whether these effects can be transferred to emerging technologies such as virtual reality and augmented reality. It is plausible that some of the effects presented in this paper will be particularly helpful for such learning settings. For instance, considering that virtual reality can induce additional cognitive load through the feeling of immersion (Frederiksen et al., 2020), it is reasonable to assume that signaling may be of particular importance in virtual reality learning environments.

A highly important future task is to disentangle the underlying causes and relations between the positive effects of realism presented in this paper. For example, can realism be used as a means of signaling because realistic visualizations are more credible and thus support learning? Such questions should be answered to be able give even more targeted design recommendations for digital learning. Furthermore, more research is needed to ascertain how different levels of realism can be used in a single learning task. For instance, some research has been done on concreteness fading in the context of realism, (i.e., lowering the level of realism during a learning task, e.g., Fyfe et al., 2014). As some researchers hypothesize that a progression from realistic towards schematic representations could foster learning (e.g., Fyfe & Nathan, 2018) while others found no benefits of such a strategy (Scheiter et al., 2009), further research on this aspect is needed.

We hope that this discussion can contribute to a more balanced perspective on realism in digital learning by presenting potential uses for realistic visualizations and ways to minimize cognitive load. Importantly, the results summarized in this paper need to be considered in the context of specific tasks and their demands as well as the learners and their abilities. Based on the results reviewed using our cognitive model of realism, it is evident that overly general statements concerning the effectiveness of realistic visualizations may be misleading. Instead, we should be concerned with the effects of the different facets of realism as they act on learners under specific task circumstances. Additionally, the GSR model suggests several combinations of visualization components that have not been widely investigated.

While some readers may find the GSR model rather technical, we are convinced that researchers studying the effects of realism should consider the individual components of realism, their role in the production process of visualizations, and their effects on learners. We argue that a basic understanding of how realism is achieved using visualization software is essential to conduct more comparable and well-defined studies.

As our lives are destined to be enriched with a growing number of contacts with virtual and augmented realities (Slater et al., 2020), the need for differentiated perspectives on realism becomes increasingly apparent. While assuming that realism is always the better option can certainly be said to be a naive point of view, dismissing the positive effects of realism is equally an overly generalized stance.