Introduction

The increasing use of dynamic multimedia construction as an assessment task is based on a dual aim to improve science content knowledge and develop multimodal literacies (Hoban, Nielsen, & Shepherd, 2016). These two aims have long been linked in the science education literature, as it is understood that working with and constructing scientific representations is central to learning (Tytler et al. 2014). This is because how meanings are made in scientific discourse is considered fundamental to the epistemic processes through which knowledge is generated, validated and communicated in science disciplines (Tytler et al., 2018). In science, the different modalities used (such as images, scientific symbols or language) serve to represent different aspects of scientific understanding (Lemke, 1998) and offer opportunities for exploring concepts, conducting experiments, recording findings and disseminating understandings. This functional relationship between scientific enterprise and the modalities used signal that learning how to use representations in science is a means through which individuals learn how to reason and theorize in science (Gooding, 2006; Latour, 1999). Accordingly, science educators are increasingly seeking to foster the development of multimodal disciplinary literacy (see, for example Airey & Larsson 2020; Linder et al., 2014; Tang et al., 2014; Tang & Danielsson, 2014; Tytler et al. 2014).

Although we are aware of the need to work across multiple scientific representations in assessment tasks, the potential of multimodal environments has not yet been fully realized.

Traditionally, the focus in education has been on the more established disciplinary modes, such as language. As technologies develop, particularly those that enable dynamic representations, this focus has broadened to include other modes, including image (moving and still) (e.g. He, 2020; Unsworth, 2020) and gesture (e.g. Lim, 2011, 2019). Explicit references to multimodality are now more commonplace in curricula. For instance, in Australia, students studying science from the late primary school level are expected to ‘Communicate ideas, explanations and processes using scientific representations in a variety of ways, including multi-modal texts’ (Australian Curriculum, Assessment and Reporting Authority [ACARA], 2017) and university science students ‘gather, synthesize and evaluate information from a range of sources and to communicate science to a range of audiences for a range of purposes and in a variety of modes’ (ALTC, 2011, p. 14).

The field of research that considers multimodal communication, ‘multimodality’, is still in its infancy (Bateman et al., 2017). Although it is described as a field that draws from a range of different philosophies, it is heavily influenced by social semiotics, which posits the close relationship between texts and their social contexts. Though the central assumption of meaning making as a social practice is common, it manifests differently in different fields. For instance, in science education, social semiotics has been used to describe the use of representations in the discipline of physics (Airey & Linder, 2009). Theorizing in this space is particularly useful for science educators because specific disciplinary characteristics, such as the concurrent use of modes such as mathematical equations and diagrams, are discussed. Another branch of social semiotics stems from systemic functional theory (SFT) (Halliday, 1978). Most fully developed for language, SFT focuses on the function of language (or other modes) and considers meaning making as involving choices made as part of systems of possible options. Systemic functional theory, together with genre theory, has informed current approaches to multimodality (Bateman et al., 2017; Kress, 2010; O’Halloran, 2011)

Given the increasing use of student-generated digital explanations as assessment in science education (Hoban, Nielsen, & Shepherd, 2016), there is a pressing need to better understand these products, both in terms of their construction as a meaning-making activity and their pedagogic potential. We build on the concept of cumulative semiotic progression (Hoban & Nielsen, 2013) to ask the question: How does a social semiotic account of digital media production help us to understand meaning making by a preservice teacher?

Background

Digital Explanations as Assessment

Digital explanations are dynamic multimodal texts created by the learner to communicate science meanings (Nielsen & Hoban, 2015). These objects are referred to using a range of different terms depending on form and media, but the type we are considering is characterized by being student generated, multimodal and dynamic. Typically, this looks like a 3–5 min, stand-alone, digital, dynamic product employing narration, image and language on screen, with the possibility of animation and video (film) also being included (Nielsen et al., 2018). There are a number of different techniques that can be used in the creation, all of which are based on the ubiquity of personal, web-enabled digital devices such as smartphones or tablets and readily available media production software such as iMovie or Explain Everything. The ubiquity and ease of use of contemporary educational technologies have unlocked potential for new and more sophisticated uses as assessment items (see, for example, Bennett et al., 2018; Mayer, 2009; Prensky, 2001). This is particularly important in science, and in preservice teacher education, given the inherent multimodalities of both science and the primary classroom.

Digital explanation developed from an earlier technique called ‘slowmation’ (Hoban, 2005, 2009, 2020; Hoban et al., 2011). Slowmation is a form of stop-motion animation where the learner plans a sequence of representations through storyboarding and model making. The models are then moved manually while taking a sequence of still images and uploading them into a movie-making programme. The sequence is then played at 2 frames per second and narrated. The result is a stand-alone animated explanation of a science concept or process that aims to help the viewer visualize the ideas and make meaning. Hoban’s science learners were preservice primary teachers who, while developing technical skills in creating a digital artefact, also developed conceptual understanding of the science content (Hoban & Nielsen, 2013; Nielsen & Hoban, 2015). Similar results with slowmation have also been demonstrated across a range of learning contexts and in many different content areas: secondary science teachers in their teacher education programmes (Amos & Campbell, 2016; Keast & Cooper, 2016; Kidman et al., 2012; Paige et al., 2016); Aboriginal ways of knowing (McKnight et al., 2011); high school students studying geology (Mills et al., 2019, 2020) and preschool children learning about science concepts (Fleer & Hoban, 2012).

Recent work on digital explanations reveals significant potential for learning (see, for example Mills et al., 2019; Paige et al., 2016). In addition, results from a range of studies show that there might also be potential to improve communication skills, collaboration and engagement (Hoban & Nielsen, 2014; Nielsen et al., 2017). However, multimodal texts are challenging to construct, assess and teach (Jones et al., 2020). Researchers in this area highlight that the use of digital explanations (or learner-generated digital products more generally) has occurred without a substantial assessment framework and lacks theoretical underpinning (Reyna & Meier, 2018). Constructs from the field of social semiotics are thus helpful if we hope to realize the potential of digital explanations.

Social Semiotics

Social semiotics has proven useful in the fields of multimodality and science education, which is where this particular research lies. However, social semiotics is not a uniform theoretical framework, but rather, different traditions have adapted to different objects of study. In this paper, we draw from a range of concepts from science education, linguistics and multimodality to describe the meaning making and its relationship to learning in preservice primary teacher education.

We begin with the notion of the cumulative semiotic progression (Hoban & Nielsen, 2013) which identifies the process of constructing a multimodal product as constituting a number of stages (e.g. research notes, storyboard, models). Hoban and Nielsen argue that throughout these stages, meaning is made through the use of different ‘signs’. Key implications from this theorization include that, first, the signs used in each stage have particular potential for different kinds of meaning making and, second, that the process of recontextualization of meaning across the different signs in these stages results in improved reasoning and understanding.

The potential for meaning making by the different modes, described above, is captured in the concept of ‘affordance’. Here, we use the definition offered by Airey and Linder (2009) within their disciplinary application of social semiotics. Identifying the discipline of physics, Airey and Linder describe learning or development in the discipline as having access to the ‘disciplinary ways of knowing’, which is achieved through familiarly with the discourse of the discipline. The discourse, in turn, involves tools, representations and activities. Each of these aspects of the disciplinary discourse can occur in different modes, which the authors identify as spoken language, written language, image, gesture, apparatus and so on. Airey and Linder explain that there are different facets to the ways of knowing in a discipline that can only be achieved through the use of multiple modes. They explain that all facets of knowing cannot be expressed monomodally and theorize that each way of knowing is achieved by a ‘critical constellation of modes’: a ‘minimum’, specific combination of the modes detailed above that gives a learner holistic ‘access’ to the disciplinary discourse and thus, ways of knowing. This critical constellation differs for different concepts, and, once fluent in the critical constellation, learners can readily employ any other range of individual modes.

We can thus see how the process of constructing a multimodal resource (such as slowmation, as a multimedia product) provides an opportunity for a learner to achieve a critical constellation of modes, and here, we draw on more recent research in social semiotics to more precisely describe this process. In generating representations, learners must construe scientific meanings and then translate the meanings into different forms, which is where the learning is theorized to happen (Kress et al., 2001; Kress & van Leeuwen, 2006). Translation has technical meaning here and is used as a general semiotic term to capture changes in meanings from one mode to another that result from having to work in a new medium or for a different audience. The concept of ‘transduction’ refers more specifically to ‘meaning material’ being moved from one semiotic mode to another, for example the kinds of changes that emerge in moving from a written explanation of a process (language as a mode) to a diagram that represents that same process (image as a tool) (Bezemer & Kress, 2008). Bezemer and Kress’s idea of ‘epistemological commitments’ is also useful here, in terms of illustrating the additional work required to translate or transduct meaning and further explain that different modes impose a different set of such commitments. Transduction of meaning from written to visual modes, for instance, might require additional detail, such as specific placements, colours or addition of other representations. Such commitments demand the attention of the author or creator of the digital product too.

Svensson and Eriksson (2020) also made a distinction between the disciplinary and pedagogical affordances of the semiotic resources (Airey and Linder’s ‘modes’), bringing in a consideration of the audience. Depending on the context of their use, the resources may not reflect ‘traditional’ representations in the sense that the disciplinary representations may need to be modified to better suit the intended audience (see also Airey & Linder, 2009). Thus, when generating a digital explanation, how a particular learner selects resources may form part of a transductive process as meanings are translated from one representational form to another. Selecting and modifying multiple representations are thus of interest in order to better understand the meaning-making potential of this form of media production.

In addition to the meaning made by the creator, we also consider the multimodal literacies used to address an audience. In order to thus describe the purpose of the text, we draw on systemic functional linguistic theory (SFL) (Halliday & Hasan, 1976; Halliday & Matthiessen, 2014) together with genre theory (Martin & Rose, 2008). Genre theory describes the different patterning of texts according to their social purposes and has impacted curriculum literacies making it possible to identify the major genres associated with disciplinary practices (Martin & Rose, 2008). Explanation genres are important in communicating different kinds of scientific processes such as sequencing (cyclical and sequential explanations), expressing causation (causal, factorial and consequential explanations) and describing systems (system explanations) (Derewianka & Jones, 2016; Martin & Rose, 2008). A causal explanation, for example, consists of the phenomenon identification (identifying the relevant concept) and one or more explanation sequences (linking cause and effect). Different kinds of explanations are recognizable as ‘recurrent patterns of meaning’ (Martin & Rose, 2008, p. 231) so that it is possible, for analytical and pedagogical purposes, to expect to find distinct stages through which an instance of a genre or text unfolds. Combinations of genres are common in the digital world as evidenced in websites, simulations and other digital artefacts such as videos. In the disciplinary contexts of higher education, texts are expanded through embedding and combining these different genres (Szenes, 2017) and learners build an understanding of these systems through countless instances of listening, speaking, writing and reading texts. Furthermore, genres configure three dimensions of context: fields of social activities, tenors of social relations and modes of meaning making (linguistic, visual, spatial, aural and gestural). Collectively, field, tenor and mode comprise register and patterns of register are realized as patterns of meaning in the text. The purpose of the explanation (genre) under focus in this paper is to explain a science concept (field), to children (tenor) drawing on a range of semiotic resources available through digital media (mode).

Method

We adopt an interpretive approach to characterize the meaning making, learning and communicative elements of the construction of a digital explanation. We use case methods (Stake, 2005; Yin, 2017) to present a study of one preservice primary teacher (PST) who constructed a digital explanation artefact start-to-finish during a 3-h research period. ‘Stacy’ (pseudonym) had previous experience with ‘slowmation’ and multimedia production having created a slowmation in a science methods class in year 1 of her degree programme. The methods class included workshops and assignments to help PSTs develop techniques for digital media creation. During the pre-interview, Stacy was tasked to create a digital explanation about the concept of ‘transparency’ for children in years 5 and 6 (ages 11–12). We assigned the topic of transparency for the digital explanation because it is a familiar but complex concept and is part of the primary science curriculum. At the time of this study, Stacy was mid-way through the second year of the 4-year Bachelor of Primary Education degree programme. During the pre-interview, we asked questions such as what do you know about the topic? How confident are you with your knowledge of the topic? What experience have you had with digital media production? What are your plans for creating this digital explanation? Which keywords will you use to search for information? We also asked Stacy to talk aloud as she worked.

To support the construction, we provided a range of models and construction materials: scissors, coloured paper and plastic sheets, pens, plasticine, etc., as well as a small whiteboard and assorted equipment, such as ‘optics kit’ materials, including concave and convex lenses of different sizes, as well as mirrors, prisms, a light box and flashlights. The study was conducted in a sound studio on the university campus and the construction period was video and audio recorded. The audio recording was transcribed verbatim. Interviews with Stacy before and after the construction period were also transcribed and artefacts she generated during the study period were collected. Additionally, still, photographs were taken throughout the study period including workspace layouts, websites she accessed and other salient details of her construction activities. The resulting digital explanation as well as a summary of the multimodal transcription of the construction period are both included as Supplementary Material for this paper.

We acknowledge that a single case is a limiter to wider generalizations; however, the wide range of data sources for the case offers an unusually rich opportunity to examine how these many resources were used by this preservice teacher to build meanings. Focusing on one or a small number of texts or artefacts allows detail and depth as well as richness to analysis and descriptions and is not uncommon when studying meaning-making processes or representations (see, for example Jamini, 2011; Morgan & Kynigos, 2014; Unsworth, 2006; Zappavigna, 2016). The depth provided in this single case allows us to look at how relevant theories inform our understandings in the rich spaces of multimodalities, multiple representations, scientific explanations and the digital environment where student-generated products like digital explanations are artefacts of learning.

Analysis and Results

The research aims to describe the links between meaning making, learning and communication of a scientific concept. There were two key steps in the analysis. Firstly, we used genre theory to identify the stages of the digital explanation that Stacy generated, including the phenomenon identification (which includes a definition), two explanation sequences, rounded out by a concluding statement. The stages also serve to organize the multimodal transcription of the construction period and presentation of the multiple representations she accessed or generated during the study period. Table 1 presents the transcription of the digital explanation, with time-stamped stages, that ran for 1 min and 28 s in total.

Table 1 A staged multimodal analysis of the digital explanation
Full size table

The transcription in Table 1 characterizes the digital explanation that Stacy created in terms of the purpose of the different stages in the genre, alongside a screenshot from that stage and our generalized multimodal analysis of visual and aural elements, similar to the analysis done by Tan (2014). The videographic elements are further characterized in terms of the static and dynamic representations used. An additional visual element is ‘on-screen text’, which was typically overlaid as a title or a label. In Stacy’s digital explanation, a scripted narration comprises the aural element.

The second step in the analysis process involved exploring, in more detail, the context of the construction period where each of the stages was developed more expressly, in terms of how Stacy selected and worked with representations. The full summary of the multimodal transcript of the construction period is found in Supplementary Materials, while Table 2 presents a part of this analysis. The construction period transcript includes timestamps, images of websites Stacy visited as source material and interpretive commentary for the segment. The quotes in red are phrases that Stacy read directly from the source material she accessed and in the case of Table 2, the sequence displayed reflects her refining a key definition for the digital explanation. We aim to show how the many moments reflected in Tables 1 and 2 are consequential for the dual aims of Stacy’s learning about science and communicating the science content in the digital explanation.

Table 2 A multimodal analysis of the construction period transcript
Full size table

Through analysis, we thus consider how Stacy built topic-related meanings through her representational choices (field); how she oriented these choices to the needs of her young viewers (tenor) and, then, how she organised the various semiotic resources into the digital explanation (mode). Interview data complements the analysis of the artefact. We draw on the identified range of concepts from social semiotics for this work, examining each of the key informational sequences from the digital explanation she produced to understand how she chose the representation(s) and for what purpose.

Preparation for Construction

The task to create a digital explanation on the concept of transparency for children aged 11–12 was explained to Stacy during the pre-interview, and she indicated that she didn’t know very much about the concept. She noted that she would begin the task by conducting internet searches for information. Initially, she conducted a google search using the keyword ‘transparency’, but she also searched for ‘experiments with kids’ realizing that many of the definitions she found in her initial searching were too advanced for the intended audience of years 5 and 6 children.

The construction period transcript foregrounds the recursive nature of Stacy’s production process as she moved backwards and forwards between the representations as she developed her reasoning about the topic. Rows 5 to 11 in Table 2 illustrate this recursive reasoning effectively. In this sequence, we see Stacy moving between different representations as she starts to build her understanding of the concept—potentially the start of the development of a ‘critical constellation’ (Airey & Linder, 2009). She looks for further commitment in the images to understand the technical ‘scientific’ meaning of transparent (and the process of light transmission) and be ready to teach it.

Early in the sequence, Stacy is still uncertain of the relationship between transmission and transparency despite the initial definition on the ‘Science Sparks’ website linking the two congruently: ‘the reason we can see through transparent objects is that they transmit light without scattering it’ (row 5 in Table 2). By row 10, Stacy is still trying to reconcile the definitions of transparency, her reading about how light travels and the representations of the process as stylized in Fig. 1. She used this image to understand the difference between absorption, scattering and transmission.

Fig. 1
figure 1

Differences between scattering, absorption and transmission

Full size image

The image in Fig. 1 is highly abstract with minimal detail committed visually and requires viewers to decode the meanings represented; notably, it is likely that she has not been taught to do this. Although the diagram does show how light interacts with different materials, Stacy struggles to make the connection between ‘transparent’ and ‘transmission’. She continued searching the internet and making notes for about 45 min before she began choosing resources for parts of the digital explanation.

Selecting resources for inclusion into the digital explanation parallels the pedagogical work teachers do in preparing to teach science in their classrooms. For example, Stacy identified electron excitation or atoms scattering as concepts that were too advanced and thus inappropriate for the digital explanation she was tasked to make. Interestingly, while she noted information that was too advanced for children, she did not seem inclined or curious to pursue a higher level of understanding of the concepts for herself, focusing instead on meeting her aims for the task at hand. Beyond selecting conceptual content, Stacy was also aware that visual, dynamic and auditory elements contribute meanings in the digital explanation. Late in the pre-interview, she googled ‘diagram of light scattering through see-through object’, commenting on what she was looking for:

I want like an image. It says, ‘Light travels in straight lines, transparent objects let light pass through them, translucent objects let some light pass through them, opaque objects do not let any…’ I could do like a progression of that.

The quote corresponds to line 6 in Table 2. As she initiated her search and clarified the phenomenon for her digital product, her sense of the required or appropriate stages of an explanation was clear. At 42:25 of the construction period, she stated:

I’ll probably start with ‘What is transparency?’ And then give a basic definition and that will be a slide I think with a voiceover and maybe some pictures, and then…. I might just write out things I definitely want to include and then I’ll think of the order. So, I want [a] definition, I want a diagram of transmission of light passing through, then I’ll have a video maybe something like ‘think about the things in the world that you’re able to see through, the things around you that you’re able to see through’.

Consistent with the expectation that the product should include a range of representations (Nielsen et al., 2020), Stacy’s plan for the digital explanation included identifying examples or suggestions for the different stages of the product (see Fig. 2 for her planning sheet).

Fig. 2
figure 2

Stacy’s planning sheet

Full size image

Phenomenon Identification

The introduction sequence of the digital explanation shows a videographic display of an open-source, colour, still image of a magnifying glass eyeing a field of sunflowers with the on-screen text overlay ‘transparency’ as a title (see Table 1, line 1). The sequence lasts for 14 s and the narration includes a simple definition: “Transparency is when an object allows all the light to pass through it in a process called ‘transmission.’ Other words we might use to describe something transparent is ‘clear’ or ‘see-through’”. The definition is elaborated with two simple synonyms for ‘transparent’. The brief introduction was scripted during her early work in searching websites to find definitions at a suitable level of technicality for primary school children and her work to set up this stage involved field building of her own knowledge.

As an introduction, the narration is informative and sets out the topic as would be expected in an introductory sequence. In showing a magnifying glass, the image illustrates the concept of ‘see through’ even as the narration makes no explicit link between the visual element of a magnifier and the definition as narrated. The image relates the concept to the everyday lived experience of children, which is theorized to help them understand the technical nominalization and split the semiotic load between the visual and aural elements (Painter et al., 2013).

Explanation Sequence 1: Slowmation

Following the opening title and phenomenon identification sequence, the key definition for the digital explanation is presented as a ‘slowmation’ (Hoban, 2005, 2009) that lasts for about 9 s. Visually, the simple slowmation sequence consists of 15 images that Stacy drew on a small whiteboard. Table 1 (line 2) shows two screenshots from the slowmation sequence. A stylized ‘torch’ sits on the left edge of the screen and yellow horizontal lines advance from the torch toward the ‘transparent object’ (shown in the middle of the screen as a vertical green-hashed column) with light ‘rays’ passing through, emerging from the right side of the object. The narration at time 00:14 complements the visual imagery: ‘Here we see the process of transmission as the light from the torch hits the transparent object and passes through the other side’. In this case, Stacy uses a simple definitional sequence to demonstrate the abstract process of light transmission.

The slowmation sequence was based on a still image clipped from a YouTube video called ‘Translucent, Transparent and Opaque’ (Alberta Distance Learning Centre [ADLC], 2020). Stacy commented on choosing the ADLC video, even as she had viewed quite a few others in her internet searching: ‘This is probably the most helpful video I’ve seen so far because it has diagrams and a clear explanation’ (0:47:31, construction period). The ADLC educational video was produced for elementary school children and Stacy returned to it many times through the construction period. The main image she referenced is shown in Fig. 3, which depicts light passing through a transparent object. The stylized sketches she drew on the small whiteboard simplify the ADLC image. Furthermore, she used the general plan from the ADLC video to structure her own digital explanation. Notably, the slowmation illustrates the same process (light passing through a transparent object) and uses the same orientation and labels as the ADLC video.

Fig. 3
figure 3

Screenshot from Alberta Distance Learning Centre (used with permission)

Full size image

While the ADLC diagram was the source of an idea for how to represent and define the central concept of transparency, Stacy then transformed the information in the ADLC diagram to a slowmation sequence that illustrated the invisible and dynamic process of light transmission. While visually quite basic, the short slowmation sequence illustrates her focus on the key definition in a simple explanation sequence and represents it stylistically and clearly. The slowmation is also an introduction to the subsequent explanation sequences presented next as two short videos in the digital explanation.

Explanation Sequence 2: Video

The two additional explanation sequences in the digital explanation are both dynamic videographic displays (of duration 14 and 26 s, respectively). The first of these videos shows a flashlight as a light source with a glass of water in front to illustrate light passing through a transparent object (see Table 1, line 3). The flashlight is pointed at a glass of water in a darkened room to highlight the light passing through the glass of water as an example of a ‘transparent’ object. The narration focuses the viewer’s attention on the visual elements to exemplify the concept of transparency, representing the same idea, but in a different way, as part of a transductive process. These elements include the intensity of the light (requiring darkness to exemplify), the positions of the torch and glass and the appearance of the light that had been transmitted.

The second short video extends the concept of light passing through a transparent object by showing a non-example of transparency. In this video, the glass of water is replaced with a piece of cardboard in a plastic sleeve (see Table 1, line 5). Shining the flashlight toward the cardboard provided the non-example, as described in the narration (00:37): ‘When the cardboard is inside the plastic sleeve, light is unable to pass through as the cardboard is not transparent.’ With the flashlight still in place, a hand pulls the cardboard from the sleeve, offering another example of a transparent object as the narrator states: ‘The plastic on its own, however, allows for light to pass through, showing that it is indeed a transparent object.’ Notably, in the second video, Stacy chose not to introduce the additional term ‘opaque’, instead focusing on the notion of transparent (or ‘not transparent’), as she described in the post-interview:

I didn’t have to say ‘The cardboard is opaque’ and then explain what ‘opaque’ means. So I don’t think it was 100 percent necessary but I thought it was good to show the difference between ‘transparent’ and ‘opaque’, with or without the terminology.

Prior to filming the videos, Stacy modelled her set-up on the examples seen in the ADLC video and noted: ‘I’m just seeing how the light goes through the different objects and trying to see what would be a good demonstration to film for my video’. When she filmed the video sequences, Stacy used slightly different materials to those in the ADLC video but combined them similarly to show what transmission looks like in ‘real-life’. By choosing a video for these parts of the explanation, Stacy demonstrates some level of awareness of a key affordance of video as a medium that can illustrate processes that are not observable to the naked eye. This sequence also builds on the slowmation that preceded it, including how the flashlight is oriented and the direction of travel of the light through the transparent object. By including a non-example, Stacy builds complexity to the meanings represented. There is also a temporal element in Stacy’s modal choices where real-time displays are complemented with examples shown in slow motion: the videos provide real-life examples while the slowmation slows down the process of transmission through the stop-motion animation technique (Hoban, 2005, 2009). This ‘slowing down’ is an important affordance of the slowmation technique as a form of representation. Furthermore, as part of the overall digital artefact, the objects chosen as models would be familiar to children of the age range targeted for the explanation.

Examples and Definition Restatement

The final image in the digital explanation is a busy photographic image that includes a number of transparent objects such as water bottles (both plastic and glass), plastic containers, rolls of plastic film and glass ampules (see Table 1, line 6). The image remains on screen for 23 s and the narration at 00:54 links children’s experience and the visual representations in the image: ‘Think about some other transparent objects in the world around you. What about windows, or a water bottle, a plastic bag, a container, sticky tape, glasses, a snow globe, a light bulb, magnifying glass, googles, even air is transparent’. By naming a range of transparent objects, the narration complements and extends the visual element to a number of objects and materials that would be familiar to years 5 and 6 children and speaks to the pedagogic purpose of the digital explanation. In a way, this moves from the stylistic or simple representations used earlier to a different form. The familiarity suggests to children that they could also try the same thing on their own, which we note is a common feature of science videos for children.

With 11 s remaining in the digital explanation, the screen goes black and the narration concludes with a restatement of the key definition (Table 1, line 7): ‘Remember, transparent objects are those which light passes through. These are the things that we are able to see through. Bye!’ As a concluding sequence and by way of summary that restates the key definition, both the narration and the final image are elements that offer a common-sense description along with familiar examples that reinforce the initial definition of ‘transparent’. While planning for the summary sequence and aiming to finish the digital explanation with an ‘interesting ending’, Stacy commented late in the construction period (2:24:26): ‘my idea for this is just to get heaps of images and have them as a collage on the page but each image popping up one at a time.’ While the final image does not do this more technically challenging ‘popping up,’ restating the definition serves as a concluding marker for the digital explanation.

Working with Multiple Representations

In this section, we provide more detail around the translation and transduction across modes and representations. The representations Stacy accessed guided both building her field knowledge of the concept of transparency and a plan for the digital explanation. At some point of her searching for information, she identified one representation that ‘worked’ for her and she used it as a source definition for the product (the ADLC image), choosing it among the range of other representations she had accessed. The range of sources among the possible representational forms that Stacy accessed could be considered as forming part of an apparent ‘critical constellation’ of modes (Airey & Linder, 2009); we say ‘apparently’ because it was limited to ‘ways of knowing’ at a less advanced level. As summarized in Table 3, Stacy used the source definition and diagram from the ADLC video to carry forward meanings into other representations that she constructed. She revised both the definition and the ADLC diagram to nominalize the word ‘transparent’ to ‘transparency’ and elaborated the definition to include everyday words to help clarify meanings. She used the dynamic medium of slowmation to illustrate the process of light transmission. She then expanded the definition through intentional changes to the language used in the narration complemented simply by illustrating light rays passing through the transparent object in the slowmation sequence. Notably, the source diagram as a static image was transformed into a dynamic process shown in the slowmation sequence. As a way to ‘unpack’ meanings, this transformation is significant since it helps to make the invisible visible (Olympiou et al., 2013). During the post-interview, Stacy commented on why she made this transformation:

I think the slowmation is good because it doesn’t just show the still image of the diagram which I was initially going with; it actually shows the audience the physical process of what the light is doing as it travels through.

Table 3 Stacy’s transformations of the key definition and visual referent for ‘transparency’
Full size table

The animation of the process arguably foregrounds the process of transmission, helping to distinguish it from the material property of transparency, which was represented by still photographs.

Stacy is aware of the affordances of the slowmation as a representational form and uses this to bridge the more technical definition and abstract concept from the source material for the intended audience for her digital explanation.

Potentially, any representational form can be transformed in multiple ways and in Stacy’s case, she modified the definition and images to support viewer meaning-making, but this also helped her to better understand the process of light transmission. Through the process of building meanings for the intended viewer, her choices to transform representations illustrate how she resolved contradictions, more specifically, in terms of which parts of the concept to represent and which representational form to use to make a given point.

Many of Stacy’s decisions for selecting or transforming the representations she used in the digital explanation were governed by an aim to appeal to the audience of children. She emphasized this appeal to the audience explicitly in her comments about language choice while displaying images, video or slowmations, but also implicitly in her choice of words in the nominalizations made to modify terminology, also for the intended audience. The significance of explicit or implicit attention may be an interesting direction for future research.

Discussion

Constructing a digital explanation is a meaning-making exercise that has been associated with improved disciplinary understanding (Nielsen et al., 2018; Nielsen & Hoban, 2015). Although we understand the importance of working with disciplinary literacies and multiple representations, the complexity of the multimodal environment means that we are still coming to understand learners’ meaning making during the design and construction of multimodal products, particularly dynamic texts, and also, how this is related to the understanding of both a text’s purpose and its register demands. In this paper, we draw on a range of concepts from social semiotic theory to understand the process of meaning-making, and its role in learning content knowledge and acquiring specialized literacy skills.

We first consider the role of the maker’s understanding of the context in terms of the genre at stake. In the product, Stacy adheres to the expected genre of the scientific explanation. In genre theory, the explanation genre employed is ‘causal explanation’ and generally consists of phenomenon identification (e.g. ‘transparency’) and one or more explanation sequences. While genre theory has often been used in literacy pedagogy to understand expert texts and mostly in terms of written language, it is also useful in the context of this dynamic text. Here, existing online texts representing the target genre acted as models for Stacy to construct her digital explanation. Although not stated explicitly by the preservice teacher, it was implicitly understood that the digital explanation text required a particular structure and that repetition was important. This highlights the need for high-quality multimodal model texts, particularly in the classroom, since as Maton and Howard (2021) have warned ‘not all multimedia are created equal’ (p. 76). In other words, some source material may be better (or worse) suited to the pedagogical purposes in a classroom or as models for digital media creation.

The SFL register variable of field was also important. Field plays a significant role throughout our analysis. We consider two aspects of the field that are relevant here. First, we note the development of Stacy’s field knowledge in the pre-construction phase of the product design. Stacy plans and undertakes research to reach a level of sufficiency of understanding of the concept of transparency, which she admits is a concept she initially knew little about. The process involves accessing a number of different sources of information online and an intentional search for specific representations, such as diagrams. Stacy reaches sufficiency when she reaches the understanding expected of a younger audience, dismissing information considered to be too ‘advanced’. Although it is not clear whether this represents a ‘critical constellation of modes’ (Airey & Linder, 2009), there is a distinct point at which she believes she has understood enough, and the similarity to the notion of a critical constellation of modes is striking.

Second, the field knowledge represented in the product is also pertinent. We note that Stacy adopted a range of accepted disciplinary representations. She has appreciated the importance of using technical terms appropriately (e.g. transparent, translucent, opaque) and has understood that a range of disciplinary representations is also required (e.g. diagram, experiment). In the talk-aloud protocol and interview, Stacy explained that it was in fact necessary to include a diagram and from the outset of the study period searched for one that she deemed appropriate. There is also an understanding here of the affordances of different representations. Stacy explained how the experiment (shown in the two short video sequences) displayed ‘what actually happened’ when light interacted with a transparent object, while the slowmation was more useful in showing the ‘process’ of transmission, which couldn’t be seen otherwise. Furthermore, making the slowmation was an important impetus for Stacy to really make sure that her own knowledge was secure. She could have used a ready-made image to show the relationship between transparency and transmission but this would likely have remained quite abstract, whereas the need to show it dynamically prompted her to further interrogate what was going on since the maker needs to commit more meaning-making options in a dynamic semiotic mode such as slowmation.

Ultimately, when preservice teachers develop a digital explanation, they are doing so to produce a learning resource for children. Thus, aspects of tenor include the particulars of age appropriateness (of the individual representations, at a suitable level of technicality) but also moving between what is deemed familiar/unfamiliar for the audience of children. Throughout the digital explanation, and with each of the representations Stacy chose, there is clear attention to the audience through the use of familiar objects presented visually alongside a narration with age appropriate language. The purpose of the digital explanation task as an explanation for children supports this pedagogical emphasis, which is a highly relevant aim for activities in preservice teacher education since teachers must also choose suitable materials and resources for use in their classrooms while attending to content complexity, modal affordances and relevance for the audience. While earlier research in the area of slowmation and digital explanation made claims about PST learning through the construction process (Hoban & Nielsen, 2013; Nielsen & Hoban, 2015), in the current case study, our analysis has shown Stacy’s efforts to build meanings for the intended audience as part of her pedagogic purpose to explain the concept. This seems to work in tension with the underlying desire by science educators to foster PST learning through tasks such as digital explanation and is a possible constraint on how PST develop representational competence (Volkwyn et al., 2020). In other words, when PST focus on what the audience needs to know, their own learning (about the topic or concepts) may be limited to that level of understanding. On one level, this may be reasonable given primary school teachers’ need to focus field knowledge at a suitable level of abstraction when teaching children. However, if our aim as science educators is to have PST work with multiple representations so as to learn the science content, we need to find ways to push PST to advance beyond a year 5 or 6 level of understanding.

Finally, we consider the meaning-making process. As has been argued elsewhere, meanings are made through the modes of the ‘cumulative semiotic progression’ (Hoban & Nielsen, 2013). Here is where multiple transductions shape field-related meanings both for the creator and for the intended audience. Stacy followed the basic steps outlined by Hoban and Nielsen: ‘background notes’, ‘storyboard’, ‘models’, ‘digital photos’ and ‘narrated animation’, but these are not linear steps and each has different affordances for meaning-making. Coordinating among these different representational systems requires some level of technical skill in working with objects in different modes, but the semiotic work is in transforming them so that the intended meanings are built through the arranged sequence in the overall explanation. Transductions between and across different representations help to build the explanation and involve considering meanings for individual representations, but also across the steps of the sequence and how each of the representations contributes to meanings overall—both disciplinary (field related) and pedagogical (tenor related) (Airey & Linder, 2009; Svensson & Eriksson, 2020). Making these ‘epistemological commitments’ appears to play an important part in learning, particularly for conceptually demanding ideas. Importantly, the creation of a digital explanation provides a context to consider these meanings. We see this as similar to the notion of ‘representational competence’ described by Volkwyn et al. (2020) where awareness of the affordances and limitations of different forms of representation are consequential for meaning-making, both for the creator and the imagined audience.

Conclusion

Social semiotics provides a lens to examine meaning-making processes both during construction and in the resulting product when creating a digital explanation. Through the analysis in this paper, we have developed the concept of ‘cumulative semiotic progression’ by showing how a case preservice teacher developed her own field knowledge while translating source images and other online materials in the process of constructing the multimedia artefact. The construction process is not, however, linear because in working with online materials, design choices necessarily involve iterative reasoning about the science concepts being explained, the affordances of different modal forms and ways to build meaning from multiple representations. This is where the process of construction is fruitful for meaning making with multiple representations in science.