Introduction

Digital communication and the rapid circulation of multimodal texts—which combine two or more modes such as words, images, and audio—have proliferated (Kress & van Leeuwen, 2020). At the same time, students’ capabilities with multiple modes are now central in the formal curricula of many countries including Australia, Singapore, and New Zealand (ACARA, 2021; MoE New Zealand, 2014; MoE Singapore, 2021). Importantly, immersive virtual reality (VR) environments are considered a very recent development in educational technology, bringing new opportunities for 3D multimodal and immersive learning in experiential or lifelike simulations. While access to VR has historically required expensive head-mounted displays (HMDs) that were skill-intensive to use, the improved accessibility, affordability, and ease of use create urgency for an evidence base to guide the use of VR pedagogies for multimodal literacy and learning.

VR for teaching and learning

Teaching and learning practices with VR are increasingly used and researched in higher education, such as in science and medicine (Birt et al., 2018), yet the pedagogies to support multimodal capabilities have not been fully developed, applied, and researched in schools (Marsh & Yamada-Rice, 2018). Further, VR media literacies are becoming central to many youths’ digital lives outside school with the advent of low-cost immersive VR (Miguel-Alonso et al., 2023). Given that VR is a high-growth technology this decade with reduced costs and increased accessibility, there is an urgency to better understand multimodal representation with VR and develop new pedagogies for its application across the curriculum (Radianti et al., 2020).

This original, qualitative classroom-based study offers new insights into the use of immersive VR utilising HMD for students’ design of 3D multimodal texts. The research participants were upper primary students who created 3D virtual artefacts as texts, inspired by an integrated curriculum unit for History, English, Arts, and Technology. Learning experiences also involved reading, writing, or speaking about subject-specific textual material on ancient Rome.

One of our central purposes was to understand the affordances of VR gaming for 3D multimodal designing and representing to consolidate or apply new knowledge across the curriculum. We also sought to understand how teachers can pedagogically support elementary school students’ use of VR technology during a key stage of increased digital text use in the middle and upper elementary years (Golan et al., 2018). New pedagogical findings can inform and guide teachers’ use of new immersive VR technologies for application in a variety of curriculum domains.

Using VR games for multimodal learning

The use of virtual reality games has strong support in educational research. ‘Games’ refers to digital games used in learning contents, played on PCs, gaming consoles, and other devices. Virtual games can include immersive simulations that are 3D lifelike representations of social practices in the real world (de Freitas, 2006). Virtual reality games involve 3D simulated environments that afford presence—the sense of being there—and immersion—in which the virtual environment surrounds the user while the real world is blocked from view (Palmas & Klinker, 2020). VR games should be distinguished from virtual worlds that are viewed on 2D computer screens. A distinctive feature of VR is 3D or 360-degree interactive immersion, enabled using a head-mounted display (Jensen & Konradsen, 2018). Likewise, using motion sensors involves different spatial, haptic, and locomotive potentials for learning and multimodal textual design—a far different experience to being seated at a computer with one’s real-world context constantly in view (Mills et al., 2022).

One learning advantage of VR is the ability to experience virtual situations that are inaccessible in the real world, and which can be used to explore story worlds (Mills, 2022). Another, shown in our previous research with school-aged learners, demonstrated new possibilities for transmediation or shifting of modes across media using VR, inviting generative thinking to overcome the lack of equivalence between the modes of writing, drawing, and virtual painting (Mills & Brown, 2022). New forms of embodied multimodal representation in VR involving extensive bodily, haptic, and locomotive movement were also explored by us (Mills et al., 2022). Finally, many VR games require users to interact with written words in menus and commands, information displays, captioned models, environmental print, and written artefacts. For example, in the 360-degree interactive film, The Book of Distance (Okita, 2020), users can read written text on ancient scrolls, newspapers, passports, letters, photographs, and sign writing, with new interactive potentials for multimodal reading and for interpreting visual imagery. Evidently, virtual reality technologies afford students a wide variety of immersive multimodal learning experiences.

VR, multimodal design, and the multiliteracies pedagogy

This study applies the New London Group’s (2000) well-cited concepts—available design, designing, and the redesigned—in the theory of multiliteracies, extending them to a new era of immersive, three-dimensional, VR technologies used by primary students. Multimodal textual practices bring together culturally and historically available semiotic resources across varied modes of meaning. Available designs refer to how authors are positioned by available modes and their elements of meaning. Designing is the active process of drawing on modes of meaning and design elements to transform meanings. The redesigned is the outcome of designing through the conscious control or spontaneous decision-making of the designer, who transforms new meanings with varying degrees of originality (New London Group, 2000).

Using VR for multimodal textual designing involves combining assemblages of semiotic resources that are transformed by the user to produce the redesigned. Applied across the curriculum, VR has been used to leverage students’ scientific interests (Brown et al., 2021), to enhance cultural learning (Berti, 2021), and to develop knowledge of history (Allison, 2008). Recent research outside of schools involving simulations points to new potentials and complexities for designing, including new gestural and haptic expertise (Gao et al., 2019).

In VR learning experiments, scholars have noted how children’s previous physical experiences of touching objects is important in their process of grasping new forms of virtual materiality and connecting to the known (Søyland, 2020). In the New London Group’s (2000) multiliteracies pedagogy, this process of connecting new knowledge with past experiences is developed through situated practice involving experiencing the known and the new, leading to transformed practice to produce original multimodal texts (see Cope & Kalantzis, 2015). VR-based learning can support situated practice, which is aligned to a constructivist learning paradigm, by scaffolding students’ independent exploration and experimentation while connecting with learners’ prior knowledge (Radianti, et al., 2020). These key concepts and pedagogical approaches are considered in the present study for theorising multimodal designing and learning with VR games in elementary school settings.

Methods

This project adopted the methods of the qualitative, naturalistic classroom-based research. The research questions, ethics approval, site, participant description, learning experiences, data collection (video, think-aloud, and interview data), and multimodal and thematic data analysis will now be discussed.

Research questions

The research addressed the following questions:

  1. 1.

    What are the key multimodal design elements supported by VR gaming?

  2. 2.

    How can VR gaming encourage creative multimodal redesigning?

  3. 3.

    How can multimodal designing with VR strengthen students’ knowledge?

  4. 4.

    What are the pedagogical strengths and weaknesses of VR gaming?

These questions are important because the available design elements—the grammars and semiotics systems—of multimodal texts have been researched across a wide range of media—from picture books, photographs, films, to online texts, eBooks, and other digital formats—while VR games are far more recent, still rarely theorised as multimodal learning. A recent review of 99 education articles on VR specifically focused on engineering, physical education, general education, health, and science (Kavanagh et al., 2017). To date, a good portion of the VR research has been led by education-technology developers (Pottle, 2019).

Research ethics

University human research ethics approval was obtained by the authors (ACU 2018-97H), consistent with national guidelines (Australian Research Council & Australian Vice Chancellor’s Committee, 2018). Participation in the study was voluntary, with informed written consent provided by the students, caregivers, and teachers, including for the use of video recording and the application of pseudonyms.

Site and participant description

Three classes of Year 6 students (n = 48, ages 10–12 years) and three teachers located in Queensland (Australia), participated in the study. The school was situated in a culturally diverse area with 65.2% Australian-born compared to 66.9% nationally (Australian Bureau of Statistics, 2021). Having parents born overseas (35.1%) was higher for participants than for the state (27.9%). English as the only language spoken at home (80.3%) was comparable to the state average (80.5%). Achievement of an undergraduate degree (17.2%) among adults was lower than across the nation (22%, ABS, 2021). Across the three classes of students, the overall caregiver consent rate to participate was 87%.

Description of student learning experiences

Three upper elementary teachers and the researchers planned the digital media experiences addressing the Australian Curriculum capabilities and school curriculum. The Australian Curriculum requires students from Preparatory to Year 12 to compose media texts through speaking, writing, and creating, including multimodal composition, which can include print or digital formats. Students learn to combine modes, such as written or spoken language, gestures, music, and visual images for varied social purposes (Australian Curriculum, Assessment, and Reporting Authority, 2021).

Through traditional classroom learning experiences, students gained knowledge of ancient Roman history, learning about social roles, beliefs, military strengths, and daily life. They used books, videos, and learned new vocabulary, becoming familiar with Roman iconography and society. Learners sketched and illustrated ancient Roman pottery using paper and pencils. They then created 3D artefacts in a virtual game using Let’s Create! Pottery VR 2, supported by the HTC Vive Pro HMD in a room-scale virtual play space (see Fig. 1). They also wrote information texts about Roman life as part of the school’s integrated curriculum.

Fig. 1
figure 1

A student using a VR head-mounted display

Full size image

Data collection

Data collection was conducted over several weeks, including planning with teachers, conducting the student writing tasks, and implementing the VR experiences. The following multimodal datasets were collected prior, during, and after the learning experiences, further elaborated in this section:

  1. i.

    Student writing samples and drawings describing ancient Roman artefacts.

  2. ii.

    Student semi-structured think-aloud interview conducted during the VR designing task, and a post-activity interview, both video-recorded and transcribed.

  3. iii.

    Continuous video recording of classroom VR experiences (View sample: https://drive.google.com/file/d/1PMeu5hG4pnaXJNs9IU95lNwX1I4aDPgA/view?usp=sharing

  4. iv.

    Continuous screencasts using OBS Studio of the VR gaming (View sample: https://drive.google.com/file/d/1jxH8-exGj1c1c_pMi0ccwSZtC-oszf4v/view?usp=sharing)

  5. v.

    Audio recorded teacher interviews (20 min)

Students wrote information texts about their VR artefacts, sharing the history-informed knowledge embedded in their multimodal designs (See, for example, Fig. 2).

Fig. 2
figure 2

Sample of a student’s handwritten description of their Roman vessel

Full size image

Think-aloud interviews were conducted with students during their VR experience to access the students’ thought processes about the event in real time when retention is strong (Hevey, 2020). For example, student interviewees were asked: ‘What knowledge of ancient Rome are you using to shape, colour, and pattern your virtual design?’ (See Fig. 3).

Fig. 3
figure 3

An excerpt of a student think-aloud interview

Full size image

After participating in the learning activities students were interviewed to capture their critical reflections and enrich the depth of insight (Hevey, 2020). Both semi-structured interviews (during and post) were recorded, transcribed verbatim, and thematically coded using NVivo 12 (QSR International, 2021).

Classroom fieldwork involved a total of 20 hours of video observation. The research team visited the classroom at weekly intervals, documenting students’ digital media use as they created VR pottery using the HMD. Continuous classroom video and VR screen capture using OBS Studio software were recorded to support the analysis of the students’ speech, action, and multimodal designing in the VR game (Garcez et al., 2011). Still image sequences were created of key moments illustrating students’ creative processes and knowledge of ancient Rome.

Individual semi-structured interviews were recorded with the three teachers (20 min) for pedagogic analysis. The New London Group’s (2000) multiliteracies pedagogy, which combines four practices (overt instruction, situated practice, transformed practice, and critical framing), informed the interview protocol to theorise teachers’ pedagogical choices. Sample questions included: ‘How do you guide multimodal designing in the curriculum?’ and ‘Can you give examples of pedagogies that involve the students acting on the world?’.

Data analysis

The following section describes the data analysis applied to examine VR and students’ multimodal designing, including the resulting themes that emerged from the coding of the student think-aloud interviews and teacher interviews, described below.

Multimodal data analysis

Multimodal data analysis was applied to the student drawings, writing, video recordings of students’ interactions, and screen recordings of their VR pottery making. A systematic record of multimodal data was compiled into a summary for each student in order to analyse the students’ creations (drawn, virtual, verbal reflection) by modes (for example, see Fig. 4).

Fig. 4
figure 4

Five work samples: drawing, 3D virtual pottery, and student comments

Full size image

Analysis of screen recordings and video data

The screen capture and video recordings of students were reviewed and analysed to address Research Question 1: ‘What are the key multimodal design elements supported by VR gaming?’ The following multimodal design elements were identified as central in the pottery creation platform and the students’ processes in crafting the virtual artefacts: (i) 3D open shapes, balance, proportion, (ii) colour, and (iii) pattern.

The analysis of screen recordings of the students’ interaction in the virtual game aimed to address Research Question 2—to understand how VR gaming can encourage creative multimodal redesigning. We replayed and annotated key sections of the screen recordings to analyse the students’ processes of multimodal designing. These data were combined with the coding and analysis of students’ think-aloud responses described in the next section.

Analysis of student think-aloud interviews

Thematic analysis was conducted on the semi-structured think-aloud interviews (during and post) which were also matched to Research Question 2: ‘How can VR gaming encourage creative multimodal redesigning?’ and to Research Question 3: ‘How can multimodal designing with VR strengthen students’ knowledge?’ The researchers coded the students’ responses by the following themes that were derived both inductively and deductively, supported using NVivo 12 software (QSR, 2021). The students’ responses were coded by knowledge themes (see Fig. 5). This dataset was combined with the analysis of students’ written paragraphs about their Roman artefacts.

Fig. 5
figure 5

Sample coding of student interviews by themes with student quotes

Full size image

Analysis of teacher interviews

Reflexive thematic analysis was applied to the teacher interviews to address Research Question 4 regarding the pedagogical strengths and weaknesses of the VR learning experiences and their integration into the classroom (Byrne, 2022). The responses were organised thematically and coded deductively to the four approaches of the multiliteracies pedagogy (see Cope & Kalantzis, 2015), with themes reviewed for accuracy and consistency (see samples in Fig. 6). These approaches are as follows: overt instruction—the active interventions of teachers or experts to scaffold learning experiences; situated practice—the immersion of a community of learners that recruits or builds on their previous experiences; critical framing—the critique of ideological and value-centred assumptions of knowledge; and transformed practice—demonstrating new knowledge and skills for genuine purposes leading to learner change (Cope & Kalantzis, 2015).

Fig. 6
figure 6

Sample coding of teacher interviews by themes with teacher quotes

Full size image

Results

The findings in answer to each of the four research questions will now be addressed in terms of the features of the VR-supported digital media practices: (i) The key multimodal design elements supported by VR gaming; (ii) How VR encouraged creative multimodal redesigning; (iii) How multimodal designing with VR can strengthen students’ knowledge; and (iv) Teacher perspectives of the pedagogical strengths and weaknesses of VR.

Research finding 1: VR gaming supported 3D multimodal designing through haptic and embodied experience

VR gaming was found to be particularly useful for supporting students’ 3D visual and spatial text making applied to the cultural knowledge of ancient Rome. The visuospatial design elements emphasised in the VR platform included the making of 3D historical artefacts—open forms characterised by balance and proportion, colour, and pattern. The representational fidelity of the historical artefact making in VR afforded 3D realism and embodied interactivity using haptic controls that support cognition and spatial reasoning, advantageous when compared to disembodied forms of web-based online learning with a computer keyboard and mouse (Dalgarno & Lee, 2010).

Visuospatial design elements: open forms

The VR pottery design elements focused the users’ attention on the haptic formation of open 3D shapes. As Emily considered, in forming the shapes, the VR technology proved easy to use: ‘It was easy to shape the clay and it was really good that you could press the clay with your hand to make it taller or wider’. The program had an expansive facility to design varied open forms such as globular (globe shaped), cylindrical, trumpet-shaped, pyriform (pear shaped), pomiform (apple shaped), or ovoid (see Fig. 7). Students could also create variations of realistic texture, such as smooth or variegated.

Fig. 7
figure 7

Clay forms in Let’s Create! Pottery VR 2 before firing in a kiln

Full size image

The VR simulation enabled the haptic creation of three-dimensional forms (see Fig. 8), requiring students to apply their capacity for visuospatial and motor skills. Savannah described co-ordinating the movement of her hand as an avatar: ‘At first, it was tricky because the hand isn’t my hand, so it felt really weird to shape and colour’.

Fig. 8
figure 8

Drawing, virtual shaped clay, and finished virtual pottery

Full size image

The students experimented with ways to shape the virtual clay haptically with embodied action: ‘I liked how you could make it taller and shorter, and use fingers to make it bumpy, and palms to make it kind of smooth’. The multimodal design of the 3D open forms in immersive VR involved visuospatial design elements characterised by balance and proportion—elements that the students achieved with precision and aesthetically pleasing outcomes. The vessels in Fig. 9 below illustrate how complex vessels consistently demonstrated balance.

Fig. 9
figure 9

VR vessels showing balance and proportion: Hailey, Liam, and Evie

Full size image

Another key spatial feature of the immersive game was that players could walk around the virtual space to view the clayware from multiple perspectives on the horizontal plan, such as inside the pot or from the rear, whereas the 2D drawing required multiple depictions to show front and back view, as depicted in Ava’s drawings (Fig. 10).

Fig. 10
figure 10

Ava’s designs: drawing, virtual shaped clay, and finished virtual design

Full size image

Visuospatial design elements: colour and pattern

The VR game involved the students firing the clay in a kiln, which unlocked a new palette of design options for patterning and colouring the clay vessels. The available patterns depicted different cultural origins, with students selecting Roman designs. A laser was used to apply the patterns and colours (as shown in Fig. 11), with options to intensify the hue, and a preview function for low-risk experimentation.

Fig. 11
figure 11

Visual progression of virtual pottery: Pattern and colour

Full size image

The colour palette offered varied hues and degrees of saturation, and once applied to the pottery, automatically followed the contours of the 3D vessels with representational fidelity. Some of the available resources for patterning included the Greek key meander, guilloche, wave bands, and repeated gladiator motifs (see Fig. 12).

Fig. 12
figure 12

Virtual menu displaying colour palettes

Full size image

The application of patterns to the 3D shapes produced visually pleasing designs of Roman patterns that ensured a successful outcome (see Fig. 13). As Emily noted: ‘Choosing the colours was quite easy, and making the patterns wasn’t too hard’.

Fig. 13
figure 13

Virtual vessels with patterns: Lily, Xavier, and Harper

Full size image

A novelty of the immersive, virtual creating is that the patterns and design elements appeared both outside and inside the fired 3D clayware with perfect uniformity, which would be difficult to achieve in real life (see Fig. 14).

Fig. 14
figure 14

Interior and exterior patterns in virtual pottery

Full size image

In sum, in relation to the question—‘what are the key multimodal design elements supported by VR gaming?’—the platform specifically focused the students’ attention on visuospatial designing of historical artefacts that were open 3D shapes, focusing on balance and proportion, colour, and pattern, in the context of cultural meanings in Roman times. Importantly, VR multimodal designing offered opportunities for interactive, haptic, and embodied learning characterised by 3D representational fidelity—advantages for experiential learning compared to web-based and 2D forms of visual designing online (Dalgarno & Lee, 2010).

Research finding 2: VR improved performance through creative

Redesigning

A number of affordances of VR gaming supported students’ processes of creative redesigning. Our findings with regard to how VR gaming encourages multimodal designing are organised by the following themes: (i) VR gaming supported repeatable practice to improve performance, (ii) multimodal redesigning involved intention and spontaneous adaptation of intentions, and (iii) students arrived at a key point of design satisfaction.

VR-supported repeatable practice to improve performance

The fully immersive process of pottery design supported the students to manipulate the virtual clay through iterative cycles of creative multimodal redesigning, which allowed them to make mistakes and learn through practise to improve performance. The students’ interactive manipulation of the artefacts involved aesthetic decision-making and repeated experimentation. For example, at first, Bailey aimed to use ‘an ancient orange colour’, inspired by ‘patterns and colours the Romans used when colours were limited’ (see Fig. 15).

Fig. 15
figure 15

Completed virtual pottery: Bailey

Full size image

The students used the preview tool to experiment with colours and patterns before applying them, since this feature allowed the user to visualise the pottery in a risk-free context before selecting the preferred colour or patterning option (see Fig. 16).

Fig. 16
figure 16

Visual progression of virtual design: experimenting with colour and pattern: Bailey

Full size image

When Bailey noticed that he could layer the paint to darken and blend colours, he continuously re-applied colours to create darker tones, leaving other sections lighter to achieve contrast. Bailey visualised four different patterns before finalising his selection, supporting intuitive spontaneity and experimentation (see Fig. 17).

Fig. 17
figure 17

Virtual menu: Previewing pattern application

Full size image

Correspondingly, students such as Cooper alternated multiple options between the patterning and colouring palettes, realising his intentions through repeated practise, evaluating, and applying design choices. Cooper described how he applied his familiarity with Roman pottery shapes to ‘create a wide bowl shape… I envision it holding fruit’—a design that involved significant modifications of the initial auto-generated clay form (see Fig. 18).

Fig. 18
figure 18

Roman bowl: drawing and virtual vessel

Full size image

In cycles of previewing, evaluating, and selecting colours and patterns, Cooper combined a complex array of Roman motifs. Ancient Roman key and wave designs juxtaposed with modern-day chequers were sequenced repeatedly to follow the contours of the bowl. While most students painted from the outside of the pottery, the open shape of the bowl permitted designing from within, while the software duplicated the patterns on the external sides (see Fig. 19).

Fig. 19
figure 19

Cooper’s progression of virtual pottery: applying pattern and colour

Full size image

The VR-supported multi-faceted stages of multimodal redesigning as students repeatedly applied their semiotic choices with continuous and fluid cycles of practice to improve their performance. Sienna elaborated on the VR design process to achieve representational accuracy: ‘It’s about looking closely and double-checking. With programs like Free Brushing, you can go about your way, but with Pottery, you’ve got to focus hard to get every single little design right’.

Multimodal redesigning merged intentions with spontaneous adaptation

In these cycles of multimodal redesigning, the students demonstrated clear intentionality, as well as spontaneous adaptation to the mediating tools at hand. For example, the immersive VR tools allowed Hannah to create a shape, pattern, and colour that reflected her drawing through cycles of redesigning (see Fig. 20).

Fig. 20
figure 20

Hannah’s pottery: drawing and virtual vessel

Full size image

Both the drawing and vessel consisted of a wide body that narrowed to the foot, with a pattern in red, gold, and purple, and banded by blue. Hannah chose to use her drawing as a guide: ‘I’m thinking about how my design looked when I first drew it’. VR Pottery allowed students to shape the virtual clay as planned but they could also spontaneously adapt the shape as they progressed. Hannah trialled a range of shapes before returning to her original design (see Fig. 21):

Fig. 21
figure 21

Experimentation while shaping virtual clay: Hannah

Full size image

During the process of colouring and patterning, Hannah repeatedly evaluated and adjusted her design, moving back and forth between colouring and patterning, rather than following a linear procedure. Hannah used the preview tool to test the design options and to avoid mistakes, an advantage of VR. She also used the erase tool five times to edit her multimodal design, evaluate the design, and sometimes to return to a pattern considered earlier (see Fig. 22).

Fig. 22
figure 22

Progression of virtual pottery: experiments with patterns and colours: Hannah

Full size image

The zigzag pattern of her drawing appeared on the 3D design against a terracotta background, aimed to make the pottery appear ‘old and worn’ (see Fig. 22). The final design reflected her personal intentions while involving careful adaptations: ‘It looks pretty good compared to what I was picturing. I used Greek patterns and the royal Roman colours—purple, red, and blue’.

After repeated evaluation and adjustment, another student, Ryan, discovered an ombre effect while layering colours in the virtual context, using only bands of colour to create repetition, rather than the pre-formed patterns: ‘I like the fade effect’. Ryan used 12 applications of paint to create three rings of black down the neck of the pottery, and ombre gradients, from purple through shades of blue (see Fig. 23).

Fig. 23
figure 23

Visual progression of virtual pottery: experimenting with layered colours

Full size image

Mia’s pottery shaping followed a somewhat unplanned and spontaneous process. Mia initially stretched the virtual clay to form a taller pot as she initially planned. She then formed a neck and lip, changing the overall shape. As she rounded body of the pot, Mia adjusted the height and produced a vessel of short and wide proportions—revisions that were executed quickly (see Fig. 24). Mia was pleased with the outcome of this intuitive process that the VR technology had allowed. ‘It didn’t turn out how I imagined it… but I think it actually turned out better’.

Fig. 24
figure 24

Visual progression of virtual wet clay shaping, with final pottery: Mia

Full size image

Students arrived at a key point of design satisfaction

Each student came to a key point in the process where they decided that the vessel no longer required further redesigning, frequently noting how the resultant design was intentional, as Archie noted: ‘I like how it fits a design I was working towards’ (Fig. 25a). Diego assessed his 3D vessel: ‘It was creative and colourful. I like to do patterned and artistic stuff with my designs’ (Fig. 25b). Olivia was satisfied when her design resembled an authentic ancient Roman artefact: ‘It looks pretty realistic’ (see Fig. 25c).

Fig. 25
figure 25

Virtual pottery: a Archie, b Diego, and c Olivia

Full size image

A sample of students’ 2D drawing and 3D virtual Roman artefacts is provided below illustrating both planning and adaptation to the resources of each media, with their evaluative reflections (see Fig. 26).

Fig. 26
figure 26

Student drawings, virtual artefacts, and verbal reflections

Full size image

The students reached a point of satisfaction with their multimodal designs with intentionality, whether it reflected authentic Roman culture (Alexander, Sienna, Evelyn, and Bailey—Fig. 26), was aesthetically pleasing (Zara), mirrored their plans (Zara and Alexander above), or told a story (Evelyn). At the same time, the design process involved spontaneous adaptation to the media and resources at hand.

Research finding 3: VR-supported knowledge application, consolidation, and transfer

In relation to the potentials of VR to support student knowledge (RQ3), VR-supported students to apply, consolidate, and transfer their cultural and historical knowledge specifically through making knowledge artefacts. Knowledge artefacts are a medium to represent understanding, forming linkages between learning experiences to create a knowledge flow, and useful for transferring and sharing knowledge (Abuhimbed et al., 2013). The virtual pottery referenced their understanding of Roman iconography, geography, beliefs, architecture, customs, and other cultural and historical knowledge, including the domestic, military, aesthetic (e.g. colour, pattern, shape), and functional uses of Roman vessels.

Alexander explained the symbolic significance of his artefact to the Roman military:

Roman soldiers fighting for 25 years would leave the army and get land. So, I made a water jug that a Roman soldier, who fought for 25 years, took with him to remember his role. The colours are from the Roman soldier’s uniform (see Fig. 27a).

Fig. 27
figure 27

Virtual pottery: a Alexander, b Layla, c Sienna

Full size image

Students referred to historical locations and landmarks, such as rivers in Roman times. As Layla described: ‘The patterns I used, the waves, were for the Rubicon River’ (see Fig. 27b). Sienna’s design (see Fig. 27c) was informed by geographical location. Sienna spoke about her Roman artefact: ‘The design is based on the river Tiber and the seven mountains surrounding Rome’. She continued: ‘I needed historical context of the patterns I was using because the historical context allowed me to make a story in my pottery’. Sienna’s written description (see Fig. 28) about her initial pottery illustration also identifies historical uses of pottery and geography, with motifs signifying mountains, water, and a bull for the earth goddess Tellus.

Fig. 28
figure 28

Sienna's handwritten description of her illustrated vessel

Full size image

In terms of cultural knowledge, the students made references to ancient Roman civilisation, including the domestic, military, aesthetic, and functional uses of Roman vessels that were customary in daily life. Ethan’s pottery drew on cultural knowledge and imaginings of Roman life and architecture: ‘I imagined pictures and designs from the outer rims of the Colosseum with gladiators fighting’. Similarly. Emily connected her knowledge of Roman culture to direct the shape of her Roman ware: ‘You need to know how they used the pot to design the shape—a long tube for wine, or a medium-sized one for water, or a small jar for olives. Different shapes contain different things’.

Others applied their knowledge of the use of colour, geometry, motifs, and patterns in ancient Roman life, as Ethan explained: ‘I took Roman colours, like red, orange, yellow’. He depicted ‘the Romans mostly wearing red, with their helmets yellow—the golden colour used by Romans’. He described that Roman designs were often ‘triangles, squares, rectangles, circles, formed to make little pictures’, depicting ‘pink and purple chequers for the capes that the Praetors [judicial offices] wore in Ancient Rome’ (see Fig. 29).

Fig. 29
figure 29

Ethan’s virtual vessel depicting Roman soldiers and colours

Full size image

The pottery VR game functioned as a springboard for students to materialise their knowledge of Roman culture through knowledge artefact creation, including spiritual beliefs. Olivia suggested that VR pottery can include designs of ‘the gods they had, because that was a very big thing in Rome’. Teachers noted some unique benefits of VR for the consolidation of students’ knowledge: ‘While students had fun it also comes down to why is this beneficial… If we unpack how much they learnt—it was a very rich experience. Sometimes enjoying things deepens skills and knowledge’.

In connecting cultural knowledge to virtual artefact creation, the students reflected on the material production of colour in Roman times, as Ryan observed: ‘Back then you would have made the paint. Colours would be expensive, purple especially’. Lachlan also described the production of pigments in Roman times: ‘They wouldn’t have all of the colours—they would just have a limited amount that they made themselves’. Matilda reflected on the time to produce real artefacts and the comparative ease of making it virtually: ‘Pottery in Rome would take a lot longer because you would need skill to carve different patterns and images into it’. Ethan reflected:

It gave me the idea of how hard it was in Rome, because in VR and in life today we have shelves full of artefacts—millions and billions of them! Then I think: how long would it have taken to make one?

The VR platform offered students an interactive experience that would otherwise have been difficult to achieve in the real world due to constraints on time, access to equipment and materials, and expertise.

Research finding 4: teacher perspectives—situated and transformed practice are strengths of VR

The teachers’ perspectives of the pedagogical strengths of VR gaming and how they augmented these with other pedagogies will now be discussed. The teachers’ responses have been analysed according to the New London Group’s (2000) pedagogy of multiliteracies and learning by design framework: (i) situated practice: experiencing, (ii) overt instruction: conceptualising, (iii) critical framing: analysing, and (iv) transformed practice: applying (Cope & Kalantzis, 2015). These are not practised as a linear hierarchy but are revisited and related in fluid ways, separated here for the purpose of analysis.

A pedagogical strength of VR gaming was that it provided opportunities for situated practice through students’ immersion in simulated experiences of ancient Roman knowledge artefacts through virtual pottery making. Situated practice enables students to connect their learning experiences and knowledge to new experiences, while teachers actively consider the diverse socio-cultural needs of all learners in a community of practice (New London Group, 2000). The work of situated practice requires experts to develop individuals as members of a learning community, leading them to a broader range of multimodal resources for designing.

Situated practice in this study involved immersion in the lifelike world of Roman pottery making, but also more broadly in their curriculum through other texts and media. A key to situated practice is to weave familiar and unfamiliar knowledge together through experiential learning, as one teacher explained: ‘I consider their knowledge and cultural backgrounds’. Another similarly noted that a key is ‘reflecting on what they already know, what they need to know’, and then considering ‘how to use technology and real experiences’ as situated practice.

In terms of overt instruction, the teachers described that overt instruction is an aspect of pedagogy that leads to the students conceptualising knowledge: ‘We’ll look at designs first and then model whatever we’re introducing’. The teachers emphasised guiding the students’ multimodal composing: ‘we help students to understand the importance of using multimodal designing, to guide them to think: “What are the multimodal elements that will really captivate my audience?”’.

Overt instruction was important in scaffolding the students’ use of VR to investigate Roman times. While the students chose topics of interest, teachers showed them ‘how to research, where to find information, and how to use that information’. Importantly here, overt instruction guided the students to develop awareness of historically relevant design elements, providing this knowledge through varied instructional formats in ways that are timed to be most useful (New London Group, 2000).

Applying critical framing, the teachers emphasised the importance of students needing to stand back from their multimodal designs and the texts that they read, viewing them critically in relation to their contexts and the authors’ intentions. One teacher noted: ‘we ask the students: “Are there some assumptions and values? Can we look from a different point of view?” With the Romans, we challenged them to think about power. “What is power? What about their use of slavery?”’.

In critical framing, teachers guide students to critique the value-centred assumptions of texts and their cultural, ideological, and social construction (New London Group, 2000). Importantly, this critique was not limited to theoretical reflection, but ideally leads to critical action on the world. For example, one teacher explained that students need to ‘act on the world in some way—becoming aware of what their consumption means to the world and how to limit it’. Likewise, the VR experience was not exempt from critical analysis, as they noted: ‘The teacher needs to bring a critical lens to VR and ask, “How do you teach students to become discerning consumers of multimodal stimuli?”’.

The VR learning experience supported students to transform meanings, the redesigned, to enact their knowledge of Roman history and culture to work in a virtual context. Anna observed: ‘VR made it easier to see the type of designs that Romans made, and I’m interested in how they did it’. The VR game had a particular advantage for creating knowledge artefacts to enable the application, consolidation, and transfer of knowledge through simulated experience. Teachers also noted that the game fostered ‘independence’, encouraging students to ‘use many ways of expressing what they know digitally to inspire them to explore new horizons’.

Discussion

This paper has generated new understandings about the use of virtual reality technologies for multimodal designing in simulated, three-dimensional virtual gaming worlds. The findings have significance because they demonstrate that VR technology focuses the user’s attention on key visuospatial design elements that differ from 2D learning, such as realistic historical artefacts with representational fidelity. Additionally, it draws on the strengths of haptic and embodied learning through simulated experience. The ability to design multimodal texts that involve unique 3D design elements in VR has been found vital for spatial reasoning—learners who design with immersive VR develop stronger spatial visualisation and rotation skills compared to those who use a non-immersive computer and mouse (Molina-Carmona et al., 2018).

The VR platform offered students functions that supported intuitive cycles of creative designing and modification, allowing them to create with intentionality, while scaffolding continual evaluation and adaptation of their designs beyond the use of traditional sculpting media, and transcending historical time periods. Research points to specific challenges of developing creativity in elementary contexts, since creativity takes domain-specific forms, and elementary teachers are required to translate curricula knowledge across all disciplines (McLean et al., 2023).

Importantly, giving students the opportunity to make ancient vessels worked in powerful ways to consolidate students’ knowledge, with virtual knowledge artefacts functioning as visual and mental anchors for students to apply, consolidate, and transfer their knowledge that was acquired through varied modes and media to practical applications within the VR simulation (Abuhimbed et al., 2013).

Pedagogically, the VR gaming platform was found to be particularly good at simulating lifelike experiences through situated and transformed practice, but required meaningful integration into the curriculum, supplemented with equally important pedagogies that were missing, such as overt instruction and critical framing. It should be noted that while it is vital for students to design multimodal texts, it is equally important for them to critically deconstruct whose interests are served in the production, use, and distribution of any new media (Dezuanni, 2021).

The integration of any new digital learning tools into classroom learning needs to be supportive of teacher professional identity and training (Abbott, 2016; Kopcha, 2012). One teacher reflected that they had not trained in ‘advanced digital technology’ but remained supportive of the school focus on a multimodal integrated curriculum. Another observed the need for teachers’ ‘open-mindedness’ to consider how ‘VR is beneficial’ for distinct forms of learning, such as generative multimodal design, which allowed students to learn from powerful and immediate simulated experiences from different time periods, geographical locations, and knowledge domains on demand.

Conclusion

This research has explored the teaching of multimodal designing and learning at a time when VR platforms are anticipated to have a phenomenal impact on the workforce. The fastest changing occupations now depend on such skills, with VR texts already widely used in higher education worldwide, and in K-12 education in leading nations (Tilhou et al., 2020), while 78% of workforce leaders fear negative impacts of digital disruption by virtual work (World Economic Forum, 2020). Existing knowledge of both conventional and screen-based text processing is now insufficient for understanding and teaching the text processing associated with creative 3D modelling, spatial skills, and the visualising required in VR environments. This research has addressed the introduction of new multimodal skill sets for elementary students that will be essential in the future—such as 3D sensory learning through simulated virtual experiences already used in education, healthcare, manufacturing, marketing, and transportation (Lauterman & Ackerman, 2014).

Multimodal texts have changed in immersive VR environments, with the urgent need to expand current understandings of these hybrid genres across the curriculum, such as in history and science. The findings address a widening gap in literacies and digital media research that has previously focused on VR technologies that are not fully immersive, such as screen or web-based VR (Huang et al., 2010), and avatar-based applications in virtual worlds (O’Connor & Domingo, 2017). Likewise, it extends the application of multimodal semiotics and the multiliteracies pedagogy of the New London Group to support the teaching of new kinds of 3D visual and spatial design elements that feature prominently in immersive virtual environments, including video games, 360-degree films, virtual tours, and other applications now used in retail, engineering, healthcare, education, and other workplaces (Mills et al., 2023).

Future research of VR technology will be needed to examine new gaming texts and technology advances that are augmenting the nature of multimodal literacy across different curriculum areas, media applications, social contexts, and levels of education. The expansion of VR applications for digital media designing and consolidating knowledge offers significant potential to challenge conventional notions of 2D textual practices, with distinctly diverse ways to experience, represent, and remake the world in transformed virtual simulations.