Fully immersive learning with virtual reality for assessing students in art history

Abstract

Immersive learning systems using consumer grade headsets for virtual reality (VR) software can now reach the classroom. VR represents an important step toward situated learning for bringing realistic experiences to show to students different types of content the way it appears in real life. With the three-dimensional effect, it adds an immersive dimension that can bring the student into a unique environment that is directly connected to the learning content. While VR experiences can be helpful in educational settings, they must satisfy the learning objectives of the course and the specific needs of the students. In this paper, we propose a customizable VR application for displaying paintings for their analysis, and their associated questions designed for instructors in art history. To improve the accessibility and adaptivity to instructors and specific learning materials, we propose the definition of the paintings’ characteristics and the questionnaires associated with the paintings in the JSON open-standard file format, facilitating application changes without any prior programming knowledge. We compare the proposed VR modality with a web-based application on a computer desktop with 35 undergraduate students with art history experience. In both modalities, we assess the workload and usability; the VR sickness symptoms and the motivation in the VR condition. The results indicate better usability and lower workload with the VR condition. While there are no differences in terms of students’ performance for answering the questions, 77% of students prefer the VR condition. The Reduced Instructional Materials Motivation Survey shows a high motivation in the student population. Finally, the system evaluation supports the conclusion that the proposed VR system can be deployed in the art history classroom as the system has a high usability and medium workload.

1 Introduction

Immersive learning technologies implemented with virtual reality (VR) enable the creation of realistic environments in which learning can take place (Herrington et al. 2007). Immersive learning has gained a growing interest among educational technology practitioners. It is due to the availability of commercial consumer grade VR headsets that can bring the technology outside of the laboratory. VR technology provides novel opportunities for implementing and researching immersive learning tools facilitating the transmission of learning goals. Immersive and interactive technologies such as VR represent a novel way how students could interact with the environment (Rubio-Tamayo et al. 2017). Immersion can be defined as the subjective impression that an individual is participating in a comprehensive and realistic experience (Dede 2009). Immersion in a digital environment can enhance education through multiple ways based on how the materials are presented and accessed. Students can get exposed to the learning materials through different perspectives, e.g., from different vantage points. It provides an important advantage when the objects to study require context and the perception of their physical dimensions. In the example of an art museum, the perspective of being inside enables users’ actional immersion and motivation through embodied, concrete learning, as would happen in a real museum. Software implementing a fully immersive VR environment can develop educational experiences through situated learning (Dawley and Dede 2013). This situated learning experience using a virtual museum can provide a genuine representation of a real museum. Furthermore, immersion through the simulation of a real environment can enhance transfer. Transfer is defined as the application of knowledge acquired in a given situation to another place (Jackson et al. 2019). However, immersion can induce a lack of awareness of both time and the real world when it is achieved through a game, while providing a sense of being within the environment (Jennett et al. 2008). VR technologies offer new opportunities in education, but they give new research challenges in human–computer interaction (e.g., hardware and software requirements, user experience and engagement, pedagogical design and evaluation) as well in developing VR applications and immersion at scale in both formal and informal learning (Liu et al. 2017; Osberg 1995).

We propose in this paper a novel customizable learning tool in a fully immersive VR environment for art history students. The VR environment includes rooms in which students can visualize paintings and answer questions about the paintings. Art history delves into the study of artworks within their cultural context. Art historians analyze the production and meaning of visual arts (painting, sculpture, architecture, etc.) at the time they were created. Another mission of art history is to identify the authorship of artworks, i.e., finding who created a particular object, when, and for what reason (Wölfflin 2012). Most definitions of what VR is report a digital representation of an object and/or an environment in 3D. VR can be described as the use of computer technology to create the effect of an interactive three-dimensional world in which the objects have a sense of spatial presence (Bryson 2013) or the combination of interaction, immersion, and imagination (I3) (Burdea and Coiffet 2003). In this paper, we focus on head-mounted display devices, a.k.a., fully immersive VR.

The goal of this application is to provide a customizable immersive learning environment for students of art history and a means for instructors to determine its content. The content includes paintings that are presented like in a real museum and quizzes. While an art gallery can represent a learning tool, it is not geared toward specific learning outcomes, it does not include learning activities and assessments. The current status of VR-based museums shows that there is a substantial lack of educational content. Moreover, such VR-based museums do not possess the ability to align the assessments with the required learning outcomes (Biggs and Tang 2007). Educational VR software can be considered as a pedagogical support for an illustration of art history. Immersive VR reinforces this relationship by putting the visitor/student in a meaningful context (Freina and Ott 2015; Lorenz et al. 2015). Students primarily learn about art history through images of famous works in textbooks and presentation programs (Powerpoint, Google Slide, etc.). These methods remove the perception of lighting, size, and texture, which can be experienced while observing the original work. If students cannot visit the museum containing the actual original work, observing a painting in a VR environment can, at least, add back the perception of size, lighting, and even potentially texture. Using VR brings back all the physical dimensions of the work, which have been removed when presented as a printed image or on a computer screen. A large number of virtual reality museums are now available online in which it is possible to browse and visualize art works. However, these virtual museums contain a predefined set of paintings that cannot be adapted for the needs of a class. Many VR applications are static, and they cannot be modified by an art history instructor or a person with programming skills. Hence, these software cannot fit accurately the needs of an instructor, for the needs of a specific course, for displaying selected visual content. To extend the deployment of immersive VR environments in the classroom, it is necessary to provide a structure and tools that can leverage their application in multiple settings. It is necessary to create customizable VR museums at a meta level in which it is possible to establish a collection of paintings and to give the possibility to create assessments within the virtual museum. An instructor in art or art history should be able to easily edit the content, in direct relation to the needs of the class. For art history instructors, the software inputs correspond to a set of paintings (i.e., images) and their relative information (painter, year, and size) that can be stored in a structured data format (e.g., XML and JSON). All these parameters are carefully chosen by the instructor in relation to the desired learning outcomes. In particular, a key aspect of this research project is to simplify how new content can be integrated within a VR environment. Given these constraints, the proposed VR application aims at offering both the functionality of a fully immersive VR museum and allowing instructors to determine a sequence of paintings and their associated questions.

The contributions of this paper are as follows:

• The description of a fully immersive procedural VR environment for the presentation of paintings and their questions.

• The evaluation at different levels (workload, usability, students’ performance, VR symptoms, and motivation).

• The comparison between VR and computer screen modalities for the presentation of paintings and associated questions with 35 undergraduate students with an art history background as participants.

The remainder of the paper is organized as follows. First related VR applications in education are presented in Sect. 2. The system and the evaluation of the proposed VR system for art history are presented in Sect. 3. The results are described in Sect. 5. The results and their implications on fully immersive VR in higher education are discussed in Sect. 6. Finally, Sect. 7 provides the key outcomes of the study.

2 Related works

Prior to consumer grade VR headsets, i.e., before 2014, multiple studies of VRs conducted in education provided positive findings. These results highlighted an increase in enjoyment, motivation, deeper learning and long-term retention, and time-on-task (Kavanagh et al. 2017). VR applications give the opportunity for participating in lifelike simulations or virtual explorations that would otherwise be impracticable or too dangerous to perform (Wei et al. 2013). Schools are unlikely to consider frequent domestic or international travels to visit museums or historical landmarks (Cecotti 2022). Yet, these students could explore museums and historical places through VR without leaving the classroom. In (Gaitatzes et al. 2001), they developed a system to simulate and allow the virtual tour of cultural heritage sites in Greece. VR technology has been utilized in education and training (Jensen and Konradsen 2018). Immersive learning technologies are used in different academic fields such as medical applications (Cecotti et al. 2020b, c; Mathur 2015; Ruthenbeck and Reynolds 2015), engineering (Alhalabi 2016), and chemical education (Fung et al. 2019). VR can improve online education experiences (Cortiz and Silva 2017). In addition, immersive learning has been considered for the creation of realistic simulations for military training (Bhagat et al. 2016). VR applications are now integrated in the context of art with users as art creators and art viewers. Such applications include the creation of 3D artworks using the painting tool Tilt Brush—now in open source, or for the short animated sequences that place the user directly in the center of the scene (Google 2016). RiftArt allows teachers to configure virtual museum rooms, with artwork models inside, i.e., sculptures, and enhance them with multimodal annotation (Casu et al. 2015). This application can be visualized on large screens and head-mounted displays. Gaugne et al. proposed an immersive experience where the user explores an interactive visual and musical representation of the main periods of the history of Western music (Gaugne et al. 2017). Free VR experiences containing art galleries are available to the public; their contents are various in terms of size and content, including a combination of paintings and/or sculptures. It is possible to take a virtual tour of various museums with the Google Arts and Culture application and an inexpensive VR headset, e.g., Google Cardboard. The Google Arts and Culture incorporates Google’s Street View technology, enabling users to visit museums through a 3D digital reproduction of these physical spaces (Google 2011). Other VR applications include “The VR Museum of Fine Art” by Finn Sinclair [19] that has both famous paintings and sculptures. It is worth mentioning that it includes architecture that enhances the museum experience, with a counter and a refreshment bar. Another VR-based museum is the Smithsonian American Art Museum “Beyond The Walls” by the Smithsonian American Art Museum [47]. The Infinite Art Museum is a permanent virtual reality museum and archive of art, and it contains a collection of over 800 original paintings, illustrations, digital art, and photographs from artists around the world [39]. While the concept of virtual museums is not recent as technology (Huang and Han 2014; Schweibenz 1998), the implementation of a fully immersive VR environment is more recent (Choi and Kim 2017). Furthermore, VR-based environments provide great opportunities in cultural heritage for research and communication, going beyond the educational aspect (Lercari et al. 2018; Carrozzino 2019). These different virtual museums cannot be customized by an instructor. In all the cases, the instructor has to design a lesson plan around the existing VR museum, while the instructor should in fact find the proper tools to enhance his/her lesson plan through the use of learning activities and assessments to convey the desired learning outcomes. An Art History instructor cannot modify the content of these existing virtual museums and create assessments. Hence, there is a research gap for the creation and assessment of virtual museums that can be customizable: 1) to allow instructors to change the paintings and 2) to allow instructors to create assessments such as quizzes for the different paintings.

3 System overview

The optimal presentation of a sequence of paintings in a museum, real or virtual, is a challenge. Planning the presentation and installation of artwork and objects is the responsibility of an art curator; paintings cannot be placed in any order. While virtual museums typically present a gallery with multiple paintings simultaneously, we present a single painting at a time with its corresponding questions to isolate the user from a single piece of content. The student can just focus on a single painting at a time. Such a minimal approach allows the system to set the dimensions of the room and its content in relation to a unique painting. This approach avoids arbitrary choices related to architectural designs for the inclusion of multiple paintings, which can have various sizes, genres, and types (e.g., landscape, portrait, and still life). Here, the size of the room is based on the size of the painting. While many VR museums set artworks “manually” within the scene, i.e., they are static, the proposed procedural approach is to define the scene in relation to the painting, by minimizing as many arbitrary choices as possible. It creates a small room for small paintings and a large room for large paintings. In the proposed VR environment, the size of the room and the default position of the user are both determined in relation to the size of the painting to prevent the user from moving between paintings. The goal here is to minimize the amount of motion (real or through teleportation) that the user must perform to reach the ideal position for observing a painting. At the presentation at each new painting, the user is already in place of the ideal vantage point, minimizing motion hence minimizing VR sickness and disorientation. The position of the painting on the wall is set in relation to its size.

We consider a painting of size $P_w \times P_h$ and the associated room of size $R_w \times R_l \times R_h$ that displays the painting. The painting is placed on a wall of size $R_w \times R_h$ at position $(C_x,C_y,C_z)$ corresponding to the center of the painting. The center of each painting is placed at 1.55-m height if the distance between the ground and the painting is superior to 0.75 m. If not, the bottom of the painting is placed at 0.75 m. A 3D painting frame with a wood texture is used for each painting.

The position of the painting is defined by: $C_x = 2+P_w/2$; if $1.55-P_h/2>=0.75$ then $C_y=1.55$ else $C_y=P_h/2+0.75$; $C_z=R_l$ corresponding to the painting is being placed on the wall. The value 1.55 was chosen arbitrarily but may be chosen in relation to the height of the user. We consider a field of view (FOV) of 100 degrees (it could be set in relation to the FOV of the VR headset). The default distance between the user and the painting is set to $D_{fov} = (P_w/2)*tan(50)$. The size of the room is dynamically set in relation to the size of the painting following these rules:

$$\begin{aligned} R_l= & {} \text {min}(2D_{\text {fov}},4) \end{aligned}$$

(1)

$$\begin{aligned} R_w= & {} 4+P_w \end{aligned}$$

(2)

$$\begin{aligned} R_h= & {} \text {min}\left( 3.5,C_y+\frac{P_h}{2}+2\right) \end{aligned}$$

(3)

The user is placed at position $(U_x,U_z)$ with $U_x=C_x$. The minimum default distance between the user and a painting is set to 0.5 m. If $(D_{\text {fov}}<0.5)$ then $U_z=3.5$ else if $D_{\text {fov}}<2$ then $U_z=4-D_{\text {fov}}$, otherwise $U_z=D_{\text {fov}}$. Examples of paintings presented in the VR environment are provided in Fig. 1.

Fig. 1

Examples of paintings presented in the VR environment

3.1 Quiz

The sequence of paintings and the corresponding questionnaire are described using the JSON (JavaScript Object Notation) data format. This data format is a text syntax which facilitates structured data interchange between different programming languages (Crockford and Morningstar 2017). After answering each question, the user’s position ((x, z) coordinate) is saved; where the (x, z) plane represents the ground, with the origin is at the center of the painting. The duration needed by the user to answer each question is also saved. The instructor sets a sequence of paintings that appear in the selected order. A set of questions is determined by the instructor for each painting. The structure is defined in the data 1. An example of data representing the metadata of a painting is presented in the data 2. It includes the file name containing the source of the image, its title, author, date, and description. The size of the painting is expressed in meters. The field related to the quiz contains a series of questions. The JSON schema is presented in Appendix A.

An example of a question is given in the data 3. It is possible to set a time limit corresponding to the duration (in second) of the question’s presentation. For questions with multiple choices, the list of answers contains the text mentioned for each possible answer. Depending on the questions, the order of the proposed answers can be shuffled or not. At the end of the sequence of questions, a file is saved containing the questions and the answers. Furthermore, it is possible to go beyond the usual questions with textual responses, i.e., multiple choice, multiple answers, by considering specific features that are only possible in the VR environment. For instance, it is possible to define different types of light to illuminate the room and, therefore, the painting. These different light options provide different experiences on how the painting is perceived, and they enhance different types of details in the painting.

Data 1

Sequence of paintaings

Data 2

Description of a painting

Data 3

Example of a question with multiple choices

3.2 User interface

For an immersive learning virtual reality application that should be used in educational settings with students from various backgrounds, including students who have no prior experience with VR and controllers, it is necessary to give to all users easy controls that can be mastered quickly. Potential users can be of all ages and may not have prior experience in gaming (Carrozzino 2019). While it may seem more natural to use virtual hands to manipulate and hold objects to replicate what can be performed in the real world, the proposed approach is to use a laser pointer (point and click interface) approach, i.e., like using a TV remote control, or a computer mouse: move the pointer to the desired location and click. Using a laser pointer allows to reduce locomotion within the VR environment and therefore to decrease VR-related symptoms. The virtual hands do not allow the same precision and speed as the pointer of a mouse on a computer screen. User would need to press buttons at a close distance, so the panel with the questions would need to be close as well, which can hide some elements of the scene. A small panel containing the questions would not be practical either as the text would be too small to be easily read (Fig. 2).

Therefore, the student needs to use only two buttons: the trigger for the selection and the pad or stick for teleportation. In the menu, we provide visual feedback when the user selects a command (change of color of the font of the text and the background of the button) to highlight the area that is selected (see Fig. 3). Then, for questions with multiple choices, we display a box that is filled or not depending on the selection of the answer. The panel with the questions and answers is always displayed next to the user, i.e., when the user teleports in a different location, the panel will remain next to the user. This project uses the SteamVR2 plugin for its implementation so the application can be used by various headsets (HTC Vive, Valve Index, Oculus Rift, Oculus Quest 2, and Windows Mixed Reality), which is a substantial improvement compared to a previous iteration for the creation of a VR museum where the application that was limited to HTC Vive headsets (Cecotti et al. 2020a). While this approach prevents the exploitation of functionalities that are directly dependent on the controller hardware such as the number of buttons; it increases the number of potential users without the implementation of a specific user interface for each headset (Fig. 4).

Fig. 2

Buttons used in the user interface for item selection and locomotion (left: Valve Index, middle: Oculus Touch, and right: HTC Vive)

Fig. 3

Example of question with multiple choices in the VR environment

4 Experimental protocol

4.1 Paintings selection

For the evaluation of the proposed system, we consider a set of 16 paintings from a variety of artistic periods: Renaissance, Baroque, Neoclassicism, and Romanticism. The paintings are listed in Table 1; the description includes the painting identifier, the name of the painter, the name of the painting, the date, and size (height $\times$ width) in cm. They have been chosen in relation to the likelihood that the students who participate in the study may know the paintings as they have been studied in class. In addition, they have been chosen for their different sizes and to highlight the contrast between small and large paintings. On the one hand, the Madonna in the Church by Jan van Eyck (1390-1441) and the Self-Portrait by Sofonisba Anguissola (1532-1625) correspond to small paintings; on the other hand, the Feast in the House of Levi by Paolo Veronese (1528-1588) and The Coronation of the Napoleon by Jacques-Louis David (1748-1825) correspond to large paintings, which require the user to look closer to grasp all the details. The spatial resolution in a painting is not directly related to its size: The Madonna in the Church is a small detailed painting, it is necessary to be close to the painting to see all the details. In VR, the user has to teleport in the scene in order to get close to large paintings. The painting from Hans Holbein the Younger (1497-1543), The French Ambassadors, was chosen as it includes an example of anamorphism. An anamorphism is a distorted projection requiring the viewer to occupy a specific vantage point to see an object in its true dimensions. The painting entitled Looking Down Yosemite Valley by Albert Bierstadt (1830-1902) was chosen, as students participating in the study live 2 h away from Yosemite National Park. The texture from all the paintings was found in open-access resources such as Wikipedia or on the website of the museums where they are currently exposed. None of the paintings are present on the West coast of the USA; hence, they are all difficult to access for the students. For the evaluation of the system, we consider two modalities for presenting the paintings and their corresponding questions: (1) the VR condition (the proposed system) and (2) the desktop condition as a baseline, using the Canvas Learning Management System (LMS) (Instructure 2019). The paintings are decomposed into two sets: S1: 1-8 and S2: 9-16. They are depicted in Fig. 5.

Images of paintings can be found online, free of cost. They can be utilized by researchers, art history instructors, and educational institutions who wish to incorporate the images of paintings into learning materials. High-resolution images can be obtained from different sources (e.g., Wikipedia and Wikimedia Commons) where famous painters and their most renowned paintings are provided. In Wikipedia, the number of paintings depends on the selected language in the page; it is likely to find more information in the language corresponding to the country of origin of the author. In addition, it is worth mentioning that multiple errors can be found between the information related to the size and the years of some paintings; the height and width are often swapped depending on the source that is used. The images that were selected for the experiment in this paper come from the Google Art project or from the museums where the paintings are displayed. The application can be easily extended with images from The Metropolitan Museum of Art which gives open access to images and metadata of about 300 paintings (The Met 2023). The Getty Museum also provides 727 high-resolution images of paintings through its Getty Search Gateway (https://search.getty.edu/gateway/landing). The paintings’ resolutions change substantially across museums. The low quality of the images prevents their addition into a VR-based art gallery. For teaching materials for art history courses with a focus on the early modern era, images are typically in the public domain. Images of paintings from painters before the twentieth century are in the public domain. There are multiple reasons: 1) The work is in the public domain in its country of origin and other countries and areas where the copyright term is the author’s life plus 100 years or fewer; and 2) the work is in the public domain in the USA because it was published (or registered with the US Copyright Office) before January 1, 1926.

Fig. 4

Example of anamorphism from The French Ambassadors, as shown from the VR environment. The skull is circled in red. Left: frontal view and right: lateral view from the right side

Table 1 List of paintings used in the evaluation

Each painting has a corresponding set of five questions with multiple choices, with up to four different answers.

Fig. 5

Selected paintings for the evaluation (Group 1: 1-8 and Group 2: 9-16)

4.2 Participants

Thirty-five healthy adults participated in the study (mean age: $22.51\pm 9.1$, min: 18, max: 65; genders: 14 females, 20 males, and 2 others). Eighteen participants had corrected vision, 32 were right-handed. All the participants were undergraduate students at California State University, Fresno, and had at least one course in art history, such as ARTH 11 (The Early Modern World) and/or ARTH 12 (Asian Art). All the students had prior experience with the paintings that were presented during the experiment as they were presented in courses they have taken. These students have also a better ability to appreciate the paintings compared to students who have not studied art history. Twenty-one have already used VR. Twenty-five participants played video games, 17 played first person shooter games. There was no financial reward provided for the participants, only extra credits in their art history course for their participation, independent of their performance. The Helsinki Declaration of 2000 was followed while conducting experiments.

4.3 Design

Participants in the experiment had to answer a series of questions with multiple choices (four choices) on eight paintings in the VR condition and the desktop condition with Canvas LMS. The desktop condition with Canvas LMS represents the baseline condition. All the participants are students who use Canvas LMS for their courses, so they are familiar with its user interface, and they can complete a quiz. It represents the condition that is currently used by art history instructors for providing learning materials and quizzes. The paintings were separated into two groups: 1-8 and 9-16. The VR controls and the task were explained to the participants prior to the experiment. The order of the conditions and the groups of paintings associated with each condition (Canvas and VR) were randomized across participants so all the questions were given in both modalities. A screenshot of a question on the Canvas quiz is depicted in Fig. 6. While it is possible to create individual questions with radio buttons for questions with multiple choices, we have grouped all the questions related to a painting into a single Canvas question, where the questions with multiple choices are a “fill the blank” type of question. The user has to type directly the number of the answer in a textbox. Such a presentation also saves screen space. The quiz on Canvas LMS was integrated in the Canvas LMS courses related to art history (ARTH 11 and ARTH 12). Students had to scroll through the page to access the questions related to each painting. Quizzes in both the VR and Canvas LMS conditions were graded automatically.

Fig. 6

Example of a question with multiple choices in the desktop condition (Canvas)

4.4 Materials and software

For this project, we used the Valve Index and HTC Vive with Deluxe Audio Strap headsets and peripherals. Unity 2019.4.8f1 was used for the implementation of the project with the SteamVR plugin to allow different types of VR headsets to be used (Unity Technologies 2019). The evaluation of the application was conducted on a desktop with an i7-6700K 4.00 Ghz, 32 GB Ram, Nvidia Geforce GTX 2080 Super, Windows 10 Enterprise 64 bits.

4.5 Performance evaluation

Students had to complete the National Aeronautics and Space Administration Task Load Index (NASA-TLX) and System Usability Scale (SUS) tests for both the VR and Canvas condition. The NASA-TLX assesses the workload (Hart and Staveland 1988). It consists of six scales (subjective effort, mental demand, temporal demand, physical demand, perception of performance, and frustration). Each scale is ranked with values between 1 and 20. The SUS provides a reliable tool for measuring usability. It includes a 10-item questionnaire with five response options, from strongly agree to strongly disagree (Brooke 1986). In addition, for the VR condition, participants had to answer the questions from the virtual reality sickness questionnaire (VRSQ) (Ames et al. 2005), and the Reduced Instructional Materials Motivation Survey (RIMMS) questionnaire (Loorbach et al. 2015). The VRSQ consists of 13 symptoms found to be the most useful. It includes eight general body-related symptoms questions and five eye-related symptoms questions. In each question, the score goes from 0 (none) to 6 (severe). The RIMMS includes 12 items that are based on the 36-item situational measure of people’s reactions to instructional materials considering the ARCS Model. The full Instructional Materials Motivation Survey is based on the ARCS Model of Motivational Design; it focuses on Attention, Relevance, Confidence, and Satisfaction to motivate students. Additional questions were asked related to the background in art history of the participants. For the evaluation of the quiz, each painting is worth 1 point. For a given painting, all the questions have the same weight; hence, the maximum score is 8 points. In the subsequent section, we report the mean and standard deviation across all the participants. The value for Cronbach’s Alpha is denoted by $\alpha$ to measure the internal consistency of a questionnaire.

5 Results

At the level of the quiz itself, the score across the participants for the Canvas and VR conditions is $5.04\pm 1.7$ and $4.40\pm 1.2$. A Wilcoxon signed-rank test indicated no significant difference between both conditions, suggesting that both modalities allow us to assess the students in the same way. The average and standard deviation across 35 participants for the NASA-TLX and SUS are presented in Tables 2 and 3. The average usability score was $80.71\pm 12.43$ ($\alpha =0.70$) for the VR condition and $67.64\pm 18.19$ ($\alpha =0.87$) for the Canvas condition. The usability score for the VR condition is rated as an excellent usability (>80.3), and the score for the Canvas condition is rated as okay (about 68). The Cronbach’s alpha is good for acceptable for the VR condition and good for the Canvas condition. The average workload for the VR condition was $25.31\pm 15.62$ ($\alpha =0.82$) while it was $32.76\pm 18.79$ ($\alpha =0.80$) for the Canvas condition. In both cases, the workload can be considered as medium (Prabaswari et al. 2019). The Cronbach’s alpha is good for good for both conditions. A Wilcoxon signed-rank test revealed a significant difference between the VR and Canvas conditions for both the NASA-TLX test ($p<0.05$) and the SUS test ($p<10e-4$). For the VR and Canvas conditions, there is a moderate negative linear relationship between workload and usability ($-$0.48 for the workload and $-$0.53 for the usability). These results show that when the usability increases, the workload decreases.

Table 2 NASA-TLX evaluation

Table 3 System usability evaluation

The average and standard deviation across participants related to the questions of the virtual reality sickness questionnaire are presented in Table 4. All the values are below 1, except for the blurred vision which is at 1.23. The results suggest that some participants did not properly adjust the VR headset on their head, despite the requests prior to the experiments. The value for Cronbach’s alpha shows an excellent internal consistency with $\alpha =0.92$.

Table 4 Virtual reality sickness questionnaire (VRSQ) in the VR condition

The results of the Reduced Instructional Materials Motivation Survey are given in Table 5. The values are between 1 and 5, from “not true” to “very true.” The twelve statements assess (A), attention (A), relevance (R), confidence (C), and satisfaction (S). All the scores are above 4 (“mostly true”). The highest value is for question 12, highlighting the engagement of the users into the VR environment. The results of this questionnaire suggest that the virtual reality environment enhances students’ motivation during the task as they are transported directly in front of the learning materials. The value for Cronbach’s alpha shows an excellent internal consistency with $\alpha =0.91$.

Table 5 Reduced instructional materials motivation survey (RIMMS) in the VR condition

The correlations across participants, with the bias and R-square values, between the different questionnaires for the VR condition are provided in Fig. 7. The matrix of correlation coefficients is given in Table 6. It indicates the extent to which some questionnaires are related and may provide the same trends across users in the game. The correlation coefficients between NASA-TLX and SUS, NASA-TLX and VRSQ, NASA-TLX and RIMMS, SUS and VRSQ, SUS and RIMMS, VRSQ and RIMMS indicate a moderate correlation. There exists a low correlation between RIMMS and the quiz results ($-$0.38). The correlation values between the VRSQ and NASA-TLX, SUS, and RIMMS show how VR symptoms can impact the workload and the usability.

Fig. 7

Correlations between the different questionnaires for the VR condition

Table 6 Matrix of correlation coefficients

Besides the standardized questionnaires, we also collected subjective feedback from the participants. Additional questions were given to the participants at the end of the session related to the VR condition. To the question “Navigating in the VR environment was:”, with answers going from easy (1) to difficult (5), the average answer is $1.34\pm 1.77$. To the question “Answer the quiz questions in VR was:”, the average answer is $0.91\pm 1.21$. About 77.14% of the participants prefer the VR condition to the Canvas condition. In addition to the questionnaire, students were able to give some justifications about their answers. Some students mentioned that they absolutely prefer the VR modality because “it helps establish an immediate interest and emotional connection that textbooks don’t.” VR feels more hands-on and personal, it allows users “to see the paintings in real size, it is more fun, more engaging, as it provides up close and real-life interpretations that a book can’t give.” A student mentioned that VR is more entertaining, and this student felt that she would learn more easily. “VR gives more depth and impact to the artwork.” A student appreciated that it is possible to see the actual size and proportions. Yet, some students highlight that textbooks are more accessible and easier to understand. Overall, the survey given to the students gave positive results about the use of the VR application. Some students requested to use the application longer and to see other paintings that were not included in the test. These students performed the best in the test and were the most interested in art history.

6 Discussion

We have proposed a new learning tool using fully immersive virtual reality that combines a virtual museum with multiple choice questions, allowing instructors in art history to assess their students by presenting paintings in their real size in a realistic environment. The visual analysis of paintings is organized in three steps. The first step is observation. It means closely looking at and identifying the visual attributes of an artwork, trying to describe them carefully and accurately in the student’s own words. This phase encourages students to look, think, and find good language to communicate what they notice, all without reading about the work. This phase tends to be the most challenging for students, so they can spend adequate time modeling and rehearsing these skills in VR; the painting size matters and can indicate where the painting should be displayed. Focusing on visual elements such as color, line, space, texture, light, and space will help temporarily suspend any symbolic or interpretive impulses, which is supported by the VR modality. The second step is about the analysis, requiring students to think about their observations and try to make statements about the work based on the evidence of their observations. This phase encourages students to think about how the specific visual elements they have identified combine to create a whole, and what effect that whole has on the viewer. Using VR technology and sharing experiences with multiple students could be an opportunity to emphasize the legitimacy of multiple viewpoints and voices in the museum experience. A visual analysis does not require any research, though it does have a central argument or thesis. The third step is interpretation. The difference between visual analysis and interpretation is research. To use visual analysis as the basis for an interpretation of an artwork, students discuss and formulate research questions based on what they have observed and argued thus far. Students balance observations, description, and analysis with facts about the artist and historical context from trustworthy published sources. All these different steps can be performed with the students being able to appreciate artworks chosen by the art history instructor in the VR application.

There exists a lack of equity in access to arts and art education for youth, especially in communities that have little access due to geography, money, or physical handicaps. The proposed system is one step toward improving access to art in schools. VR applications are mainly at the stage of demos and are not fully deployed in the classroom. There are several obstacles to providing immersive learning experiences to students. First, there is the cost barrier as it requires having multiple headsets and the place to deliver learning activities. With a classroom with many students, it is not possible to have one VR headset per student. The instructor must create asymmetric learning activities in which some students use the VR application while other students are working on a different learning activity. In the worst case, some students can be really engaged while using the VR system while other students are just observers, waiting for the VR headset to be ready. Second, instructors should have the possibility to modify the content in relation to the needs of the course. While there exist many VR experiences and applications related to art history, none of them allow the content to be easily modified and none of them offer a way to assess the students with questions. VR experiences are limited to the ability to browse among art galleries and paintings. Allowing instructors without any programming experience to edit the file that contains the list of paintings and questions gives them the freedom to determine the paintings and their order, with the relevant questions as required by a course. The instructor can easily prepare the learning activity—with a summative or formative assessment directly included in the activity. Third, the COVID-19 global pandemic has raised the level of hygiene that is necessary during learning activities. Beyond the possibilities to have users experiencing motion sickness in some applications, wearing a VR headset is not without any constraints. The VR headset should be sanitized after being used by someone. The sanitation process includes the VR headset and the controllers.

The VR condition was better than the Canvas condition for both the workload and the usability as suggested by the evaluation of the VR system and its comparisons with the presentation of the paintings with associated questions. These results are positive and surprising as the VR modalities include multiple challenges in the way people process information within the environment and how they are interacting with elements within it. In both cases, questions were provided with multiple choices. Users had to type their answers in a textbox while all the controls could be done through the laser pointer in the VR condition.

The “wow” factor can have an influence of the results, as the immersive aspect of the application and the display can hide potential usability defects. Longitudinal studies with cohorts of users with different levels of experience with VR and video games could improve the reliability of the results and understanding the level of acceptance over time. The “wow” factor and novelty of VR may be counterbalanced with the comfort of using an existing tool such as Canvas. The novelty aspect can go against the VR modality as it can take time to get used to a new interface and learn how to navigate in the VR environment. The comparison with Canvas, as a baseline, is not just about comparing with a 2D modality, but it is about comparing with a pragmatic modality that is already implemented in the classroom by art history instructor.

The experiment took place at California State University (CSU), Fresno, which is far away from large cities that have museums. Students at CSU Fresno have less opportunities to visit large art museums. The Fine Arts Museums of San Francisco and the Getty Center in Los Angeles are both about 3-4 h away from Fresno. Students who have direct access to art museums, living close to large cities, may not have the same enthusiasm as the students who participated in the experiment. Ideally, the participants of these experiments should experience the real paintings in museums where they belong and determine the extent to which the virtual reality condition allows to transcribe faithfully that in-person view and study of the original artwork. It is worth noting that some real museums may not provide the best experience for studying painting as the crowd and the environmental noise can prevent an observer from coming to an appropriate distance to the painting and focusing on its meaning. Comments from participants indicated that while the application is attractive, it should remain a way to provide formative feedback, to enjoy the experience without having the stress of the grade.

The results indicated a high usability and a medium workload. The workload performance could have been enhanced as the definition of what a task is can be a challenge and some participants did not answer as the task was defined. While the task was to go through the paintings and their questions, some participants felt that the task completion was related to their own ability to successfully complete the questions. Some participants rated the level of completion of the task higher (toward failure), despite the instructions that the task was to go through the paintings and answer the corresponding questions. These participants felt that they were failing the task while they were only failing the test, i.e., they did not know the correct answers. The usability of the VR application is ranked as excellent, with all the controls behind accessible through the left and the right hand, using only two buttons on the controllers: the trigger for the selection of items with the laser pointer, and the stick for the teleportation. The good performance of the VRSQ is probably related to how users can interact and move within the VR environment. With the dynamic creation of the rooms and the automatic positioning of the user in the room, users do not need to teleport themselves to observe the paintings. The main challenge when setting up the VR headset was to determine if the headset was well placed or not in relation to the expectation of users: Participants had to be reminded that a blurry screen was not a problem from the application, and they had to adjust the position of the headset.

The current application is not a serious game and remains a quiz with a series of questions that should be answered by someone with prior knowledge of art history. Potential improvements include the addition of gaming components and hints to allow users with no prior knowledge of art history to still be able to complete the questions and have a sense of accomplishment. By using different levels of difficulty, the quizzes can offer a scaffolding to support student learning. If a question is answered incorrectly, the student could be redirected to an additional question with support. Scaffolding provides a struggling student with support that fades away as the student becomes more independent. Additionally, providing a mode for instant feedback when a question is answered incorrectly can be used as a learning tool and practice quiz. Gamification approaches using models such as the learning mechanics and game mechanics, with scoring and feedback would help reduce the overall workload and improve the usability (Cecotti et al. 2021; Proulx et al. 2017). The questions are currently set to individual paintings, potential new learning activities would include the presentation of multiple paintings within the same room, where students can compare multiple paintings. When appreciating visual art such as paintings, it is critical to have high-resolution paintings and VR headsets that provide a high resolution.

While it is possible for an art history instructor to add paintings and questions automatically in the application without knowledge about programming, it is still necessary to understand the structure of JSON files and the overall organization of the quizzes. Future work will include the creation of automatic questions and answers based on the meta-data related to the paintings, so it would be possible to set questions for asking the artistic period, the painter, the date, without having the instructor typing these questions. However, rich questions related to the content of the paintings would still require the help of the instructors, but they could use the help of natural language processing and machine learning for setting questions based on the description of the paintings and their visual content.

7 Conclusion

Fully immersive virtual reality is able to reach the classroom in many academic fields, including arts and humanities. It is necessary to facilitate the use of such technology by allowing instructors to modify their content to satisfy the needs of their courses. This study represents one step toward the deployment of VR applications in the classroom in art history courses. We have proposed a customizable VR-based application that propels students into a virtual environment in which they can browse among paintings with associated questions. The VR environments are generated in a procedural way based on the paintings and questions selected by the art history instructor. Students can travel through space and time without stepping out of the university while discovering cultural and historical landmarks. More importantly, the application is fully customizable and set automatically based on the chosen paintings and questions that the instructors would like to use. The VR application has been compared with a desktop modality with the paintings and questions presented through the Canvas learning management system as a baseline. The comparison indicated better usability and lower workload with the VR condition, with no difference in terms of students’ performance for answering the questions. These results highlight the maturity of fully immersive VR learning applications, which can be easily deployed in the classroom and customized by instructors.

A: Appendix

Data 4

JSON Schema for a sequence of paintings with questions