Abstract
Significant advances in virtual reality (VR) technology have called into question the traditional methods of cinema storytelling and dissemination. New VR devices, such as the Meta (Oculus) Quest, have expanded the possibilities for viewing movies. The purpose of this study is to compare the emotional and cognitive impacts of VR and traditional 2D movies. In this study, sixty volunteers were divided into two groups and presented a movie (Gala) in 2D or VR format. We employed a multimodal method to assess the cognitive and emotional effects of the film both during and after watching. Our technique combined self-reports, interviews, questionnaires, and objective heart rate and EEG brain activity data. After quantitative and qualitative evaluation, it was discovered, that regardless of media, there was a substantial influence of the movie on the emotional state of the participant’s mood. Moreover, compared to the traditional 2D-movie, the VR movie led to more consistent and robust positive effect on all aspects of self-rated affect. The difference in self-reported mood was corroborated by reduced EEG amplitudes in the beta frequency band, indicating higher levels of positive affectivity, which was only observed for the VR movie. Lastly, the VR movie also leads to overall higher self-rated immersion and engagement than the 2D version. Our results highlight the potential of VR movies to engage and emotionally affect audiences beyond traditional cinema. Moreover, our study highlights the value of using a multidisciplinary method for analysing audience impacts.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Due to its ability to fully immerse the viewer, movies are a kind of entertainment that are renowned for their capability to elicit and alter emotional and cognitive states (Gross and Levenson 1995). A series of emotional, cognitive, and perceptual processes are sparked by cinema, and they occur during an experience that changes over time (Hasson et al. 2008).
Virtual reality (VR) is an immersive technological paradigm that simulates multisensory environments through computer-generated stimuli, inducing perceptual experiences resembling physical reality (Rauschnabel et al. 2022). By employing specialized hardware and software, VR systems exploit visual, auditory, and haptic cues to engender an illusory sense of presence and interactivity within a digitally constructed realm (Korkut and Surer 2023). Three fundamental attributes underpin VR irrespective of its application: first, Immersion, which engenders an enveloping verisimilar milieu incorporating visual and auditory elements, achieved through the synchronization of user actions, such as head and body movements, with corresponding virtual world dynamics; second, Presence, inducing a sense of integration within the virtual environment, emerging in response to a calibrated level of immersion and manifesting in the interplay between the natural and immersive technological systems; and third, Embodiment, often facilitated by avatars, which synergistically merges virtual and corporeal entities, enabling users to navigate and interact within the virtual realm (Slater and Sanchez-Vives 2016).
Films like Bonfire (directed by Eric Darnell 2019) (CIFF45 2020), The Line (directed by Ricardo Laganaro, 2020) (Roettgers 2019), The Key (directed by Celine Tricart 2019) (Tricart 2019), and Wolves in the Walls (directed by Pete Bellington, 2018) (Billington and Shamash 2018) are popular examples of immersive VR movies. However, a thorough and empirical analysis of the cognitive and emotional effects of VR movies has yet to be completed.
The term ‘user experience’ (UX) has a broad range of meanings (Forlizzi and Battarbee 2004), varying from traditional usability to hedonic, beauty, affective, or experiential aspects of technology use. Yet, frequently critiqued for being vague, elusive, ephemeral (Hassenzahl and Tractinsky 2006). Consequently, UX is probably not an appropriate term to use when referring to VR movie audiences and subsequently analysing cognitive and emotional effects.
Therefore, drawing from previous research (Abdullah et al. 2021; Marechal et al. 2019; Saganowski et al. 2020), we propose a multimodal approach to assess audience impact, which can also contribute to the cognitive movement in film theory. Our approach comprises of a combination of self-report (interviews, questionnaires) and objective measures (heart rate and brain activity measures) to measure the cognitive and affective impact of new generation movies. More specifically, in the current study we compare a VR movie with a 2D version of the same movie created using a virtual production process (Carpio et al. 2023). This approach might be valuable for the film industry to better assess its content and may open the way for a new interdisciplinary field of research application and new job roles (Carpio and Birt 2021).
1.1 Perceptional cinema
In traditional filmmaking, a variety of cinematic techniques such as editing, close-ups, and montage are used to direct viewers’ minds when watching a movie. These techniques affect how viewers respond to the movie (Hasson et al. 2008). VR movies provide the user with an alternative realm of simulated experiences, where users can physically acknowledge their existence within that world, but nevertheless surrounded by intangible environments (Girvan 2018; Maravilla et al. 2019; Parger et al. 2018; Shin 2018), this links back to the greater theory of Virtual Worlds.
In addition, to simulating an experience, VR also provides the ability to use avatars to move around the virtual space (Maravilla et al. 2019; Tüzün and Özdinç 2016). This is possible, as avatars provide direct means of interaction within virtual environments by combining physical and simulated properties between virtual bodies and virtual objects (Waltemate et al. 2018).
Finally, in traditional cinema using the 180° editing rule contributes to the movie’s continuity. Once the cameras are placed on the chosen side of the axis space (as shown in Fig. 1), all other shots should be recorded from the same side of the axis, as a cut across the line would therefore create a discontinuity (Smith 2010).
In contrast, in cinematic VR the 180° rule is challenged, as the camera or viewpoint is placed in a 360-degree space breaking the axis (Rothe et al. 2019). As a result, the scenes are constructed from a single 360-degree long take angle, allowing viewers freedom of view. In this regard, the audience’s perception has changed. Consequently, the same rules do not apply in a cinematic VR scenario and new rules must be studied to make better use of the medium.
Connecting cinema and cognitive neuroscience research is part of an effort to understand and explore the connections between the human mind and art (Kawabata and Zeki 2004; Kim and Blake 2007; Livingstone and Hubel 2002; Ramachandran and Hirstein 1999; Zeki 2002). Hasson et al. (2008) proposed Neurocinematics, an approach which suggests using neuroscientific techniques to measure the impacts of cinematic experiences on viewers. One promising neuroscientific technique for this endeavour is the non-invasive recoding of electric brain potentials using EEG (Lim et al. 2013). EEG has been used extensively to study various cognitive, emotional, and perceptual processes (Bhayee et al. 2016; Krigolson et al. 2017). In addition to brain potentials, physiological measures, such as heart rate and heart rate variability, have also been used in the study of the UX within entertainment technologies (Mandryk et al. 2006; Rebelo et al. 2012).
Finally, in addition to those objective measures of viewer experience, it is also critical to gather subjective experiences in a systematic fashion, such as by using validated self-rating scales and structured interviews (Urquhart et al. 2003). This supports the objective measures, with richness and feedback on the individuals experience and feelings.
Insufficient research has delved into the contrasting viewer effects between traditional 2D cinema and 3D virtual reality movies, particularly regarding the dearth of objective data pertaining to user experiences. This study thus seeks to address this gap by investigating whether discernible disparities exist in the subjective and objective dimensions of emotional response and physiological reactions between the two mediums.
2 Methodology
In regard to the mood and physiological measures, the study followed a pre/post-design. It included two distinct and independent groups: one group experienced the 2D version of the movie, while the other group experienced the movie in VR format. It is important to note that the assessment of engagement and immersion with the movie occurred solely after the exposure period.
2.1 Participants
The sample size was estimated a priori using G*Power (version 3.1.9.6) (Faul et al. 2007). To reliably detect pre- vs post-movie-exposure changes, at a medium effect size of d = 0.50, with the alpha level of significance set at 0.05 (two-tailed), and power at 80%, it was determined that 27 participants were required per group.
Participants were a convenience sample, recruited via advertising, consisting of 60 voluntary students and university staff from Bond University Gold Coast. Collectively, all 60 participants completed the study procedures (30 males and 30 females). Their ages ranged from 18 to 75 (mean age = 33.77) years. No one received individual feedback on their results. The study consisted of two groups, the VR group and the 2D group. The final sample was 28 participants (14 male and 14 female) for the 2D and 32 participants (16 male and 16 female) for the VR study. Due to technical errors, 1 alpha, 1 beta, and 1 HRV data assets from the objective measures are missing.
2.2 Self-reports
Self-report measures such as surveys, questionnaires, and interviews are commonly used in social sciences to investigate complex psychological well-being factors (Kahneman and Krueger 2006), for its ability to gather large amounts of sample data in a way that is inexpensive and fast. Nevertheless, self-report evaluations deal mainly with conscious measures, consequently, the presence of biases is eminent, and it may threaten its validity (Caputo 2017; Nestor and Schutt 2018; Vesely and Klöckner 2020).
2.2.1 Positive and negative affect scale (PANAS)
A popular self-report measured which was used in this study is the 20-item International Positive and Negative Affect Schedule known as PANAS (Watson et al. 1988). This is used to evaluate participants’ subjective experiences on two mood scales:
Positive affect (PA), this consists of 10 items (positive words: 1, 3, 5, 9, 10, 12, 14, 16, 17, 19), where participants are rated in regard to feeling ‘attentive’ or ‘inspired.’
Negative affect (NA), this consists of 10 items (negative words: 2, 4, 6, 7, 8, 11, 13, 15, 18, 20), where participants are rated in regard to feeling ‘upset’ or ‘nervous.’
The participants were asked to indicate how they felt before and after the movie experience on a five-point Likert scale spanning from ‘very slightly or not at all’ (1) to ‘extremely’ (5). As a result, lower PA scores indicated lower levels of positive mood while higher levels of positive mood. Equally, lower NA scores represented lower levels of negative mood and higher scores represented higher levels of negative mood.
2.2.2 Self-assessment manikin (SAM)
Another popular mood measure scale used within this research is the Self-Assessment Manikin (SAM) (Bradley and Lang 1994), which is extensively used for assessing the 3-dimensional structure of emotional events, situations, and objects.
The SAM consists of a set of 18 bipolar adjective pairs (see Fig. 2). Each structure is rated along a 9-point scale which results in 18 ratings generate scores on the dimensions of pleasure, arousal, and dominance. The SAM is a nonverbal illustration of graphics that depicts various points of affective dimensions. The SAM scales from a smiling, happy figure to a frowning, unhappy figure to represent the pleasure dimension, and scopes from an excited, wide-eyed figure to a relaxed, sleepy figure for the arousal dimension (Bradley and Lang 1994). The SAM has a strong test–retest reliability after two years (r = 0.96–0.97; Kanske and Kotz 2010). The subscales of the SAM correlate well (0.56 < r < 0.90) with physiological measures associated with emotional expression (i.e. heart rate, blood pressure, skin conductance; Azarbarzin et al. 2014).
2.2.3 Immersion and emotional engagement
We queried the participants’ degree of emotional engagement in the two movie conditions via a single quantitative item using a 5-point Liker scale (‘How emotionally engaged were you in the movie’ 1 is less, 5 is more). In addition, we posed 10 open-ended interview questions probing participants’ level of immersion and emotion that were subsequently qualitatively analysed. The questions range from how participants felt about the characters, what scenes they preferred, and overall experiences.
2.3 Objective measures
Recent advancements in mobile EEG and heart rate devices have also been used for research purposes (Mavros et al. 2016), as previous studies have shown that temporarily elevated heart rates can be an objective arousal/stress indicator (Kim et al. 2018; Vrijkotte et al. 2000). In this regard, heart rate was measured using a mobile Elite HRV CorSense device (https://elitehrv.com). A few studies have employed mobile heart rate devices for being precise enough, non-invasive, inexpensive, and easy to use (Sloan et al. 2017; Vecchiato et al. 2010; Yetton et al. 2019).
Moreover, brain activities were recorded using the Muse EEG portable headband device (https://choosemuse.com/) see Fig. 3.
Portable Muse EEG devices can reliably replicate EEG effects related to emotion/stress (Bhayee et al. 2016; Krigolson et al. 2017). This is done through the devices’ four main dry sensor electrodes: AF7 (left anterior frontal electrode) and AF8 (right anterior frontal electrode), which records the frontal areas of the forehead, TP9 (left temporal-parietal) and TP10 (right temporal-parietal), which records spontaneous electrical brain activity on the outer side of the forehead (Krigolson et al. 2017).
2.4 Motion sickness considerations
Ethical questions regarding motion sickness were addressed at each phase of the study. Consequently, good design practice was used in the development of the movie, including reasonable frame rate. Therefore, no acceleration, no action, no teleportation were implemented. The VR design only includes physical walking motion. Our procedure also ensured that participants were aware that they could abort the study at any time in case of motion sickness or any feeling of discomfort. However, none of the participants indicated that they felt motions sick.
2.5 Movie set-up and baseline measures
The showcased movie was Gala (Carpio 2022), as it was developed for the purpose of this study and in line with published virtual reality movie production guidelines (Carpio et al. 2023). Before and after the movie experience, participants were asked to complete the self-report surveys and participant objective data were measured. The movie screening and measures were captured in a self-contained soundproof cinema room (6.8 × 3.7 m) where participants were seated on a sofa. After the movie experience, participants were asked to complete an interview. We refrained from acquiring EEG and HR data during the movie experience, since this would have introduced movement artefacts in the VR version of the movie. Therefore, the measurements reflect the emotional and cognitive state immediately after the movie.
2.5.1 VR group
The used devices were a Meta Quest 2 headset and a pair of Sennheiser headphones in the self-contained cinema room as illustrated in Fig. 4.
The point of view (POV) VR version of the movie can be seen here https://osf.io/zrfqt/. VR users were able to move and look around the Gala scenes as if they were present in the scenario where the story was unfolding. However, the director directed the narrative arc and performance using the virtual avatars’ performances and light and music cues.
Moreover, this version of the movie contains interactions designed to involve the user within the story and the environment, as the user is directly addressed by the characters and asked to perform some actions within the virtual world.
2.5.2 2D group
The 2D group was presented the movie via a traditional projection screen (2.2 × 1.24 m) with users seated (5 m) from the screen. Sound was delivered via a Sony AV Receiver STR-DH540, and Gala movie in 2D can be seen here https://osf.io/ec6j7/.
In contrast to the VR version, traditional cinematography techniques were employed to direct the camera within the scene. Users were unable to choose where to look. Therefore, the user attention was directed by the film director using the Cinemachine virtual camera system within the Game Engine Unity. Cinemachine allows the director to place several virtual cameras to replicate a real film set within the virtual Unity scene. The resulting recorded footage can be exported as 2D videos.
3 Results
The descriptive results for all outcome measures are shown in Table 1.
The study involved two independent variables (i.e. the 2D-movie and VR movie groups) and four groups of dependent quantitative variables, i.e. self-reported affect (valence, arousal and dominance), heart rate variability, EEG activity (alpha, beta, and theta frequency bands), and self-reported immersiveness and engagement. Individual statistical data are available from https://osf.io/m842s/.
To decide on appropriate statistical tests, firstly, all dependent variables were assessed in regard to the normality of their distributions. Shapiro–Wilk tests (p > 0.05) indicated violations of normality assumptions for the following measures: EEG-alpha, self-rated affect, and self-rated immersion and engagement. Therefore, nonparametric statistics were employed for these variables. The difference score (pre-movie minus post-movie) was computed for all measures and used for the subsequent statistical analysis. The exception is the self-reported immersion and engagement measure, which was only measured post-movie.
3.1 Affect
For the 2D-movie group, nonparametric one-sample Kolmogorov–Smirnov tests were conducted on the difference scores of all three measures of self-rated affect. All three comparisons returned significant results (p < 0.001), indicating that watching the movie significantly increased valence (M = 0.014, SD = 1.96), decreased arousal (M = 0.11, SD = 2.39), and increased dominance (0.04, SD = 1.31). The effect sizes are, however, very small, i.e. d < 0.1 for all three. For the VR movie group, all three measures of self-rated affect indicated significant change pre/post-movie viewing (p < 0.001). The VR movie experiences lead to increases in valence (M = 0.50, SD = 1.08), arousal (M = 0.75, SD = 2.20), and dominance (M = 0.69, SD = 1.71). The effect sizes here were small to moderate, ranging between (d = 0.34) and (d = 0.46). Taken together the results suggest that while the 2D movie had a weak and inconsistent effect of self-reported affect, the VR movie had a consistent and robust positive effect on all aspects of self-rated affect.
3.2 Heartrate variability (HRV)
A one-sample t test was conducted on the pre/post-difference score for RMSSD as a measure of HRV. In the 2D group, viewing the movie led to a significant (t(27) = 3.56, p = 0.001) increase in HRV (M = 15.99, SD = 23.77), of moderate effect size, i.e. (d = 0.67). The same pattern was evident in the VR group, in which watching the movie also led to a significant increase in HRV (M = 10.11, SD = 19.01) of moderate effect size, i.e. (d = 0.53).
3.3 EEG
A nonparametric one-sample Kolmogorov–Smirnov test did not show significant change in alpha frequency for any of the two groups: 2D (p = 0.200) and VR (p = 0.063). Similarly, for the theta frequency range, one-sample t tests did not show significant change for any of the two groups: 2D (t(27) = 0.633, p = 0.532) and VR (t(32) = 1.90, p = 0.068). In the beta frequency range, however, the VR group showed a significant decrease in average amplitude (M = − 0.014, SD = 0.04), indicating a small effect, i.e. (d = 0.35), while the equivalent comparison for the 2D group was non-significant (t(26) = 0.46, p = 0.652).
3.4 Immersion and emotional engagement
The post-movie self-rated immersion and engagement scores were compared between the two groups using a nonparametric independent-samples Mann–Whitney U test, indicating a significantly (U = 630, p = 0.007) higher scores in the VR group (M = 8.23, SD = 1.11) than the 2D group (M = 7.27, SD = 1.54).
The answers to the interview questions for the VR group indicated that participants felt immersed and emotionally engaged by the VR version of the movie, triggering the following comments:
Participant 9, ‘My Favourite scene was the Crematorium because it was so real. I was about to hide’.
Participant 43, ‘The hover scene kind of reminds me when I was growing up and I watched Blade Runner just kind of took me to that and it was just quite exciting and fun and, and I wanted to drive. I wanted to explore the city. And when the woman began to speak to me, I was just drawn into the story’.
Participant 26, ‘The underwater scene, because it was very relaxing. And also, the little bits floating past just reach out and touch the little bits floating close to my eyes’.
For the 2D group, asked about whether they would be interested to see the movie in VR:
Participant 11, ‘it feels more realistic in VR. And then it's more personalized. And can bring more emotion to really relate to the movie itself. And you can explore more, instead of just to look at whatever is presented to you in traditional 2D films’.
Participant 39, ‘I think it's interesting in VR. And if it's used in a way such as that, like, it could like for example, movies such as David Attenborough's life on our planet, and kind of environmental stuff that tackles issues could hit more emotional in VR, which could lead to more change in people’.
Participant 47, ‘I like virtual reality. Makes you feel like you're more involved’.
4 Discussion
Regarding the question on what the differential emotional and cognitive effects of VR versus 2D movies are, the answer is not straightforward. Consider there are differences across visual field, haptic affordances, narrative structure, and audience participation in the story. VR’s core attributes of immersion, presence, and embodiment contribute to its unique impact (Slater and Sanchez-Vives 2016) eliciting experiences closely resembling the physical world (Rauschnabel et al. 2022) by fabricating a convincing visualization of presence and interaction within the digitally constructed environment (Korkut and Surer 2023).
However, both media achieve an emotional reaction on viewers that can be considered positive and of comparable size. In both cases irrespective of medium, it led to a significant effect on participant’s emotional state, which means that they are not always driven by the technology. This suggests that viewer’s perception is affected by the narrative. This occurs when the viewer assembles the movie’s temporal and spatial compositions by connecting shots logically and chronologically to the story. This process creates expectations and provokes emotions that move the viewer continually from the cognitive to the emotional stage while creating meaning and providing the viewer with an aesthetic experience based on the arrangement of the film units known as montage. Montage can be defined as ‘the juxtaposition of visual characteristics of successive images or scenes can produce emotional and cognitive reactions in the audience’ (Eisenstein 2014). Within this frame of research engagement is influenced using video and animation (Webster and Ahuja 2006), format, including audio and text (Chapman 1997; Jacques 1995; Laarni et al. 2004).
However, in other cases viewers were more affected due to technology, as it is in the case of the VR movie group, where viewers felt more immersed, emotionally engaged and aroused. However, in other cases, viewers were more affected due to technology, as it is in the case of the VR movie group, where viewers felt more immersed, emotionally engaged, and aroused; this aligns with the view that visual realism effects presence through place and plausibility illusion (Slater et al. 2009). The difference in self-reported mood was corroborated by reduced EEG amplitudes in the beta frequency band, which were only observed for the VR movie. Earlier EEG experiments showed increased beta responses upon application of emotionally negative stimuli (Güntekin and Başar 2010; Ray and Cole 1985). The results were also corroborated by the qualitative data, indicating that the viewers were more immersed and emotionally engaged in the VR version of the movie. The impact of salient actions introduced by a movie augments its influence because of increased arousal (Comstock et al. 1978). Based on a multidisciplinary study from film theory, communications, and psychology, when a movie provokes emotional arousal, the emotional perceptions experienced will influence the cognitive processing of the movie narrative (Roberts et al. 1996) that affect the audience’s attitudes and behaviours (Horowitz and Wilner 1976). In contrast with 2D films, VR movies allow for interactivity and exploration (Haywood and Cairns 2006); aesthetics and sensory appeal (Haywood and Cairns 2006; Hull and Reid 2018; Laarni et al. 2004); and socialization and communication with others (Haywood and Cairns 2006; Hull and Reid 2018), for instance, when interacting with CGI characters (avatars). In their 2018 study, Bindman et al. (2018) found that participants who identified as characters in the narrative reported higher levels of engagement and empathy, underscoring the significance of role comprehension in enhancing both story comprehension and empathy within immersive viewing platforms.
This can arguably lead to intellectual challenge (Douglas and Hargadon 2000; Mandryk 2004; Said 2004) and affective involvement (Laarni et al. 2004; Said 2004; Schraw 1998). These results also suggest that the emotional impacts of antagonistic characters are enhanced in VR. This heightened level of emotional engagement should be considered when translating traditional movies into VR. Consequently, it can be useful in creating strong emotions of disgust or disdain if used strategically.
There is considerable freedom in VR movie worlds that makes the user feel in control, as to when and how viewers let themselves be driven by the narrative, which in contrast to a 2D movie viewers feel more controlled as they have been directed to where to look at. Consequently, the immersion and engagement were clearly skewed towards the VR movie group, as evidenced by our self-report measure. This aligns with presence theory, which posits that the immersive and interactive nature of VR leads to a heightened sense of being present within the virtual environment (Slater et al. 2009). From this result, we may also conclude that within this study we are capable of getting an insight into people’s minds, which is objective and in principle not limited to how we can use that to infer cognitive state.
The excitement about the possibility is to interrogate cognitive-emotional processes using new scientific methods in a way that is feasible both financially and in terms of effort. As we are not restricted to big and expensive science laboratory equipment. Consequently, independent filmmakers and practitioners can have access to accessible equipment to look at the emotional effect of the crafts they create on their audiences. This can be of critical importance to help filmmakers as they can benefit from the feedback they receive, as it might help them improve on specific crafts (Singh et al. 2013; Zhuang et al. 2006). This can include the strategical use of characters within the film narrative by understanding audience disposition. Consequently, this allows filmmakers to create more original and meaningful film content (Brylla 2018). In addition, it can help viewers identify which films to watch (Parkhe and Biswas 2016).
The novelty of this approach could mark the beginning of the future of how we determine audience sentiment or audience connection to film and games. Currently, this is done using sentiment analysis. A downside of these techniques is that they can deliver a subjective opinion; this might be positive or negative (Parkhe and Biswas 2015; Yu et al. 2011). Sentiment analyses are used to study and understand the user’s emotions (Raison et al. 2012) towards a film. These studies have identified key features to encompass a film review and to score it in relation to the different film aspects. These aspects include acting, directing, and plot (Parkhe and Biswas 2016; Singh et al. 2013; Zhuang et al. 2006). Another weakness of subjective measures is that people often hide their emotions (due to social pressure) or do not know how to translate those emotions verbally. Therefore, by using a multimodal approach to measure the cognitive and emotional impacts of the movie experience, we can get a more comprehensive story within a more complex picture. Moreover, as the data show, objective measures have the advantage of not being biased by social pressures or expectations. Consequently, by combining them with subjective measures we can compare the results to analysis if the subjective measures support what we find in our objective measures.
4.1 Limitations of the study
Although both the VR and 2D versions of the study had a significant impact on participants’ moods, there are still certain issues that could be resolved in future studies. The first and most important restriction is that only one film from one director was analysed. To gain a wider view, it would be fascinating to compare several films from various genres by various directors. The employment of inexpensive equipment, despite meeting the goals, is a second restriction, as it is not the precision laboratory equipment. Finally, a future study might be expanded by including a bigger size sample of participants and using a variety of VR headsets.
5 Conclusions
This paper has proposed a multimodal method to assess the cognitive and emotional effects of VR movies by comparing Gala in its VR and 2D versions. Our technique combines self-reports, interviews, questionnaires, and objective data such as heart rate and EEG brain activity. In this study, sixty volunteers were divided into two groups (VR and 2D groups) to measure their mood, both during and after watching the movies. After qualitative and quantitative evaluation, it was discovered, that regardless of media, there was a substantial influence on the emotional state of the participant’s mood. Moreover, compared to the 2D-movie, the VR movie led to more consistent and robust positive effect on all aspects of self-rated affect. The difference in self-reported mood was corroborated by reduced EEG amplitudes in the beta frequency band, indicating higher levels of positive affectivity, which was only observed for the VR movie. Lastly, as expected, the VR movie also leads to overall higher self-rated immersion and engagement than the 2D version. Our results highlight the potential of VR movies to engage and emotionally affect audiences beyond traditional cinema. Moreover, our study highlights the value of using a multidisciplinary method for analysing audience impacts.
This technique should help filmmakers analyse content and encourages to further research to broaden the perspective related to new media storytelling through virtual reality.
References
Abdullah SMSA, Ameen SYA, Sadeeq MA, Zeebaree S (2021) Multimodal emotion recognition using deep learning. J Appl Sci Technol Trends 2(02):52–58. https://doi.org/10.38094/jastt20291
Azarbarzin A, Ostrowski M, Hanly P, Younes M (2014) Relationship between arousal intensity and heart rate response to arousal. Sleep 37(4):645–653. https://doi.org/10.5665/sleep.3560
Bhayee S, Tomaszewski P, Lee DH, Moffat G, Pino L, Moreno S, Farb NA (2016) Attentional and affective consequences of technology supported mindfulness training: a randomised, active control, efficacy trial. BMC Psychol 4(1):1–14. https://doi.org/10.1186/s40359-016-0168-6
Billington P, Shamash J (2018) Wolves in the walls: chapter 1. ACM SIGGRAPH 2018 Virtual, Augmented, and Mixed Reality
Bindman SW, Castaneda LM, Scanlon M, Cechony A (2018) Am I a bunny? The impact of high and low immersion platforms and viewers' perceptions of role on presence, narrative engagement, and empathy during an animated 360 video. In: Proceedings of the 2018 CHI conference on human factors in computing systems
Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59
Brylla C (2018) The benefits of content analysis for filmmakers. Stud Austral Cinema 12(2–3):150–161
Caputo A (2017) Social desirability bias in self-reported well-being measures: Evidence from an online survey. Univ Psychol 16(2):245–255. https://doi.org/10.11144/javeriana.upsy16-2.sdsw
Carpio R (2022) Gala, VR movie. https://osf.io/zrfqt/
Carpio R, Birt J (2021) The role of the Embodiment Director in virtual reality film production. Creat Ind J. https://doi.org/10.1080/17510694.2021.2017634
Carpio R, Baumann BJO (2023) Using case study analysis to develop heuristics to guide new filmmaking techniques in embodied virtual reality films. Creat Ind J 1:22. https://doi.org/10.1080/17510694.2023.2171336
Chapman PM (1997) Models of engagement: intrinsically motivated interaction with multimedia learning software University of Waterloo]
CIFF45 (2020) Bonfire https://www.clevelandfilm.org/films/2020/bonfire
Comstock G, Chaffee S, Katzman N, McCombs M, Roberts D (1978) Television and human behavior. In: Natl acad television arts sciences, vol 15, New York, pp 5–12111
Douglas Y, Hargadon A (2000) The pleasure principle: immersion, engagement, flow. In: Proceedings of the eleventh ACM on hypertext and hypermedia
Eisenstein S (2014) Film form: essays in film theory. HMH
Faul F, Erdfelder E, Lang A-G, Buchner A (2007) G* Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39(2):175–191
Forlizzi J, Battarbee K (2004) Understanding experience in interactive systems. In: Proceedings of the 5th conference on designing interactive systems: processes, practices, methods, and techniques
Girvan C (2018) What is a virtual world? Definition and classification. Educ Tech Res Dev 66(5):1087–1100. https://doi.org/10.1007/s11423-018-9577-y
Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cogn Emot 9(1):87–108. https://doi.org/10.1080/02699939508408966
Güntekin B, Başar E (2010) Event-related beta oscillations are affected by emotional eliciting stimuli. Neurosci Lett 483(3):173–178
Hassenzahl M, Tractinsky N (2006) User experience—a research agenda. Behav Inf Technol 25(2):91–97. https://doi.org/10.1080/01449290500330331
Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ (2008) Neurocinematics: the neuroscience of film. Projections 2(1):1–26. https://doi.org/10.3167/proj.2008.020102
Haywood N, Cairns P (2006) Engagement with an interactive museum exhibit. In: People and computers XIX—the bigger picture. Springer, pp 113–129
Horowitz M, Wilner N (1976) Stress films, emotion, and cognitive response. Arch Gen Psychiatry 33(11):1339–1344
Hull R, Reid J (2018) Designing engaging experiences with children and artists. In: Funology 2. Springer, pp 469–478
Jacques R (1995) Engagement as a design concept for multimedia. Canad J Educ Commun 24(1):49–59
Kahneman D, Krueger AB (2006) Developments in the measurement of subjective well-being. J Econ Perspect 20(1):3–24. https://doi.org/10.1257/089533006776526030
Kanske P, Kotz SA (2010) Leipzig affective norms for German: a reliability study. Behav Res Methods 42:987–991. https://doi.org/10.3758/BRM.42.4.987
Kawabata H, Zeki S (2004) Neural correlates of beauty. J Neurophysiol 91(4):1699–1705
Kim C-Y, Blake R (2007) Brain activity accompanying perception of implied motion in abstract paintings. Spat vis 20(6):545–560
Kim H-G, Cheon E-J, Bai D-S, Lee YH, Koo B-H (2018) Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investig 15(3):235. https://doi.org/10.30773/pi.2017.08.17
Korkut EH, Surer E (2023) Visualization in virtual reality: a systematic review. Virtual Real 27:1–34
Krigolson OE, Williams CC, Norton A, Hassall CD, Colino FL (2017) Choosing MUSE: Validation of a low-cost, portable EEG system for ERP research. Front Neurosci 11:109. https://doi.org/10.3389/fnins.2017.00109
Laarni J, Ravaja N, Kallinen K, Saari T (2004) Transcendent experience in the use of computer-based media. In: Proceedings of the third Nordic conference on human–computer interaction
Lim J, Quevenco F-C, Kwok K (2013) EEG alpha activity is associated with individual differences in post-break improvement. Neuroimage 76:81–89. https://doi.org/10.1016/j.neuroimage.2013.03.018
Livingstone M, Hubel DH (2002) Vision and art: the biology of seeing, vol 2. Harry N. Abrams, New York
Mandryk RL (2004) Objectively evaluating entertainment technology. In: CHI'04 extended abstracts on human factors in computing systems
Mandryk RL, Inkpen KM, Calvert TW (2006) Using psychophysiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25(2):141–158. https://doi.org/10.1080/01449290500331156
Maravilla MM, Cisneros A, Stoddard A, Scretching DD, Murray BK, Redmiles E (2019) Defining virtual reality: Insights from research and practice. In: iConference 2019 proceedings. https://doi.org/10.21900/iconf.2019.103338
Marechal C, Mikolajewski D, Tyburek K, Prokopowicz P, Bougueroua L, Ancourt C, Wegrzyn-Wolska K (2019) Survey on AI-based multimodal methods for emotion detection. High Perform Model Simul Big Data Appl 11400:307–324. https://doi.org/10.1007/978-3-030-16272-6_11
Mavros P, Austwick MZ, Smith AH (2016) Geo-EEG: towards the use of EEG in the study of urban behaviour. Appl Spat Anal Policy 9(2):191–212
Nestor PG, Schutt RK (2018) Research methods in psychology: Investigating human behavior. Sage Publications, New York. https://doi.org/10.2304/plat.2012.11.1.106
Parger M, Mueller JH, Schmalstieg D, Steinberger M (2018) Human upper-body inverse kinematics for increased embodiment in consumer-grade virtual reality. In: Proceedings of the 24th ACM symposium on virtual reality software and technology
Parkhe V, Biswas B (2015) Genre specific aspect based sentiment analysis of movie reviews. In: 2015 international conference on advances in computing, communications and informatics (ICACCI)
Parkhe V, Biswas B (2016) Sentiment analysis of movie reviews: finding most important movie aspects using driving factors. Soft Comput 20(9):3373–3379. https://doi.org/10.1007/s00500-015-1779-1
Raison K, Tomuro N, Lytinen S, Zagal JP (2012) Extraction of user opinions by adjective-context co-clustering for game review texts. In: International conference on NLP
Ramachandran VS, Hirstein W (1999) The science of art: a neurological theory of aesthetic experience. J Conscious Stud 6(6–7):15–51
Rauschnabel PA, Felix R, Hinsch C, Shahab H, Alt F (2022) What is XR? Towards a framework for augmented and virtual reality. Comput Hum Behav 133:107289
Ray WJ, Cole HW (1985) EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228(4700):750–752
Rebelo F, Noriega P, Duarte E, Soares M (2012) Using virtual reality to assess user experience. Hum Factors 54(6):964–982. https://doi.org/10.1177/0018720812465006
Roberts DS, Cowen PS, MacDonald BE (1996) Effects of narrative structure and emotional content on cognitive and evaluative responses to film and text. Empir Stud Arts 14(1):33–47
Roettgers J (2019) First look: trailer for venice-bound VR experience ‘the line’ (EXCLUSIVE). Variery. https://variety.com/2019/digital/news/venice-vr-the-line-trailer-arvore-1203315106/
Rothe S, Buschek D, Hußmann H (2019) Guidance in cinematic virtual reality-taxonomy, research status and challenges. Multimodal Technol Interact 3(1):19. https://doi.org/10.3390/mti3010019
Saganowski S, Dutkowiak A, Dziadek A, Dzieżyc M, Komoszyńska J, Michalska W, Polak A, Ujma M, Kazienko P (2020) Emotion recognition using wearables: a systematic literature review-work-in-progress. In: 2020 IEEE international conference on pervasive computing and communications workshops (PerCom Workshops)
Said NS (2004) An engaging multimedia design model. In: Proceedings of the 2004 conference on Interaction design and children: building a community
Schraw G (1998) Promoting General Metacognitive Awareness Instructional. Science 26(1):113–125
Shin D (2018) Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience? Comput Hum Behav 78:64–73. https://doi.org/10.1016/j.chb.2017.09.012
Singh VK, Piryani R, Uddin A, Waila P (2013). Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level sentiment classification. In: 2013 International mutli-conference on automation, computing, communication, control and compressed sensing (iMac4s)
Slater M, Khanna P, Mortensen J, Yu I (2009) Visual realism enhances realistic response in an immersive virtual environment. IEEE Comput Graphics Appl 29(3):76–84
Slater M, Sanchez-Vives MV (2016) Enhancing our lives with immersive virtual reality. Front Robot AI 3:74
Sloan RP, Schwarz E, McKinley PS, Weinstein M, Love G, Ryff C, Mroczek D, Choo T-H, Lee S, Seeman T (2017) Vagally-mediated heart rate variability and indices of well-being: Results of a nationally representative study. Health Psychol 36(1):73. https://doi.org/10.1037/hea0000397
Smith TJ (2010) Film (cinema) perception. In: Goldstein EB (ed) The sage encyclopedia of perception
Tricart C (2019) The Key. thekey-vr.com. Retrieved April 30 from https://thekey-vr.com/
Tüzün H, Özdinç F (2016) The effects of 3D multi-user virtual environments on freshmen university students’ conceptual and spatial learning and presence in departmental orientation. Comput Educ 94:228–240
Urquhart C, Light A, Thomas R, Barker A, Yeoman A, Cooper J, Armstrong C, Fenton R, Lonsdale R, Spink S (2003) Critical incident technique and explicitation interviewing in studies of information behavior. Libr Inf Sci Res 25(1):63–88. https://doi.org/10.1016/S0740-8188(02)00166-4
Vecchiato G, Astolfi L, De Vico Fallani F, Cincotti F, Mattia D, Salinari S, Soranzo R, Babiloni F (2010) Changes in brain activity during the observation of TV commercials by using EEG GSR and HR measurements. Brain Topogr 23(2):165–179
Vesely S, Klöckner CA (2020) Social desirability in environmental psychology research: three meta-analyses. Front Psychol 11:1395. https://doi.org/10.3389/fpsyg.2020.01395
Vrijkotte TG, Van Doornen LJ, De Geus EJ (2000) Effects of work stress on ambulatory blood pressure, heart rate, and heart rate variability. Hypertension 35(4):880–886. https://doi.org/10.1161/01.HYP.35.4.880
Waltemate T, Gall D, Roth D, Botsch M, Latoschik ME (2018) The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response. IEEE Trans Visual Comput Graphics 24(4):1643–1652
Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54(6):1063. https://doi.org/10.1037/0022-3514.54.6.1063
Webster J, Ahuja JS (2006) Enhancing the design of web navigation systems: the influence of user disorientation on engagement and performance. MIS Q 30:661–678
Yetton BD, Revord J, Margolis S, Lyubomirsky S, Seitz AR (2019) Cognitive and physiological measures in well-being science: Limitations and lessons. Front Psychol 10:1630
Yu J, Zha Z-J, Wang M, Chua T-S (2011) Aspect ranking: identifying important product aspects from online consumer reviews. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies
Zeki S (2002) Inner vision: an exploration of art and the brain. J Aesthet Art Crit 60(4):365–366
Zhuang L, Jing F, Zhu X-Y (2006) Movie review mining and summarization. In: Proceedings of the 15th ACM international conference on information and knowledge management
Funding
Open Access funding enabled and organized by CAUL and its Member Institutions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Carpio, R., Baumann, O. & Birt, J. Evaluating the viewer experience of interactive virtual reality movies. Virtual Reality 27, 3181–3190 (2023). https://doi.org/10.1007/s10055-023-00864-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-023-00864-2