Experiences in Virtual Reality: a Window to Autobiographical Memory

Abstract

VR-based paradigms could substantially increase the ecological validity of various psychological research topics as VR allows for submerging into real-life experiences under controlled laboratory conditions. In particular, in the field of mnemonic research, concerns have been raised that “laboratory memory” differs significantly from “real-life” autobiographical memory. Our study aimed to assess the immersive qualities of VR not only upon application but -more importantly- during the retrieval of the virtual experiences subsequent to a VR session. We presented participants with either a 360° VR or a 2D video of a motorcycle ride followed by an unannounced recognition memory task 48 h later. Increased retrieval success and delayed reaction times in the VR group indicate that immersive VR experiences become part of an extensive autobiographical associative network, whereas the conventional video experience remains an isolated episodic event.

Introduction

With the recent advent of virtual reality technology (VR), the toolkit for psychological research might get a powerful extension. While clinical research has already applied these methods (cf. Riva et al. 2016; Rizzo et al. 2004), experimental psychology has largely neglected them; VR-based paradigms could substantially increase the ecological validity (Kvavilashvili and Ellis 2004) of various psychological research topics as VR allows to submerge into real-life experiences under controlled laboratory conditions. In particular, in the field of mnemonic research, previous concerns have been raised that “laboratory memory” differs significantly from “real-life” autobiographical memory regarding its underlying mechanisms (e.g. Cabeza et al. 2004). Consequently, the question emerges if VR experiments could overcome the limitations posed by conventional laboratory experiments. To address this issue, at first, it has to be clarified whether VR is merely a fleeting visual or multimodal illusion or if the brain accepts the VR experience as an imposed alternative reality on a larger time scale. If the latter is the case, one will assume that the immersive qualities of VR do not only emerge upon application of the VR equipment (Burns and Fairclough 2015), but -more importantly- during retrieval of the virtual experiences subsequent to a VR session.

To test this assumption, we examined whether VR experiences are remembered as an integrated part of the personal biography (autobiographic memory) or merely as shallow episodic events (episodic memory). In general, autobiographic and episodic memory are subsystems of declarative memory (Renoult et al. 2012). Whereas episodic memory is characterized by unique events in a first-person perspective as well as their encoding context (Conway 2005; Coronel and Federmeier 2016; Grilli and Verfaellie 2014; Renoult et al. 2012; Tulving 1972), autobiographic memory comprises a larger set of operations (including episodic memory), namely self-reflection, emotional evaluation and semantic processes (Svoboda et al. 2006). Noteworthy, definitions of autobiographic memory are relatively inconsistent within the literature and the demarcation from episodic memory is not always clear: In a recent model of autobiographic memory unique but highly self-relevant experiences are not included (Renoult et al. 2016). However, pictures of such events are applied as an autobiographical condition in a frequently acknowledged imaging study (e.g. Cabeza et al. 2004).

There is nevertheless broad and stable consensus that personal relevance and particularly self-involvement are the key-elements of autobiographic memory (Roediger and Marsh 2003; Conway 2005). In order to avoid confusion within the present experiment, we use the term participation based memory (PBM) for singular episodic events with a high degree of self-involvement, observation based memory (OBM) for incidentally, and shallow acquired episodic memories.

We hypothesize that the more complex and full-featured configuration of PBM-related engrams entails longer retrieval times as compared to OBM (Burianova et al. 2010; Kemp et al. 2009). Furthermore, content in PBM is defined by higher personal relevance (i.e. salience), and should thus be remembered more accurately (Leshikar and Duarte 2012).

The higher personal relevance of immersive experiences might not only modulate memory consolidation, but also influence the evaluation of perceived affect. Actually, affective processing in connection with conventional immersive experiences, such as video games, has been under investigation for a long time (for critical review Ferguson 2007). As mnemonic and affective processing are intrinsically intertwined (Blaney 1986; Kensinger 2009), we hypothesized that upcoming affect in the VR and the video condition would differentially facilitate memory formation. More precisely, affect, which is generated or perceived under the impression of higher personal relevance, might have a higher informative value and thus exhibit stronger influence than a rather superficial affect. “Real emotions” point to a real danger or something pleasant, conversely, “screen emotions” might be equally strong, but do not point at something with an immediate effect on the organism and are thus processed differently.

Methods

Forty-three participants were recruited from Osnabrück University and gave informed consent. Participants were randomly assigned to the VR- or the video-encoding condition (VR: N = 22, Mage = 21.14, SD = 2.38, 20 right-handed, 10 female; Video: N = 21, Mage = 21.57, SD = 2.06, 20 right-handed, 12 female;). They had normal or corrected vision and were screened for psychological and neurological disorders.

Participants in the VR group were seated on a piano stool and placed their hands on bicycle handlebars. They wore a HTC Vive head-mounted display with headphones (see Fig. 1a). In the video condition participants sat in front of a 55′-monitor (visual angle: 32.94°× 55.87°), put their hands on the table and watched a 2D video (Video can be retrieved from https://osf.io/g3k97/?view_only=9efcfe0ad15f4d52b4a24ea1a5e5400f). The large screen allowed for covert and overt attention similar to the VR display. The video was a 28-min motorcycle ride in the region of Osnabrück recorded with a Samsung 360 Gear camera (Pre-experimental route knowledge = 6.69%). Classic rock and contemporary pop music were added to the video, as wind noise had an annoying effect and had to be completely removed. Both groups watched the same video. After the (virtual) drive participants rated their mood on a 5-point Likert scale from negative to positive. No participant aborted the experiment due to motion sickness or other reasons. Forty-eight hours later, they performed an unannounced recognition memory task on a 19′-screen at a distance of 120 cm (visual angle: 7.15° × 4.77°). Each trial consisted of a fixation with a randomized length between 500 ms and 750 ms followed by a picture of a scene for two sec (details below). The task was to indicate as quickly and accurately as possible via the left and right arrow key whether they recognized the scene or not. The key assignment was counterbalanced across participants.

Fig. 1
figure1

a Schematic representation of a virtual motorcycle ride with an HTC Vive head mounted display. Please note that the actual experience was a 360° video. b Results of the recognition task 48 h after the VR or video motorcycle ride: Retrieval success indicated by d’ is significantly increased in the VR group compared to the video group. Furthermore, the old/new classification in the VR group was prolonged as compared to the 2D video group. Standard error is presented

For the memory task, 90 screenshots were taken from the ride as well as from an equally long control ride. All screenshots had a slight tilt, as the stationary nature of the video condition would otherwise have led to an advantage, as within the video condition encoding and retrieval cues would be identical.

Results

The VR-group rated their experience significantly more realistic as compared to the Video-group (VR: M = 3.09, SD = 0.75; Video: M = 2.52, SD = 1.03; t(36.47) = 2.06, p < =.05; Cohen’s d = .633). They performed more than twice as well in the memory task (VR: M = .46, SD = .38; Video: M = .179, SD = .41; t(41) = 2.30, p < =.05; Cohen’s d = .713), and it took them significantly longer to reply (VR: M = 2.46, SD = 1.16; Video: M = 1.86, SD = 0.68; t(34.20) = 2.07, p < =.05; Cohen’s d = .631) (see Fig. 1b). No differences with respect to the post-ride mood were found (VR: M = 3.41, SD = 0.80; Video: M = 3.57, SD = 0.60; t(41) = 0.75, p = .456). Interestingly, the mood correlated positively with d’ within the VR-group, r = .427, p < =.05, but not within the Video-group, r = .272, p = .232. To complement our analysis, we additionally applied a Bayesian t-test considering the differences in retrieval success and reaction time. The resulting Bayesian factor replaces the traditional p-value by estimating the likelihood of the null versus the alternative hypothesis given a prior probability and the observed data. Unlike the traditional t-test, the Bayesian t-test thus can accept the null value (Kruschke 2013; Kruschke and Liddell 2017; Ly et al. 2016). For both t-tests, we used the default prior probability of 0.7071. As a result, for the retrieval success the Bayesian factor was 2.34, indicating that the likelihood of H1 is 2.34 higher than the null-hypothesis. The Bayesian factor for the mean reaction time was rather neglectable with 1.54 (Table 1).

Table 1 Absolute values of participants’ response over all subjects and for each group separately. ​Standard deviation is presented in brackets

Discussion

Given the assumption that reaction times and retrieval success are valid indicators for a widely ramified network of memory traces, we were able to show that PBM (“autobiographical memory”) and OBM (“episodic memory”) vary as a function of immersion during encoding of identical visual experiences. Crucially, subsequent memory performance was based on identical retrieval cues. Memory traces formed under the immersive sensation of a virtual experience are characterized by richer content and more elaborate associative networks. They thus bear a resemblance to PBM. Activation and retrieval of engrams in these networks are more time consuming opposed to video experiences, which remain isolated and shallow events and resemble OBM. The richness (level of detail) of PBM furthermore might promote correct recognition decisions (Levine et al. 2002).

Additionally, our results revealed that the post-ride mood did not differ between the groups, but clearly affected the memory consolidation process for PBM. The study replicates previous findings showing that positive emotions enhance memory performance (Ashby and Isen 1999). Importantly, this only holds for circumstances of high immersion, i.e. whenever the emotional evaluation is of self-relevance. Emotions perceived during personal or real experiences have a higher informative value and are potentially relevant for adaptive behavior compared to rather superficial emotions elicited by screen events. They thus have higher salience than those of shallow episodes.

These considerations lead to the question of the psychological harm of virtual experiences, similar to the debate about violence in video games promoting aggressive behavior. It should be noted that the effects of video games on well-being are rather small (for a recent meta-analysis see Ferguson 2015). They are even smaller than the effects of television (Sherry 2001), and are highly dependent on state (i.e., pre-game) affect (Unsworth et al. 2007). Video game violence seems furthermore not to be related to real-world violence (Markey et al. 2014). However, the (envisioned) immersion of VR applications might even more blur the border between the real world and computer games. Whereas the difference regarding immersion between different types of video games can be considered to be gradual, VR represents a new category of immersion. Conventional emotion regulation strategies aiming at the reduction of negative affect by reappraising a situation in non-emotional terms or by suppression (Gross 2002; Gross and John 2003) might not be equally applicable. Importantly, our data just gives a hint to another type of affective processing under the highly immersive conditions of VR, and we do not know if our results generalize to other affective domains.

A pioneering study investigating the effects of immersion on memory by comparing virtual reality with real events and a conventional Desktop-PC experience provided mixed results considering the relation between a sense of being present in the scene and memory performance (Mania and Chalmers 2001). Supposedly, the used prototype of a VR headset was technologically not advanced enough to create a fully immersive experience. Hence, the overall memory performance of VR was the lowest. However, the data gives the first hint that the VR application facilitates especially the formation of vivid memories. Generally, the underlying mechanism of autobiographical (PBM) and laboratory (OBM) memories seem to differ fundamentally (Roedinger and McDermont 2013). Roediger and McDermont likewise assume that real-life events are weaved into the daily narrative, whereas laboratory events are not. Indeed only a few overlapping regions seem to be involved in both types of retrieval (McDermott et al. 2009). Experience in virtual reality might take the same route into autobiographical memory. Importantly, a recent study comparing autobiographical and laboratory memory provides further evidence that commonly used paradigms used to study episodic memory are not approximations to real life events as the underlying neural substrates differ significantly (Chen et al. 2017).

Summing up, VR can be successfully applied to control for factors, which otherwise make it difficult to differentiate between PBM and OBM. It thus is a powerful tool to increase the ecological validity of future mnemonic research, complementing conventional findings on “laboratory memories”. On a more general level, future cognitive-psychological and clinical studies should take into account that experimental findings might vary as a function of immersion and self-relevance. Virtual reality actually deserves the term reality - with all its prospects and risks. Scientific, clinical and commercial applications benefit from the ability to administer virtual, alternative events. On the downside, disturbing or traumatic VR experiences also might have adverse long-term effects.

References

  1. Ashby, F. G., & Isen, A. M. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106(3), 529.

    Article  PubMed  Google Scholar 

  2. Blaney, P. H. (1986). Affect and memory: a review. Psychological Bulletin, 99(2), 229.

    Article  PubMed  Google Scholar 

  3. Burianova, H., McIntosh, A. R., & Grady, C. L. (2010). A common functional brain network for autobiographical, episodic, and semantic memory retrieval. NeuroImage, 49(1), 865–874. doi:10.1016/j.neuroimage.2009.08.066.

    Article  PubMed  Google Scholar 

  4. Burns, C. G., & Fairclough, S. H. (2015). Use of auditory event-related potentials to measure immersion during a computer game. Journal of Human Computer Studies, 73(C), 107–114. doi:10.1016/j.ijhcs.2014.09.002.

    Article  Google Scholar 

  5. Cabeza, R., Prince, S. E., Daselaar, S. M., Greenberg, D. L., Budde, M., Dolcos, F., et al. (2004). Brain activity during episodic retrieval of autobiographical and laboratory events: an fMRI study using a novel photo paradigm. Journal of Cognitive Neuroscience, 16(9), 1583–1594. doi:10.1162/0898929042568578.

    Article  PubMed  Google Scholar 

  6. Chen, H.-Y., Gilmore, A. W., Nelson, S. M., & McDermott, K. B. (2017). Are there multiple kinds of episodic memory? An fMRI investigation comparing autobiographical and recognition memory tasks. Journal of Neuroscience, 37(10), 2764–2775. doi:10.1523/JNEUROSCI.1534-16.2017.

    Article  PubMed  Google Scholar 

  7. Conway, M. A. (2005). Memory and the self. Journal of Memory and Language, 53(4), 594–628. doi:10.1016/j.jml.2005.08.005.

    Article  Google Scholar 

  8. Coronel, J. C., & Federmeier, K. D. (2016). The N400 reveals how personal semantics is processed: insights into the nature and organization of self-knowledge. Neuropsychologia, 84(C), 36–43. doi:10.1016/j.neuropsychologia.2016.01.029.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Ferguson, C. J. (2007). The good, the bad and the ugly: a meta-analytic review of positive and negative effects of violent video games. Psychiatric Quarterly, 78(4), 309–316.

    Article  PubMed  Google Scholar 

  10. Ferguson, C. J. (2015). Do angry birds make for angry children? A meta-analysis of video game influences on children’s and adolescents’ aggression, mental health, prosocial behavior, and academic performance. Perspectives on Psychological Science, 10(5), 646–666.

    Article  PubMed  Google Scholar 

  11. Grilli, M. D., & Verfaellie, M. (2014). Personal semantic memory: insights from neuropsychological research on amnesia. Neuropsychologia, 61, 56–64. doi:10.1016/j.neuropsychologia.2014.06.012.

    Article  PubMed  Google Scholar 

  12. Gross, J. J. (2002). Emotion regulation: affective, cognitive, and social consequences. Psychophysiology, 39(3), 281–291.

    Article  Google Scholar 

  13. Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348.

    Article  PubMed  Google Scholar 

  14. Kemp, S., Burt, C., & Malinen, S. (2009). Investigating the structure of autobiographical memory using reaction times. Memory, 17(5), 511–517. doi:10.1080/09658210902939330.

    Article  PubMed  Google Scholar 

  15. Kensinger, E. A. (2009). Remembering the details: effects of emotion. Emotion Review, 1(2), 99–113.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Kruschke, J. K. (2013). Bayesian estimation supersedes the t Test. Journal of Experimental Psychology: General, 142(2), 573–588. doi:10.1037/a0029177.

    Article  Google Scholar 

  17. Kruschke, J. K., & Liddell, T. M. (2017). Bayesian data analysis for newcomers. Psychonomic Bulletin & Review, 17(3), 1–23. doi:10.3758/s13423-017-1272-1.

    Article  Google Scholar 

  18. Kvavilashvili, L., & Ellis, J. (2004). Ecological validity and the real-life/laboratory controversy in memory research: a critical and historical review. History & Philosophy of Psychology, 6, 59–80.

    Google Scholar 

  19. Leshikar, E. D., & Duarte, A. (2012). Medial prefrontal cortex supports source memory accuracy for self-referenced items. Social Neuroscience, 7(2), 126–145.

    Article  PubMed  Google Scholar 

  20. Levine, B., Svoboda, E., Hay, J. F., Winocur, G., & Moscovitch, M. (2002). Aging and autobiographical memory: dissociating episodic from semantic retrieval. Psychology and Aging, 17(4), 677.

    Article  PubMed  Google Scholar 

  21. Ly, A., Verhagen, J., & Wagenmakers, E.-J. (2016). Harold Jeffreys’s default Bayes factor hypothesis tests: explanation, extension, and application in psychology. Journal of Mathematical Psychology, 72, 19–32. doi:10.1016/j.jmp.2015.06.004.

    Article  Google Scholar 

  22. Mania, K., & Chalmers, A. (2001). The effects of levels of immersion on memory and presence in virtual environments: a reality centered approach. Cyberpsychology & Behavior, 4(2), 247–264.

    Article  Google Scholar 

  23. Markey, P. M., Markey, C. N., & French, J. E. (2014). Violent video games and real-world violence: rhetoric versus data. Psychology of Popular Media Culture, 4(4), 277–295.

    Article  Google Scholar 

  24. McDermott, K. B., Szpunar, K. K., & Christ, S. E. (2009). Laboratory-based and autobiographical retrieval tasks differ substantially in their neural substrates. Neuropsychologia, 47(11), 2290–2298. doi:10.1016/j.neuropsychologia.2008.12.025.

    Article  PubMed  Google Scholar 

  25. Renoult, L., Davidson, P. S., Palombo, D. J., Moscovitch, M., & Levine, B. (2012). Personal semantics: at the crossroads of semantic and episodic memory. Trends in Cognitive Sciences, 16(11), 550–558. doi:10.1016/j.tics.2012.09.003.

    Article  Google Scholar 

  26. Renoult, L., Tanguay, A., Beaudry, M., Tavakoli, P., Rabipour, S., Campbell, K., et al. (2016). Personal semantics: is it distinct from episodic and semantic memory? An electrophysiological study of memory for autobiographical facts and repeated events in honor of Shlomo Bentin. Neuropsychologia, 83, 242–256. doi:10.1016/j.neuropsychologia.2015.08.013.

    Article  PubMed  Google Scholar 

  27. Riva, G., Baños, R. M., Botella, C., Mantovani, F., & Gaggioli, A. (2016). Transforming experience: the potential of augmented reality and virtual reality for enhancing personal and clinical change. Frontiers in Psychiatry, 7, 164. doi:10.3389/fpsyt.2016.00164.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Rizzo, A. A., Schultheis, M., Kerns, K. A., & Mateer, C. (2004). Analysis of assets for virtual reality applications in neuropsychology. Neuropsychological Rehabilitation, 14(1-2), 207–239. doi:10.1080/09602010343000183.

    Article  Google Scholar 

  29. Roediger, H. L., & Marsh, E. J. (2003). Episodic and autobiographical memory. In A. F. Healy & R. W. Proctor (Eds.), The handbook of psychology (pp. 475–497). New York: Wiley.

    Google Scholar 

  30. Roediger, H. L., & McDermott, K. B. (2013). Two types of event memory. Proceedings of the National Academy of Sciences, 110(52), 20856–20857

  31. Sherry, J. L. (2001). The effects of violent video games on aggression. Human Communication Research, 27(3), 409–431.

    Google Scholar 

  32. Svoboda, E., McKinnon, M. C., & Levine, B. (2006). The functional neuroanatomy of autobiographical memory: a meta-analysis. Neuropsychologia, 44(12), 2189–2208. doi:10.1016/j.neuropsychologia.2006.05.023.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Tulving, E. (1972). Episodic and semantic memory 1. Organization of memory. London: Academic.

    Google Scholar 

  34. Unsworth, G., Devilly, G. J., & Ward, T. (2007). The effect of playing violent video games on adolescents: should parents be quaking in their boots? Psychology, Crime & Law, 13(4), 383–394.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the students who conducted this study: K. Beerheide, M. Erdogan, A. Frank, F. Groß, J. Hebisch, M. Heinzerling, H. Hermnes, A. Höger, A. Springer, R. Sylvester, I. Thomßen, P. von Felde, E. Wilgenbus.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Benjamin Schöne.

Ethics declarations

The study was conducted in accordance with the Declaration of Helsinki.

Conflict of Interest

The authors declare no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schöne, B., Wessels, M. & Gruber, T. Experiences in Virtual Reality: a Window to Autobiographical Memory. Curr Psychol 38, 715–719 (2019). https://doi.org/10.1007/s12144-017-9648-y

Download citation

Keywords

  • Virtual reality
  • Autobiographical memory
  • Episodic memory