Skip to main content
Log in

A virtual reality paradigm with dynamic scene stimuli for use in memory research

  • Paper Presented at SCiP
  • Published:
Behavior Research Methods Aims and scope Submit manuscript

Abstract

Episodic memory may essentially be memory for one’s place within a temporally unfolding scene from a first-person perspective. Given this, pervasively used static stimuli may only capture one small part of episodic memory. A promising approach for advancing the study of episodic memory is immersing participants within varying scenes from a first-person perspective. We present a pool of distinct scene stimuli for use in virtual environments and a paradigm that is implementable across varying levels of immersion on multiple virtual reality (VR) platforms and adaptable to studying various aspects of scene and episodic memory. In our task, participants are placed within a series of virtual environments from a first-person perspective and guided through a virtual tour of scenes during a study phase and a test phase. In the test phase, some scenes share a spatial layout with studied scenes; others are completely novel. In three experiments with varying degrees of immersion, we measure scene recall, scene familiarity-detection during recall failure, the subjective experience of déjà vu, the ability to predict the next turn on a tour, the subjective sense of being able to predict the next turn on a tour, and the factors that influence memory search and the inclination to generate candidate recollective information. The level of first-person immersion mattered to multiple facets of episodic memory. The paradigm presents a useful means of advancing mechanistic understanding of how memory operates in realistic dynamic scene environments, including in combination with cognitive neuroscience methods such as functional magnetic resonance imaging and electrophysiology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4 
Fig. 5 
Fig. 6
Fig. 7 
Fig. 8 
Fig. 9
Fig. 10
Fig. 11 

Similar content being viewed by others

Data availability

The data, stimuli, and experiment files, including the stimuli and experiment files for unity, are available on the Open Science Framework: https://osf.io/s68rc/.

Notes

  1. Note that first-person immersion within a scene does not necessarily have to be visual; even without sight, people likely have an ability to process first-person immersion within a place based on non-visual information such as sounds and tactile information. An example includes the role of scenes in déjà vu as experienced by a person with congenital blindness; in this case, the “scene” was auditory and tactile and temporally unfolding (O’Connor & Moulin, 2006).

  2. The building in which the in-person 2D Experiments 1b and 2b were carried out experienced network shortages during the transition from remote work to in-person work at the time that these experiments were carried out. As Experiments 1b and 2b were carried out using Qualtrics over the Internet from the research laboratory, these periodic network disruptions are the likely reason behind why some participants did not finish the experiment.

  3. Data were analyzed using null hypothesis significance testing (NHST) and Bayesian methods of analysis. From the NHST, we report both p values and standard effect sizes, specifically Cohen’s d for repeated-measures and partial eta squared (\({n}_{p}^{2}\); see Lakens, 2013). We also report Bayes factors (BFs) that were computed using JASP and the JZS prior. The strength of evidence provided by the BFs was categorized using the recommendations of Wagenmakers et al. (2007), such that a BF was considered to provide anecdotal evidence (BF = 1–3), substantial evidence (BF = 3–10), strong evidence (BF = 10–30), very strong evidence (BF = 30–100), or extreme evidence (BF > 100) in favor of one hypothesis, either the null or alternative, over the other. In cases where we argue for significant effect, we report BF10, which provides the strength of evidence in favor of the alternative hypothesis over the null. In cases where we argue for a null effect, we report BF01, which provides the strength of evidence in favor of the null hypothesis.

  4. The general phenomenon of sickness that can occur in VR settings can be referenced in the literature by a variety of terms besides cybersickness. Related terms include simulator sickness, VR sickness, or visually induced motion sickness (VIMS). There are also other post hoc questionnaire measures besides the SSQ that we use here, including the VRSQ (Kim et al., 2018), CSQ (Stone, 2017), and MSAQ (Gianaros et al., 2001). In addition, there are other approaches altogether, such as real-time probing of symptomology throughout a study (e.g., Bos et al., 2005, or Keshavarz & Hecht, 2011), or objective psychophysiological indices such as postural sway, heart rate, electrodermal activity, and others (see Chang et al., 2020, for a review).

References

  • Alsbury-Nealy, K., Wang, H., Howarth, C., Gordienko, A., Schlichting, M. L., & Duncan, K. D. (2022). OpenMaze: An open-source toolbox for creating virtual navigation experiments. Behavior Research Methods, 54, 1374–1387.

    Article  PubMed  Google Scholar 

  • Bellezza, F. S., & Reddy, B. G. (1978). Mnemonic devices and natural memory. Bulletin of the Psychonomic Society., 11, 277–280. https://doi.org/10.3758/BF03336829

    Article  Google Scholar 

  • Bos, J. E., MacKinnon, S. N., & Patterson, A. (2005). Motion sickness symptoms in a ship motion simulator: effects of inside, outside, and no view. Aviation, Space, and Environmental Medicine, 76(12), 1111–1118.

    PubMed  Google Scholar 

  • Bower, G. H. (1970). Analysis of mnemonic devices. American Scientist, 58, 496–510.

    Google Scholar 

  • Brookes, J., Warburton, M., Alghadier, M., Mon-Williams, M., & Mushtaq, F. (2020). Studying human behavior with virtual reality: The Unity Experiment Framework. Behavior Research Methods, 52(2), 455–463. https://doi.org/10.3758/s13428-019-01242-0

    Article  PubMed  Google Scholar 

  • Brown, A. S. (2004). The déjà vu experience. Psychology Press.

    Book  Google Scholar 

  • Carass, A., Roy, S., Gherman, A., Reinhold, J. C., Jesson, A., Arbel, T., Maier, O., Handels, H., Ghafoorian, M., Platel, B., Birenbaum, A., Greenspan, H., Pham, D. L., Crainiceanu, C. M., Calabresi, P. A., Prince, J. L., Gray Roncal, W. R., Shinohara, R. T., & Oguz, I. (2020). Evaluation white matter lesion segmentations with refined Sørensen-Dice Analysis. Nature, 10, 8242.

    Google Scholar 

  • Carlaw, B. N., Huebert, A. M., McNeely-White, K. L., et al. (2022). Detecting a familiar person behind the surgical mask: recognition without identification among masked versus sunglasses-covered faces. Cognitive Research: Principles and Implications, 7, 90. https://doi.org/10.1186/s41235-022-00440-3

    Article  PubMed  Google Scholar 

  • Chang, E., Kim, H. T., & Yoo, B. (2020). Virtual reality sickness: A review of causes and measurements. International Journal of Human-Computer Interaction, 36(17), 1658–1682.

    Article  Google Scholar 

  • Cleary, A.M. (2019). Dependent measures in memory research: From recall to recognition. In H. Otani & B.L. Schwartz (Eds.) Research Methods in Human Memory. Routledge; New York, NY.

  • Cleary, A. M. (2004). Orthography, phonology, and meaning: Word features that give rise to feelings of familiarity in recognition. Psychonomic Bulletin & Review, 11, 446–451.

    Article  Google Scholar 

  • Cleary, A. M., & Brown, A. S. (2022). The déjà vu experience (2nd ed.). Routledge.

    Google Scholar 

  • Cleary, A. M., Brown, A. S., Sawyer, B. D., Nomi, J. S., Ajoku, A. C., & Ryals, A. J. (2012). Familiarity from the configuration of objects in 3-dimensional space and its relation to déjà vu: A virtual reality investigation. Consciousness and Cognition, 21, 969–975.

    Article  PubMed  Google Scholar 

  • Cleary, A. M., & Claxton, A. B. (2018). Déjà vu: An illusion of prediction. Psychological Science, 29, 635–644.

    Article  PubMed  Google Scholar 

  • Cleary, A. M., Huebert, A. M., McNeely-White, K. L., & Spahr, K. S. (2019). A postdictive bias associated with déjà vu. Psychonomic Bulletin & Review, 26, 1433–1439. https://doi.org/10.3758/s13423-019-01578-w

    Article  Google Scholar 

  • Cleary, A. M., Irving, Z. C., & Mills, C. (2023). What flips attention? Cognitive Science, 47(4), e13274. https://doi.org/10.1111/cogs.13274

    Article  PubMed  Google Scholar 

  • Cleary, A. M., McNeely-White, K. L., Huebert, A. M., & Claxton, A. B. (2021a). Déjà vu and the feeling of prediction: An association with familiarity strength. Memory, 29(7), 904–920. https://doi.org/10.1080/09658211.2018.1503686

  • Cleary, A. M., Neisser, J., McMahan, T., Parsons, T. D., Alwaki, A., Okada, N., Vosoughi, A., Kheder, A., Drane, D. L., & Pedersen, N. P. (2021b). Subjective distinguishability of seizure and non-seizure déjà vu: A case report, brief literature review, and research prospects. Epilepsy & Behavior, 125, 108373.

  • Cleary, A. M., Ryals, A. J., & Nomi, J. S. (2009). Can déjà vu result from similarity to a prior experience? Support for the similarity hypothesis of déjà vu. Psychonomic Bulletin & Review, 16, 1082–1088.

    Article  Google Scholar 

  • Commins, S., Duffin, J., Chaves, K., Leahy, D., Corcoran, K., Caffrey, M., Keenan, L., Finan, D., & Thornberry, C. (2020). NavWell: A simplified virtual-reality platform for spatial navigation and memory experiments. Behavior Research Methods, 52, 1189–1207.

    Article  PubMed  Google Scholar 

  • Coughlan, G., Laczó, J., Hort, J., Minihane, A.-M., & Hornberger, M. (2018). Spatial navigation deficits - overlooked cognitive marker for preclinical Alzheimer disease? Nature Reviews Neurology, 14, 496–506.

    Article  PubMed  Google Scholar 

  • Dice, L. R. (1945). Measures of the amount of ecological association between species. Ecology, 26, 297–302.

    Article  Google Scholar 

  • Drane, D. L., Parsons, T.D., McMahan, T., Cleary, A. M., Okada, N., Pedersen, N. P., ... & Neisser, J. (2022). Disruptive Technology Presents: Reimagining Memory Assessment Through Theory, Virtual Reality, and Other Novel Technologies. The Clinical Neuropsychologist, 36, 740–741.

  • Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.

    Article  PubMed  Google Scholar 

  • Gianaros, P. J., Muth, E. R., Mordkoff, J. T., Levine, M. E., & Stern, R. M. (2001). A questionnaire for the assessment of the multiple dimensions of motion sickness. Aviation, Space, and Environmental Medicine, 72(2), 115.

    PubMed  PubMed Central  Google Scholar 

  • Huebert, A. M., McNeely-White, K. L., & Cleary, A. M. (2022a). On the relationship between tip-of-the-tongue states and partial recollective experience: Illusory partial recollective access during tip-of-the-tongue states. Journal of Experimental Psychology. General, https://doi.org/10.1037/xge0001292. Advance online publication. https://doi.org/10.1037/xge0001292

  • Huebert, A. M., McNeely-White, K. L., & Cleary, A. M. (2022). Can cue familiarity during recall failure prompt illusory recollective experience? Memory & Cognition, 50, 681–695.

    Article  Google Scholar 

  • Jebara, N., Orriols, E., Zaoui, M., Berthoz, A., & Piolino, P. (2014). Effects of enactment in episodic memory: A pilot virtual reality study with young and elderly adults. Frontiers in Aging Neuroscience, 6, 338. https://doi.org/10.3389/fnagi.2014.00338

    Article  PubMed  PubMed Central  Google Scholar 

  • Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203–220.

    Article  Google Scholar 

  • Keshavarz, B., & Hecht, H. (2011). Validating an efficient method to quantify motion sickness. Human Factors, 53(4), 415–426.

    Article  PubMed  Google Scholar 

  • Khronos Group. (2019). OpenXR. Retrieved from https://www.khronos.org/openxr/

  • Kim, H. K., Park, J., Choi, Y., & Choe, M. (2018). Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Applied Ergonomics, 69, 66–73.

    Article  PubMed  Google Scholar 

  • Kober, S. E., & Neuper, C. (2012). Using auditory event-related EEG potentials to assess presence in virtual reality. International Journal of Human-Computer Studies, 70(9), 577–587.

    Article  Google Scholar 

  • Kunz, L., Brandt, A., Reinacher, P. C., Staresina, B. P., Reifenstein, E. T., Weidemann, C. T., Herweg, N. A., Patel, A., Tsitsiklis, M., Kempter, R., Kahana, M. J., Schulze-Bonhage, A., & Jacobs, J. (2021). A neural code for egocentric spatial maps in the human medial temporal lobe. Neuron, 109(17), 2781-2796.e10. https://doi.org/10.1016/j.neuron.2021.06.019

    Article  PubMed  PubMed Central  Google Scholar 

  • La Corte, V., Sperduti, M., Abichou, K., & Piolino, P. (2019). Episodic memory assessment and remediation in normal and pathological aging using virtual reality: A mini review. Frontiers in Psychology, 10, 173.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lakenss, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.

    Google Scholar 

  • Maguire, E. A., & Mullally, S. L. (2013). The hippocampus: A manifesto for change. Journal of Experimental Psychology: General, 142, 1180–1189.

    Article  PubMed  Google Scholar 

  • McNeely-White, K. L., & Cleary, A. M. (2023). Piquing curiosity: Déjà vu-like states are associated with feelings of curiosity and information-seeking behaviors. Journal of Intelligence, 11, 112. https://doi.org/10.3390/jintelligence11060112

    Article  PubMed  PubMed Central  Google Scholar 

  • Meade, M. E., Meade, J. G., Sauzeon, H., & Fernandes, M. A. (2019). Active navigation in virtual environments benefits spatial memory in older adults. Brain Sciences, 9(3), 47. https://doi.org/10.3390/brainsci9030047

    Article  PubMed  PubMed Central  Google Scholar 

  • Neisser, J., Abreu, G., Drane, D. L., Pedersen, N. P., Parsons, T. D., & Cleary, A. M. (2023). Opening a conceptual space for metamemory experience. New Ideas in Psychology, 69, 100995. https://doi.org/10.1016/j.newideapsych.2022.100995

    Article  Google Scholar 

  • Neisser, U. (1987). A sense of where you are: Functions of the spatial module. In: Ellen, P., Thinus-Blanc, C. (Eds). Cognitive Processes and Spatial Orientation in Animal and Man. NATO ASI Series, vol 37. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-3533-4_26

  • O’Connor, A. R., & Moulin, C. J. (2006). Normal patterns of déjà experience in a healthy, blind male: Challenging optical pathway delay theory. Brain and Cognition, 62(3), 246–249. https://doi.org/10.1016/j.bandc.2006.06.004

    Article  PubMed  Google Scholar 

  • Parsons, T. D., McMahan, T., & Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. The Clinical Neuropsychologist, 32(1), 16–41.

    Article  PubMed  Google Scholar 

  • Parsons, T. D., Gaggioli, A., & Riva, G. (2020). Extended reality for the clinical, affective, and social neurosciences. Brain Sciences, 10(12), 922.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pasqualotto, A., Spiller, M. J., Jansari, A. S., & Proulx, M. J. (2013). Visual experience facilitates allocentric spatial representation. Behavioural Brain Research, 236(1), 175–179. https://doi.org/10.1016/j.bbr.2012.08.042

    Article  PubMed  Google Scholar 

  • Peeters, D. (2018). A standardized set of 3-D objects for virtual reality research and applications. Behavior Research Methods, 50(3), 1047–1054. https://doi.org/10.3758/s13428-017-0925-3

    Article  PubMed  Google Scholar 

  • Plancher, G., Barra, J., Orriols, E., & Piolino, P. (2013). The influence of action on episodic memory: A virtual reality study. Quarterly Journal of Experimental Psychology, 66(5), 895–909. https://doi.org/10.1080/17470218.2012.722657

    Article  Google Scholar 

  • Rebenitsch, L., & Owen, C. (2016). Review on cybersickness in applications and visual displays. Virtual Reality, 20(2), 101–125.

    Article  Google Scholar 

  • Rodríguez, M. F., Ramirez Butavand, D. R., Cifuentes, M. V., Bekinschtein, P., Ballarani, F., & García Bauza, C. G. (2022). A virtual reality platform for memory evaluation: Assessing effects of spatial strategies. Behavior Research Methods, 54, 2707–2719.

    Article  PubMed  Google Scholar 

  • Rubin, D. C. (2022). A conceptual space for episodic and semantic memory. Memory & Cognition, 50, 464–477.

    Article  Google Scholar 

  • Rubin, D. C., Deffler, S. A., & Umanath, S. (2019). Scenes enable a sense of reliving: Implications for autobiographical memory. Cognition, 183, 44–56.

    Article  PubMed  Google Scholar 

  • Rubin, D. C., & Umanath, S. (2015). Event memory: A theory of memory for laboratory, autobiographical, and fictional events. Psychological Review, 122(1), 1–23.

    Article  PubMed  Google Scholar 

  • Ryals, A. J., & Cleary, A. M. (2012). The recognition without cued recall phenomenon: Support for a feature-matching theory over a partial recollection account. Journal of Memory and Language, 66, 747–762.

    Article  Google Scholar 

  • Schwartz, B.L. & Cleary, A.M. (2016). Tip-of-the-tongue states, déjà vu experiences, and other odd metacognitive experiences. In J. Dunlosky & S. K. Tauber (Eds.), The Oxford Handbook of Metamemory. Oxford University Press. (pp. 95–108).

  • Slater, M. (2003). A note on presence terminology. Presence Connect, 3(3), 1–5.

    Google Scholar 

  • Smith, S. A. (2019). Virtual reality in episodic memory research: A review. Psychonomic Bulletin & Review, 26(4), 1213–1237. https://doi.org/10.3758/s13423-019-01605-w

    Article  Google Scholar 

  • Smith, S. A., & Mulligan, N. W. (2021). Immersion, presence, and episodic memory in virtual reality environments. Memory, 29(8), 983–1005. https://doi.org/10.1080/09658211.2021.1953535

    Article  PubMed  Google Scholar 

  • Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342–353. https://doi.org/10.3758/BF03197465

    Article  Google Scholar 

  • Sno, H. N., Schalken, H. F., de Jonghe, F., & Koeter, M. W. (1994). The inventory for déjà vu experiences assessment: Development, utility, reliability, and validity. Journal of Nervous and Mental Disease, 182, 27–33.

    Article  PubMed  Google Scholar 

  • Sørensen, T. (1948). A method of establishing groups of equal amplitudes in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons. Kongelige Danske Videnskabernes Selskab, Biologiske Skrifter, 5, 1–34.

    Google Scholar 

  • Starrett, M. J., McAvan, A. S., Huffman, D. J., Stokes, J. D., Kyle, C. T., Smuda, D. N., Kolarik, B. S., Laczko, J., & Ekstrom, A. D. (2021). Landmarks: A solution for spatial navigation and memory experiments in virtual reality. Behavior Research Methods, 53, 1046–1059.

    Article  PubMed  Google Scholar 

  • Stone III, W. B. (2017). Psychometric evaluation of the Simulator Sickness Questionnaire as a measure of cybersickness (Doctoral dissertation, Iowa State University).

  • Tulving, E. (1989). Memory: Performance, knowledge, and experience. European Journal of Cognitive Psychology, 1, 3–26.

    Article  Google Scholar 

  • Unity Technologies. (2018). Unity. Retrieved from https://unity3d.com/

  • Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2007). A practical solution to the pervasive problem of p values. Psychonomic Bulletin & Review, 14, 779–804.

    Article  Google Scholar 

  • Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and cybersickness in virtual reality are negatively related: A review. Frontiers in Psychology, 10, 158.

    Article  PubMed  PubMed Central  Google Scholar 

  • Young, S. D., Adelstein, B. D., & Ellis, S. R. (2007). Demand characteristics in assessing motion sickness in a virtual environment: Or does taking a motion sickness questionnaire make you sick? IEEE Transactions on Visualization and Computer Graphics, 13(3), 422–428. https://doi.org/10.1109/TVCG.2007.1029

    Article  PubMed  Google Scholar 

Download references

Funding

NPP was supported by National Institute of Neurological Disorders and Stroke, Grant/Award Number: K08 NS105929, R01 NS088748, R21 NS122011, and a CURE Epilepsy Award. DLD’s efforts on this paper were supported in part by funding received from the National Institutes of Health/National Institute of Neurological Disorders and Stroke (NIH/NINDS, R01NS088748). He also receives funding from additional grants from the NIH (R01MH118514; R01NS110347), as well as Medtronic, Inc. (A1321808).

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Noah S. Okada or Nigel P. Pedersen.

Ethics declarations

Ethics Approval

All procedures performed in the current experiments were in accordance with the ethical standards of the institutional research committee of Colorado State University under protocols #20-9755H (Human Memory Processes) and #2722 (Cognitive Processes in Virtual Reality Settings).

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

Dr. Pedersen is on the Scientific Advisory Board of Dixi Medical, USA, a manufacturer of equipment for epilepsy surgery evaluations, but this is not related to the present work. Dr. Drane receives support from Medtronic, USA, but this is not related to the present work. The other authors declare no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Okada, N.S., McNeely-White, K.L., Cleary, A.M. et al. A virtual reality paradigm with dynamic scene stimuli for use in memory research. Behav Res (2023). https://doi.org/10.3758/s13428-023-02243-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.3758/s13428-023-02243-w

Keywords

Navigation