Abstract
Episodic memory may essentially be memory for one’s place within a temporally unfolding scene from a first-person perspective. Given this, pervasively used static stimuli may only capture one small part of episodic memory. A promising approach for advancing the study of episodic memory is immersing participants within varying scenes from a first-person perspective. We present a pool of distinct scene stimuli for use in virtual environments and a paradigm that is implementable across varying levels of immersion on multiple virtual reality (VR) platforms and adaptable to studying various aspects of scene and episodic memory. In our task, participants are placed within a series of virtual environments from a first-person perspective and guided through a virtual tour of scenes during a study phase and a test phase. In the test phase, some scenes share a spatial layout with studied scenes; others are completely novel. In three experiments with varying degrees of immersion, we measure scene recall, scene familiarity-detection during recall failure, the subjective experience of déjà vu, the ability to predict the next turn on a tour, the subjective sense of being able to predict the next turn on a tour, and the factors that influence memory search and the inclination to generate candidate recollective information. The level of first-person immersion mattered to multiple facets of episodic memory. The paradigm presents a useful means of advancing mechanistic understanding of how memory operates in realistic dynamic scene environments, including in combination with cognitive neuroscience methods such as functional magnetic resonance imaging and electrophysiology.
Similar content being viewed by others
Data availability
The data, stimuli, and experiment files, including the stimuli and experiment files for unity, are available on the Open Science Framework: https://osf.io/s68rc/.
Notes
Note that first-person immersion within a scene does not necessarily have to be visual; even without sight, people likely have an ability to process first-person immersion within a place based on non-visual information such as sounds and tactile information. An example includes the role of scenes in déjà vu as experienced by a person with congenital blindness; in this case, the “scene” was auditory and tactile and temporally unfolding (O’Connor & Moulin, 2006).
The building in which the in-person 2D Experiments 1b and 2b were carried out experienced network shortages during the transition from remote work to in-person work at the time that these experiments were carried out. As Experiments 1b and 2b were carried out using Qualtrics over the Internet from the research laboratory, these periodic network disruptions are the likely reason behind why some participants did not finish the experiment.
Data were analyzed using null hypothesis significance testing (NHST) and Bayesian methods of analysis. From the NHST, we report both p values and standard effect sizes, specifically Cohen’s d for repeated-measures and partial eta squared (\({n}_{p}^{2}\); see Lakens, 2013). We also report Bayes factors (BFs) that were computed using JASP and the JZS prior. The strength of evidence provided by the BFs was categorized using the recommendations of Wagenmakers et al. (2007), such that a BF was considered to provide anecdotal evidence (BF = 1–3), substantial evidence (BF = 3–10), strong evidence (BF = 10–30), very strong evidence (BF = 30–100), or extreme evidence (BF > 100) in favor of one hypothesis, either the null or alternative, over the other. In cases where we argue for significant effect, we report BF10, which provides the strength of evidence in favor of the alternative hypothesis over the null. In cases where we argue for a null effect, we report BF01, which provides the strength of evidence in favor of the null hypothesis.
The general phenomenon of sickness that can occur in VR settings can be referenced in the literature by a variety of terms besides cybersickness. Related terms include simulator sickness, VR sickness, or visually induced motion sickness (VIMS). There are also other post hoc questionnaire measures besides the SSQ that we use here, including the VRSQ (Kim et al., 2018), CSQ (Stone, 2017), and MSAQ (Gianaros et al., 2001). In addition, there are other approaches altogether, such as real-time probing of symptomology throughout a study (e.g., Bos et al., 2005, or Keshavarz & Hecht, 2011), or objective psychophysiological indices such as postural sway, heart rate, electrodermal activity, and others (see Chang et al., 2020, for a review).
References
Alsbury-Nealy, K., Wang, H., Howarth, C., Gordienko, A., Schlichting, M. L., & Duncan, K. D. (2022). OpenMaze: An open-source toolbox for creating virtual navigation experiments. Behavior Research Methods, 54, 1374–1387.
Bellezza, F. S., & Reddy, B. G. (1978). Mnemonic devices and natural memory. Bulletin of the Psychonomic Society., 11, 277–280. https://doi.org/10.3758/BF03336829
Bos, J. E., MacKinnon, S. N., & Patterson, A. (2005). Motion sickness symptoms in a ship motion simulator: effects of inside, outside, and no view. Aviation, Space, and Environmental Medicine, 76(12), 1111–1118.
Bower, G. H. (1970). Analysis of mnemonic devices. American Scientist, 58, 496–510.
Brookes, J., Warburton, M., Alghadier, M., Mon-Williams, M., & Mushtaq, F. (2020). Studying human behavior with virtual reality: The Unity Experiment Framework. Behavior Research Methods, 52(2), 455–463. https://doi.org/10.3758/s13428-019-01242-0
Brown, A. S. (2004). The déjà vu experience. Psychology Press.
Carass, A., Roy, S., Gherman, A., Reinhold, J. C., Jesson, A., Arbel, T., Maier, O., Handels, H., Ghafoorian, M., Platel, B., Birenbaum, A., Greenspan, H., Pham, D. L., Crainiceanu, C. M., Calabresi, P. A., Prince, J. L., Gray Roncal, W. R., Shinohara, R. T., & Oguz, I. (2020). Evaluation white matter lesion segmentations with refined Sørensen-Dice Analysis. Nature, 10, 8242.
Carlaw, B. N., Huebert, A. M., McNeely-White, K. L., et al. (2022). Detecting a familiar person behind the surgical mask: recognition without identification among masked versus sunglasses-covered faces. Cognitive Research: Principles and Implications, 7, 90. https://doi.org/10.1186/s41235-022-00440-3
Chang, E., Kim, H. T., & Yoo, B. (2020). Virtual reality sickness: A review of causes and measurements. International Journal of Human-Computer Interaction, 36(17), 1658–1682.
Cleary, A.M. (2019). Dependent measures in memory research: From recall to recognition. In H. Otani & B.L. Schwartz (Eds.) Research Methods in Human Memory. Routledge; New York, NY.
Cleary, A. M. (2004). Orthography, phonology, and meaning: Word features that give rise to feelings of familiarity in recognition. Psychonomic Bulletin & Review, 11, 446–451.
Cleary, A. M., & Brown, A. S. (2022). The déjà vu experience (2nd ed.). Routledge.
Cleary, A. M., Brown, A. S., Sawyer, B. D., Nomi, J. S., Ajoku, A. C., & Ryals, A. J. (2012). Familiarity from the configuration of objects in 3-dimensional space and its relation to déjà vu: A virtual reality investigation. Consciousness and Cognition, 21, 969–975.
Cleary, A. M., & Claxton, A. B. (2018). Déjà vu: An illusion of prediction. Psychological Science, 29, 635–644.
Cleary, A. M., Huebert, A. M., McNeely-White, K. L., & Spahr, K. S. (2019). A postdictive bias associated with déjà vu. Psychonomic Bulletin & Review, 26, 1433–1439. https://doi.org/10.3758/s13423-019-01578-w
Cleary, A. M., Irving, Z. C., & Mills, C. (2023). What flips attention? Cognitive Science, 47(4), e13274. https://doi.org/10.1111/cogs.13274
Cleary, A. M., McNeely-White, K. L., Huebert, A. M., & Claxton, A. B. (2021a). Déjà vu and the feeling of prediction: An association with familiarity strength. Memory, 29(7), 904–920. https://doi.org/10.1080/09658211.2018.1503686
Cleary, A. M., Neisser, J., McMahan, T., Parsons, T. D., Alwaki, A., Okada, N., Vosoughi, A., Kheder, A., Drane, D. L., & Pedersen, N. P. (2021b). Subjective distinguishability of seizure and non-seizure déjà vu: A case report, brief literature review, and research prospects. Epilepsy & Behavior, 125, 108373.
Cleary, A. M., Ryals, A. J., & Nomi, J. S. (2009). Can déjà vu result from similarity to a prior experience? Support for the similarity hypothesis of déjà vu. Psychonomic Bulletin & Review, 16, 1082–1088.
Commins, S., Duffin, J., Chaves, K., Leahy, D., Corcoran, K., Caffrey, M., Keenan, L., Finan, D., & Thornberry, C. (2020). NavWell: A simplified virtual-reality platform for spatial navigation and memory experiments. Behavior Research Methods, 52, 1189–1207.
Coughlan, G., Laczó, J., Hort, J., Minihane, A.-M., & Hornberger, M. (2018). Spatial navigation deficits - overlooked cognitive marker for preclinical Alzheimer disease? Nature Reviews Neurology, 14, 496–506.
Dice, L. R. (1945). Measures of the amount of ecological association between species. Ecology, 26, 297–302.
Drane, D. L., Parsons, T.D., McMahan, T., Cleary, A. M., Okada, N., Pedersen, N. P., ... & Neisser, J. (2022). Disruptive Technology Presents: Reimagining Memory Assessment Through Theory, Virtual Reality, and Other Novel Technologies. The Clinical Neuropsychologist, 36, 740–741.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
Gianaros, P. J., Muth, E. R., Mordkoff, J. T., Levine, M. E., & Stern, R. M. (2001). A questionnaire for the assessment of the multiple dimensions of motion sickness. Aviation, Space, and Environmental Medicine, 72(2), 115.
Huebert, A. M., McNeely-White, K. L., & Cleary, A. M. (2022a). On the relationship between tip-of-the-tongue states and partial recollective experience: Illusory partial recollective access during tip-of-the-tongue states. Journal of Experimental Psychology. General, https://doi.org/10.1037/xge0001292. Advance online publication. https://doi.org/10.1037/xge0001292
Huebert, A. M., McNeely-White, K. L., & Cleary, A. M. (2022). Can cue familiarity during recall failure prompt illusory recollective experience? Memory & Cognition, 50, 681–695.
Jebara, N., Orriols, E., Zaoui, M., Berthoz, A., & Piolino, P. (2014). Effects of enactment in episodic memory: A pilot virtual reality study with young and elderly adults. Frontiers in Aging Neuroscience, 6, 338. https://doi.org/10.3389/fnagi.2014.00338
Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203–220.
Keshavarz, B., & Hecht, H. (2011). Validating an efficient method to quantify motion sickness. Human Factors, 53(4), 415–426.
Khronos Group. (2019). OpenXR. Retrieved from https://www.khronos.org/openxr/
Kim, H. K., Park, J., Choi, Y., & Choe, M. (2018). Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Applied Ergonomics, 69, 66–73.
Kober, S. E., & Neuper, C. (2012). Using auditory event-related EEG potentials to assess presence in virtual reality. International Journal of Human-Computer Studies, 70(9), 577–587.
Kunz, L., Brandt, A., Reinacher, P. C., Staresina, B. P., Reifenstein, E. T., Weidemann, C. T., Herweg, N. A., Patel, A., Tsitsiklis, M., Kempter, R., Kahana, M. J., Schulze-Bonhage, A., & Jacobs, J. (2021). A neural code for egocentric spatial maps in the human medial temporal lobe. Neuron, 109(17), 2781-2796.e10. https://doi.org/10.1016/j.neuron.2021.06.019
La Corte, V., Sperduti, M., Abichou, K., & Piolino, P. (2019). Episodic memory assessment and remediation in normal and pathological aging using virtual reality: A mini review. Frontiers in Psychology, 10, 173.
Lakenss, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.
Maguire, E. A., & Mullally, S. L. (2013). The hippocampus: A manifesto for change. Journal of Experimental Psychology: General, 142, 1180–1189.
McNeely-White, K. L., & Cleary, A. M. (2023). Piquing curiosity: Déjà vu-like states are associated with feelings of curiosity and information-seeking behaviors. Journal of Intelligence, 11, 112. https://doi.org/10.3390/jintelligence11060112
Meade, M. E., Meade, J. G., Sauzeon, H., & Fernandes, M. A. (2019). Active navigation in virtual environments benefits spatial memory in older adults. Brain Sciences, 9(3), 47. https://doi.org/10.3390/brainsci9030047
Neisser, J., Abreu, G., Drane, D. L., Pedersen, N. P., Parsons, T. D., & Cleary, A. M. (2023). Opening a conceptual space for metamemory experience. New Ideas in Psychology, 69, 100995. https://doi.org/10.1016/j.newideapsych.2022.100995
Neisser, U. (1987). A sense of where you are: Functions of the spatial module. In: Ellen, P., Thinus-Blanc, C. (Eds). Cognitive Processes and Spatial Orientation in Animal and Man. NATO ASI Series, vol 37. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-3533-4_26
O’Connor, A. R., & Moulin, C. J. (2006). Normal patterns of déjà experience in a healthy, blind male: Challenging optical pathway delay theory. Brain and Cognition, 62(3), 246–249. https://doi.org/10.1016/j.bandc.2006.06.004
Parsons, T. D., McMahan, T., & Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. The Clinical Neuropsychologist, 32(1), 16–41.
Parsons, T. D., Gaggioli, A., & Riva, G. (2020). Extended reality for the clinical, affective, and social neurosciences. Brain Sciences, 10(12), 922.
Pasqualotto, A., Spiller, M. J., Jansari, A. S., & Proulx, M. J. (2013). Visual experience facilitates allocentric spatial representation. Behavioural Brain Research, 236(1), 175–179. https://doi.org/10.1016/j.bbr.2012.08.042
Peeters, D. (2018). A standardized set of 3-D objects for virtual reality research and applications. Behavior Research Methods, 50(3), 1047–1054. https://doi.org/10.3758/s13428-017-0925-3
Plancher, G., Barra, J., Orriols, E., & Piolino, P. (2013). The influence of action on episodic memory: A virtual reality study. Quarterly Journal of Experimental Psychology, 66(5), 895–909. https://doi.org/10.1080/17470218.2012.722657
Rebenitsch, L., & Owen, C. (2016). Review on cybersickness in applications and visual displays. Virtual Reality, 20(2), 101–125.
Rodríguez, M. F., Ramirez Butavand, D. R., Cifuentes, M. V., Bekinschtein, P., Ballarani, F., & García Bauza, C. G. (2022). A virtual reality platform for memory evaluation: Assessing effects of spatial strategies. Behavior Research Methods, 54, 2707–2719.
Rubin, D. C. (2022). A conceptual space for episodic and semantic memory. Memory & Cognition, 50, 464–477.
Rubin, D. C., Deffler, S. A., & Umanath, S. (2019). Scenes enable a sense of reliving: Implications for autobiographical memory. Cognition, 183, 44–56.
Rubin, D. C., & Umanath, S. (2015). Event memory: A theory of memory for laboratory, autobiographical, and fictional events. Psychological Review, 122(1), 1–23.
Ryals, A. J., & Cleary, A. M. (2012). The recognition without cued recall phenomenon: Support for a feature-matching theory over a partial recollection account. Journal of Memory and Language, 66, 747–762.
Schwartz, B.L. & Cleary, A.M. (2016). Tip-of-the-tongue states, déjà vu experiences, and other odd metacognitive experiences. In J. Dunlosky & S. K. Tauber (Eds.), The Oxford Handbook of Metamemory. Oxford University Press. (pp. 95–108).
Slater, M. (2003). A note on presence terminology. Presence Connect, 3(3), 1–5.
Smith, S. A. (2019). Virtual reality in episodic memory research: A review. Psychonomic Bulletin & Review, 26(4), 1213–1237. https://doi.org/10.3758/s13423-019-01605-w
Smith, S. A., & Mulligan, N. W. (2021). Immersion, presence, and episodic memory in virtual reality environments. Memory, 29(8), 983–1005. https://doi.org/10.1080/09658211.2021.1953535
Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342–353. https://doi.org/10.3758/BF03197465
Sno, H. N., Schalken, H. F., de Jonghe, F., & Koeter, M. W. (1994). The inventory for déjà vu experiences assessment: Development, utility, reliability, and validity. Journal of Nervous and Mental Disease, 182, 27–33.
Sørensen, T. (1948). A method of establishing groups of equal amplitudes in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons. Kongelige Danske Videnskabernes Selskab, Biologiske Skrifter, 5, 1–34.
Starrett, M. J., McAvan, A. S., Huffman, D. J., Stokes, J. D., Kyle, C. T., Smuda, D. N., Kolarik, B. S., Laczko, J., & Ekstrom, A. D. (2021). Landmarks: A solution for spatial navigation and memory experiments in virtual reality. Behavior Research Methods, 53, 1046–1059.
Stone III, W. B. (2017). Psychometric evaluation of the Simulator Sickness Questionnaire as a measure of cybersickness (Doctoral dissertation, Iowa State University).
Tulving, E. (1989). Memory: Performance, knowledge, and experience. European Journal of Cognitive Psychology, 1, 3–26.
Unity Technologies. (2018). Unity. Retrieved from https://unity3d.com/
Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2007). A practical solution to the pervasive problem of p values. Psychonomic Bulletin & Review, 14, 779–804.
Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and cybersickness in virtual reality are negatively related: A review. Frontiers in Psychology, 10, 158.
Young, S. D., Adelstein, B. D., & Ellis, S. R. (2007). Demand characteristics in assessing motion sickness in a virtual environment: Or does taking a motion sickness questionnaire make you sick? IEEE Transactions on Visualization and Computer Graphics, 13(3), 422–428. https://doi.org/10.1109/TVCG.2007.1029
Funding
NPP was supported by National Institute of Neurological Disorders and Stroke, Grant/Award Number: K08 NS105929, R01 NS088748, R21 NS122011, and a CURE Epilepsy Award. DLD’s efforts on this paper were supported in part by funding received from the National Institutes of Health/National Institute of Neurological Disorders and Stroke (NIH/NINDS, R01NS088748). He also receives funding from additional grants from the NIH (R01MH118514; R01NS110347), as well as Medtronic, Inc. (A1321808).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Ethics Approval
All procedures performed in the current experiments were in accordance with the ethical standards of the institutional research committee of Colorado State University under protocols #20-9755H (Human Memory Processes) and #2722 (Cognitive Processes in Virtual Reality Settings).
Consent to Participate
Informed consent was obtained from all individual participants included in the study.
Conflict of Interest
Dr. Pedersen is on the Scientific Advisory Board of Dixi Medical, USA, a manufacturer of equipment for epilepsy surgery evaluations, but this is not related to the present work. Dr. Drane receives support from Medtronic, USA, but this is not related to the present work. The other authors declare no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Okada, N.S., McNeely-White, K.L., Cleary, A.M. et al. A virtual reality paradigm with dynamic scene stimuli for use in memory research. Behav Res (2023). https://doi.org/10.3758/s13428-023-02243-w
Accepted:
Published:
DOI: https://doi.org/10.3758/s13428-023-02243-w