Abstract
Virtual reality (VR) is increasingly used in neuroscientific research to increase ecological validity without sacrificing experimental control, to provide a richer visual and multisensory experience, and to foster immersion and presence in study participants, which leads to increased motivation and affective experience. But the use of VR, particularly when coupled with neuroimaging or neurostimulation techniques such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), or transcranial magnetic stimulation (TMS), also yields some challenges. These include intricacies of the technical setup, increased noise in the data due to movement, and a lack of standard protocols for data collection and analysis. This chapter examines current approaches to recording, pre-processing, and analyzing electrophysiological (stationary and mobile EEG), as well as neuroimaging data recorded during VR engagement. It also discusses approaches to synchronizing these data with other data streams. In general, previous research has used a range of different approaches to technical setup and data processing, and detailed reporting of procedures is urgently needed in future studies to ensure comparability and replicability. More support for open-source VR software as well as the development of consensus and best practice papers on issues such as the handling of movement artifacts in mobile EEG-VR will be essential steps in ensuring the continued success of this exciting and powerful technique in neuroscientific research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adamovich SV, August K, Merians A, Tunik E (2009) A virtual reality-based system integrated with fmri to study neural mechanisms of action observation-execution: a proof of concept study. Restor Neurol Neurosci 27(3):209–223. https://doi.org/10.3233/RNN-2009-0471
Asbee J, Parsons TD (2021) Effects of transcranial direct current stimulation on cognitive and affective outcomes using virtual stimuli: a systematic review. Cyberpsychol Behav Soc Netw. https://doi.org/10.1089/cyber.2020.0301
Bassolino M, Franza M, Bello Ruiz J, Pinardi M, Schmidlin T, Stephan MA et al (2018) Non-invasive brain stimulation of motor cortex induces embodiment when integrated with virtual reality feedback. Eur J Neurosci 47(7):790–799. https://doi.org/10.1111/ejn.13871
Baumann S, Neff C, Fetzick S, Stangl G, Basler L, Vereneck R, Schneider W (2003) A virtual reality system for neurobehavioral and functional MRI studies. Cyberpsychol Behav 6(3):259–266. https://doi.org/10.1089/109493103322011542
Baumgartner T, Speck D, Wettstein D, Masnari O, Beeli G, Jäncke L (2008) Feeling present in arousing virtual reality worlds: prefrontal brain regions differentially orchestrate presence experience in adults and children. Front Hum Neurosci 2:8. https://doi.org/10.3389/neuro.09.008.2008
Beck L, Wolter M, Mungard NF, Vohn R, Staedtgen M, Kuhlen T, Sturm W (2010) Evaluation of spatial processing in virtual reality using functional magnetic resonance imaging (FMRI). Cyberpsychol Behav Soc Netw 13(2):211–215. https://doi.org/10.1089/cyber.2008.0343
Beeli G, Casutt G, Baumgartner T, Jäncke L (2008) Modulating presence and impulsiveness by external stimulation of the brain. Behav Brain Funct 4:33. https://doi.org/10.1186/1744-9081-4-33
Bischof WF, Boulanger P (2003) Spatial navigation in virtual reality environments: an EEG analysis. Cyberpsychol Behav 6(5):487–495. https://doi.org/10.1089/109493103769710514
Bohil CJ, Alicea B, Biocca FA (2011) Virtual reality in neuroscience research and therapy. Nat Rev Neurosci 12(12):752–762. https://doi.org/10.1038/nrn3122
Brunyé TT, Hussey EK, Fontes EB, Ward N (2019) Modulating applied task performance via transcranial electrical stimulation. Front Hum Neurosci 13:140. https://doi.org/10.3389/fnhum.2019.00140
Burgess N, Maguire EA, Spiers HJ, O'Keefe J (2001) A temporoparietal and prefrontal network for retrieving the spatial context of lifelike events. Neuroimage 14(2):439–453. https://doi.org/10.1006/nimg.2001.0806
Burns CG, Fairclough SH (2015) Use of auditory event-related potentials to measure immersion during a computer game. Int J Hum-Comput Stud 73:107–114. https://doi.org/10.1016/j.ijhcs.2014.09.002
Calhoun VD, Pekar JJ, McGinty VB, Adali T, Watson TD, Pearlson GD (2002) Different activation dynamics in multiple neural systems during simulated driving. Hum Brain Mapp 16(3):158–167. https://doi.org/10.1002/hbm.10032
Carvalho KN, Pearlson GD, Astur RS, Calhoun VD (2006) Simulated driving and brain imaging: combining behavior, brain activity, and virtual reality. CNS Spectr 11(1):52–62. https://doi.org/10.1017/S1092852900024214
Chase HW, Boudewyn MA, Carter CS, Phillips ML (2020) Transcranial direct current stimulation: a roadmap for research, from mechanism of action to clinical implementation. Mol Psychiatry 25(2):397–407. https://doi.org/10.1038/s41380-019-0499-9
Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC (1992) The CAVE: audio visual experience automatic virtual environment. Commun ACM 35(6):64–72. https://doi.org/10.1145/129888.129892
de Gelder B, Kätsyri J, de Borst AW (2018) Virtual reality and the new psychophysics. Br J Psychol 109(3):421–426. https://doi.org/10.1111/bjop.12308
de Oliveira SMS, Pimenta de Medeiros CS, Pacheco TBF, Bessa NPOS, Silva FGM, Tavares NSA et al (2018) Electroencephalographic changes using virtual reality program: technical note. Neurol Res 40(3):160–165. https://doi.org/10.1080/01616412.2017.1420584
Debener S, Minow F, Emkes R, Gandras K, de Vos M (2012) How about taking a low-cost, small, and wireless EEG for a walk? Psychophysiology 49(11):1617–1621. https://doi.org/10.1111/j.1469-8986.2012.01471.x
Delaux A, de Saint Aubert J-B, Ramanoël S, Bécu M, Gehrke L, Klug M et al (2021) Mobile brain/body imaging of landmark-based navigation with high-density EEG. Eur J Neurosci. https://doi.org/10.1111/ejn.15190
Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134(1):9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
Dey A, Chatburn A, Billinghurst M (2019) Exploration of an EEG-based cognitively adaptive training system in virtual reality. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, Osaka, pp 220–226
Ehinger BV, Fischer P, Gert AL, Kaufhold L, Weber F, Pipa G, König P (2014) Kinesthetic and vestibular information modulate alpha activity during spatial navigation: a mobile EEG study. Front Hum Neurosci 8:71. https://doi.org/10.3389/fnhum.2014.00071
El Basbasse Y, Packheiser J, Peterburs J, Maymon C, Güntürkün O, Grimshaw G, Ocklenburg S (2023) Walk the Plank! Using mobile EEG to investigate emotional lateralization of immersive fear in virtual reality. R Soc Open Sci 10:221239. https://doi.org/10.1098/rsos.221239
Eng K, Siekierka E, Pyk P, Chevrier E, Hauser Y, Cameirao M et al (2007) Interactive visuo-motor therapy system for stroke rehabilitation. Med Biol Eng Comput 45(9):901–907. https://doi.org/10.1007/s11517-007-0239-1
Ferrucci R, Serino S, Ruggiero F, Repetto C, Colombo D, Pedroli E et al (2019) Cerebellar transcranial direct current stimulation (tDCS), leaves virtual navigation performance unchanged. Front Neurosci 13:198. https://doi.org/10.3389/fnins.2019.00198
Gwin JT, Gramann K, Makeig S, Ferris DP (2010) Removal of movement artifact from high-density EEG recorded during walking and running. J Neurophysiol 103(6):3526–3534. https://doi.org/10.1152/jn.00105.2010
Herrmann CS, Strüber D, Helfrich RF, Engel AK (2016) EEG oscillations: from correlation to causality. Int J Psychophysiol 103:12–21. https://doi.org/10.1016/j.ijpsycho.2015.02.003
Hofmann SM, Klotzsche F, Mariola A, Nikulin V, Villringer A, Gaebler M (2021) Decoding subjective emotional arousal from EEG during an immersive virtual reality experience. eLife 10. https://doi.org/10.7554/eLife.64812
Holper L, Muehlemann T, Scholkmann F, Eng K, Kiper D, Wolf M (2010) Testing the potential of a virtual reality neurorehabilitation system during performance of observation, imagery and imitation of motor actions recorded by wireless functional near-infrared spectroscopy (fNIRS). J Neuroeng Rehabil 7(1):57. https://doi.org/10.1186/1743-0003-7-57
Jang SH, You SH, Hallett M, Cho YW, Park C-M, Cho S-H et al (2005) Cortical reorganization and associated functional motor recovery after virtual reality in patients with chronic stroke: an experimenter-blind preliminary study. Arch Phys Med Rehabil 86(11):2218–2223. https://doi.org/10.1016/j.apmr.2005.04.015
Jangraw DC, Johri A, Gribetz M, Sajda P (2014) NEDE: an open-source scripting suite for developing experiments in 3D virtual environments. J Neurosci Methods 235:245–251. https://doi.org/10.1016/j.jneumeth.2014.06.033
Kim G, Buntain N, Hirshfield L, Costa MR, Makana CT (2019) Processing racial stereotypes in virtual reality: an exploratory study using functional near-infrared spectroscopy (fNIRS). In: Schmorrow DD, Fidopiastis CM (eds) Augmented cognition, vol 11580. Springer. Lecture Notes in Computer Science, Cham, pp 407–417
King JA, Blair RJ, Mitchell DGV, Dolan RJ, Burgess N (2006) Doing the right thing: a common neural circuit for appropriate violent or compassionate behavior. Neuroimage 30(3):1069–1076. https://doi.org/10.1016/j.neuroimage.2005.10.011
Kober SE, Neuper C (2011) Sex differences in human EEG theta oscillations during spatial navigation in virtual reality. Int J Psychophysiol 79(3):347–355. https://doi.org/10.1016/j.ijpsycho.2010.12.002
Kober SE, Neuper C (2012) Using auditory event-related EEG potentials to assess presence in virtual reality. Int J Hum-Comput Stud 70(9):577–587. https://doi.org/10.1016/j.ijhcs.2012.03.004
Lamb R, Antonenko P, Etopio E, Seccia A (2018) Comparison of virtual reality and hands on activities in science education via functional near infrared spectroscopy. Comput Educ 124:14–26. https://doi.org/10.1016/j.compedu.2018.05.014
Landowska A, Royle S, Eachus P, Roberts D (2018) Testing the potential of combining functional near-infrared spectroscopy with different virtual reality displays—oculus rift and oCtAVE. In: Jung T, Dieck MC (eds) Augmented reality and virtual reality. Springer. Progress in IS, Cham, pp 309–321
Lapborisuth P, Faller J, Koss J, Waytowich NR, Touryan J, Sajda P (2019) Investigating evoked EEG responses to targets presented in virtual reality. In: Annual international conference of the IEEE engineering in medicine and biology society. IEEE Engineering in Medicine and Biology Society, pp 5536–5539. https://doi.org/10.1109/EMBC.2019.8856761
Lau-Zhu A, Lau MPH, McLoughlin G (2019) Mobile EEG in research on neurodevelopmental disorders: opportunities and challenges. Dev Cogn Neurosci 36:100635. https://doi.org/10.1016/j.dcn.2019.100635
Lefaucheur J-P (2019) Transcranial magnetic stimulation. In: Handbook of clinical neurology, vol 160, pp 559–580. https://doi.org/10.1016/B978-0-444-64032-1.00037-0
Liang M, Starrett MJ, Ekstrom AD (2018) Dissociation of frontal-midline delta-theta and posterior alpha oscillations: a mobile EEG study. Psychophysiology 55(9):e13090. https://doi.org/10.1111/psyp.13090
Lin C-T, Chuang S-W, Chen Y-C, Ko L-W, Liang S-F, Jung T-P (2007a) EEG effects of motion sickness induced in a dynamic virtual reality environment. In: Annual international conference of the IEEE engineering in medicine and biology society. IEEE Engineering in Medicine and Biology Society, pp 3872–3875. https://doi.org/10.1109/IEMBS.2007.4353178
Lin C-T, Chung I-F, Ko L-W, Chen Y-C, Liang S-F, Duann J-R (2007b) EEG-based assessment of driver cognitive responses in a dynamic virtual-reality driving environment. IEEE Trans Biomed Eng 54(7):1349–1352. https://doi.org/10.1109/TBME.2007.891164
Luck SJ (2014) An introduction to the event-related potential technique, A Bradford book series, 2nd edn. MIT Press, Cambridge. Available online at https://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=798344
Ma, L., Marshall, P.J. & Wright, W.G. (2022). The impact of external and internal focus of attention on visual dependence and EEG alpha oscillations during postural control. J NeuroEngineering Rehabil 19, 81. https://doi.org/10.1186/s12984-022-01059-7
Makeig S (1993) Auditory event-related dynamics of the EEG spectrum and effects of exposure to tones. Electroencephalogr Clin Neurophysiol 86(4):283–293. https://doi.org/10.1016/0013-4694(93)90110-H
Mellet E, Laou L, Petit L, Zago L, Mazoyer B, Tzourio-Mazoyer N (2010) Impact of the virtual reality on the neural representation of an environment. Hum Brain Mapp 31(7):1065–1075. https://doi.org/10.1002/hbm.20917
Metzen D, Genç E, Getzmann S, Larra MF, Wascher E, Ocklenburg S (2021) Frontal and parietal EEG alpha asymmetry: a large-scale investigation of short-term reliability on distinct EEG systems. Brain Struct Funct. https://doi.org/10.1007/s00429-021-02399-1
Mueller C, Luehrs M, Baecke S, Adolf D, Luetzkendorf R, Luchtmann M, Bernarding J (2012) Building virtual reality fMRI paradigms: a framework for presenting immersive virtual environments. J Neurosci Methods 209(2):290–298. https://doi.org/10.1016/j.jneumeth.2012.06.025
Parsons TD (2015) Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front Hum Neurosci 9:660. https://doi.org/10.3389/fnhum.2015.00660
Peterson SM, Furuichi E, Ferris DP (2018) Effects of virtual reality high heights exposure during beam-walking on physiological stress and cognitive loading. PloS One 13(7):e0200306. https://doi.org/10.1371/journal.pone.0200306
Putze F, Herff C, Tremmel C, Schultz T, Krusienski DJ (2019) Decoding mental workload in virtual environments: a fNIRS study using an immersive n-back task. In: 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, Berlin, pp 3103–3106
Reggente N, Essoe JK-Y, Aghajan ZM, Tavakoli AV, McGuire JF, Suthana NA, Rissman J (2018) Enhancing the ecological validity of fMRI memory research using virtual reality. Front Neurosci 12., Article 408:408. https://doi.org/10.3389/fnins.2018.00408
Schubring D, Kraus M, Stolz C, Weiler N, Keim DA, Schupp H (2020) Virtual reality potentiates emotion and task effects of alpha/beta brain oscillations. Brain Sci 10(8). https://doi.org/10.3390/brainsci10080537
Seraglia B, Gamberini L, Priftis K, Scatturin P, Martinelli M, Cutini S (2011) An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation. Front Hum Neurosci 5:176. https://doi.org/10.3389/fnhum.2011.00176
Slobounov SM, Ray W, Johnson B, Slobounov E, Newell KM (2015) Modulation of cortical activity in 2D versus 3D virtual reality environments: an EEG study. Int J Psychophysiol 95(3):254–260. https://doi.org/10.1016/j.ijpsycho.2014.11.003
Stolz C, Endres D, Mueller EM (2019) Threat-conditioned contexts modulate the late positive potential to faces-A mobile EEG/virtual reality study. Psychophysiology 56(4):e13308. https://doi.org/10.1111/psyp.13308
Talkington WJ, Pollard BS, Olesh EV, Gritsenko V (2015) Multifunctional setup for studying human motor control using transcranial magnetic stimulation, electromyography, motion capture, and virtual reality. J Vis Exp 103. https://doi.org/10.3791/52906
Taylor RM, Hudson TC, Seeger A, Weber H, Juliano J, Helser AT (2001) VRPN. In: Green M, Shaw C, Wang W (eds) Proceedings of the ACM symposium on virtual reality software and technology – VRST '01. The ACM symposium. Baniff, Alberta, Canada, 11/15/2001–11/17/2001. ACM Press, New York, p 55
Török A, Sulykos I, Kecskes-Kovacs K, Persa G, Galambos P, Kobor A et al (2014) Comparison between wireless and wired EEG recordings in a virtual reality lab: Case report. In: 2014 5th IEEE conference on cognitive Infocommunications (CogInfoCom). IEEE, Vietri sul Mare, pp 599–603
Tremmel C, Herff C, Sato T, Rechowicz K, Yamani Y, Krusienski DJ (2019) Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG. Front Hum Neurosci 13:401. https://doi.org/10.3389/fnhum.2019.00401
Tromp J, Peeters D, Meyer AS, Hagoort P (2018) The combined use of virtual reality and EEG to study language processing in naturalistic environments. Behav Res Methods 50(2):862–869. https://doi.org/10.3758/s13428-017-0911-9
Ulsamer P, Pfeffel K, Müller NH (2020) Brain activation in virtual reality for attention guidance. In: Zaphiris P, Ioannou A (eds) Learning and collaboration technologies. Human and technology ecosystems, vol 12206. Springer. Lecture Notes in Computer Science, Cham, pp 190–200
Wang W-E, Ho RLM, Gatto B, van Der Veen SM, Underation MK, Thomas JS et al (2020) A novel method to understand neural oscillations during full-body reaching: a combined EEG and 3D virtual reality study. IEEE Trans Neural Syst Rehabil Eng 28(12):3074–3082. https://doi.org/10.1109/TNSRE.2020.3039829
Wiesing M, Fink GR, Weidner R (2020) Accuracy and precision of stimulus timing and reaction times with unreal engine and SteamVR. PloS One 15(4):e0231152. https://doi.org/10.1371/journal.pone.0231152
Wolter M, Armbruester C, Valvoda JT, Kuhlen T (2007) High ecological validity and accurate stimulus control in VR-based psychological experiments
You SH, Jang SH, Kim Y-H, Hallett M, Ahn SH, Kwon Y-H et al (2005) Virtual reality-induced cortical reorganization and associated locomotor recovery in chronic stroke: an experimenter-blind randomized study. Stroke 36(6):1166–1171. https://doi.org/10.1161/01.STR.0000162715.43417.91
Acknowledgments
The authors thank Yasmin El Basbasse for help with literature research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Ocklenburg, S., Peterburs, J. (2023). Monitoring Brain Activity in VR: EEG and Neuroimaging. In: Maymon, C., Grimshaw, G., Wu, Y.C. (eds) Virtual Reality in Behavioral Neuroscience: New Insights and Methods. Current Topics in Behavioral Neurosciences, vol 65. Springer, Cham. https://doi.org/10.1007/7854_2023_423
Download citation
DOI: https://doi.org/10.1007/7854_2023_423
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42994-1
Online ISBN: 978-3-031-42995-8
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)