1 Introduction

With the resurgence of virtual reality (VR), cybersickness has become a growing concern for researchers, developers, and users alike. Previous studies have shown that a large portion of the population [40–60% according to a survey by Kolasinski (1995)] may experience moderate-to-severe cybersickness in virtual environments. While there are several theories on reasons underlying cybersickness, there does not exist an easy or systematic method of measuring and quantifying cybersickness from one moment to another. Without the existence of a reliable tool to measure and interactively quantify cybersickness, understanding and mitigating it remains a challenge. Early work on studying cybersickness and motion sickness relied on examining physiological changes such as sweating and increased heart rate, leading to a standardized self-evaluation form for determining the intensity of sickness the person experienced, the Simulator Sickness Questionnaire (Kennedy et al. 1993). A limitation of this approach is that measuring the effects of cybersickness requires either interrupting the subject during the experience (Fernandes and Feiner 2016) (thereby affecting the experience itself and thus the results) or waiting until the end of the experience to assess their symptoms, which relies on the subject accurately recalling their sickness (Rebenitsch and Owen 2014). This survey-based qualitative approach is unable to provide real-time quantitative measurements, making it difficult to objectively assess real-time cybersickness in the virtual environment.

In this paper, we present the results of a user study that measures and examines cybersickness experienced by participants wearing a commercially available HMD and EEG headset. For this study, we designed a 3D environment and a camera path that was likely to evoke a moderate degree of cybersickness among participants. During this experience, the subjects’ brain activity is measured using an EEG device and compared against a baseline EEG, when the scene is stationary. In addition, we also had participants continuously self-report their level of sickness with a joystick interface. We compared the self-reported data with their time–frequency spectral EEG information showing a correlation between the EEG data and the self-report data.

This paper makes the following contributions to understand and quantify cybersickness in virtual environments:

  • We establish that cybersickness in an immersive HMD is correlated with brain-wave activity measured by EEG;

  • We find a statistically significant correlation of Delta-, Theta-, and Alpha-waves with self-reported cybersickness;

  • Our approach facilitates ease of measurement and characterization of cybersickness by using inexpensive, commodity off-the-shelf devices for VR headsets and EEG devices.

2 Related work

LaViola Jr (2000), Holmes and Griffin (2001) found that common symptoms of cybersickness include nausea, increased heart rate, disorientation, sweating, eye strain, and headaches. One of the prevailing theories on the cause of cybersickness (also referred as simulator sickness or visual fatigue) is the sensory conflict theory, which attributes it to the dissonance between the visual and the vestibular sensory cues (Cobb et al. 1999; Kolasinski 1995). This happens, for instance, when a user is immersed in a moving virtual environment, while stationary in the real world. The sensory conflict between what the eyes see and what the body feels is believed to lead to a physiological sense of discomfort and associated cybersickness. Cybersickness is closely related to motion sickness. Motion sickness is often induced by the unsettling movement, such as travel in vehicles or aircraft or amusement rides, but can also be caused with a mismatch of visual and vestibular sensation. Many previous works have concluded that cybersickness is more severe in VR than on screen-based simulators (Stanney et al. 1997; Patrick et al. 2000; Sharples et al. 2008) due to increased fields of view being correlated with increased levels of reported cybersickness. Some of the techniques to mitigate cybersickness have therefore relied on adjusting the field of view (Fernandes and Feiner 2016) or minimizing the visual and vestibular mismatch (Weech et al. 2018). A highly creative solution to resolving this mismatch was devised by Maeda et al. (2005), who used galvanic vestibular stimulation to produce the sensation of vection or movement. Riecke et al. (2005) reduced motion sickness by increasing a user’s sense of self-motion without physically moving. This was elegantly accomplished through auditory cues, seat vibrations, and the introduction of subtle scratches in the periphery of the projection screen.

In contrast to the above, a highly innovative research direction has been in examining the role of peripheral vision (Sun and Varshney 2018) in cybersickness. Rebenitsch and Owen (2016) presented a thorough review of modern techniques to detect and measure cybersickness and urge for more research in the minimally understood subject. In their review, they state that the usage of EEG for such an endeavor is rare, noting only one related previous work. In a seminal study, Lin et al. (2002) found that a user’s visual field of view was positively correlated to their simulator sickness (SSQ) scores. More recently, Fernandes and Feiner (2016) devised a clever solution to mitigating cybersickness by strategically and automatically manipulating the field of view of the wearer of an HMD based on virtual camera movement (full field of view when stationary and narrow field of view when in motion).

Several biological metrics have been used to detect and measure the presence of motion sickness and cybersickness. These include heart rate, respiratory rate, finger-pulse volume, skin conductance, and gastric tachyarrhythmia (Cowings et al. 1986). A challenge with these metrics is that not all people suffer from these symptoms when experiencing cybersickness, and cybersickness is not the only cause of these symptoms (Kolasinski 1995). Other studies use a user-driven metric, where a participant uses a clicker or a joystick to continuously indicate when and how much cybersickness the participant is feeling at that moment (Chen et al. 2010).

2.1 Self-reporting cybersickness

The most common method for measuring cybersickness is to measure the severity of the users’ symptoms using subjective self-reporting surveys (Davis et al. 2014). A commonly used survey is the Simulator Sickness Questionnaire (SSQ) by Kennedy et al. (1993) which assesses sixteen symptoms, with each item rated on a scale of four (none, slight, moderate, and severe). These symptoms have been further grouped into three categories: oculomotor, disorientation, and nausea. Oculomotor symptoms include effects such as fatigue, eyestrain, and difficulty in focusing. Disorientation includes vertigo, dizziness, and blurred vision. Lastly, the nausea category includes symptoms such as sweating, burping, salivation, and nausea (LaViola Jr 2000). While the self-reporting surveys are quite informative, they have the shortcoming that they can be administered only at the end of the simulator session (Rebenitsch and Owen 2014) or require the interruption of an experiment for a study participant to fill out the questionnaire (Fernandes and Feiner 2016). Waiting till the end loses the fine temporal granularity of cybersickness reporting. At the same time, interrupting the participant in a continuous experiment may be undesirable or even impossible. Further, an interruption may result in alteration of physiological symptoms in the study participant which may impact their reporting. For instance, the interruption could result in recovery from motion sickness due to the passage of time and lack of sickness-inducing stimuli. Therefore, passive, but continuous, approaches to measuring cybersickness are highly desirable.

2.2 Measuring motion sickness with EEG

EEG has previously been widely used to measure motion sickness (Wood et al. 1994; Chen et al. 2010; Hu et al. 1999; Ko et al. 2011; Lin et al. 2008, 2007). The advantage of using EEG its passive and objective nature, not requiring interrupting the participant to obtain a subjective measurement. Previous papers have focused on four frequency ranges, Delta (1.0–4.0 Hz), Theta (4.0–7.0 Hz), Alpha (7.0–13.0 Hz) and Beta (13.0–25.0 Hz). Kim et al. (2005) found that an increase in the delta power with a decrease in beta power was indicative of cybersickness during an object-finding VR experiment which used a rear-projected CAVE (cave automatic virtual environment) display system to show the visual stimuli. Another study by Min et al. (2004) concluded that a decrease in delta power was indicative of visually-induced motion sickness in a car-driving experiment which used a standard rear-projected display to show the visual stimuli. Chen et al. (2010) built a driving simulator using a motion platform inside a \(360^{\circ }\) rear-projection display, in order to provide both visual and vestibular stimulation to induce motion sickness. In this study, each participant used a controller to continuously log their level of motion sickness. By using independent component analysis (ICA) with time–frequency analysis and cross-correlation analysis, the authors were able to examine the EEG changes in brain-wave activity when induced by both visual and vestibular stimuli. They found a more complex interaction of power increases and decreases in different regions of the brain as the level of motion sickness changed. Another set of studies by Naqvi et al. (2015, 2014) recorded the EEG signals of participants viewing a movie on a 3D LCD TV in either 3D or 2D in order to determine if 3D movies cause greater visual fatigue. Their study found a decrease in theta-power for the frontal regions of the brain in addition to a decrease in beta power in the temporal region. To the best of our knowledge, there has not yet been a systematic study that has used EEG to measure and quantify cybersickness for users in immersive virtual environments wearing head-mounted displays. Given the previous work quantifying motion sickness using EEG, we also believe EEG is appropriate for quantifying cybersickness.

3 Materials and methods

Our study evaluates the EEG dynamics of cybersickness from binocular visual stimuli in a virtual reality head-mounted display. To the best of our knowledge, this is the first study that uses EEG signals to continuously evaluate cybersickness in participants wearing head-mounted displays.

We used a 14-channel, 128 Hz, Emotiv Epoc EEG device, which has been successfully validated (Ekanayake 2010) and used for EEG research for a variety of research studies, including measuring the cognitive load (Anderson et al. 2011), examining the relationship between the environment and happiness (Aspinall et al. 2015), and as a proof of concept for robust and mobile EEG recording in the outdoors (Debener et al. 2012). We used the original HTC Vive head-mounted display, which has a \(110^{\circ }\) field of view and a resolution of \(1080 \times 1200\) per eye with a refresh rate of 90 Hz, for both eyes. In this user study, the participants were limited to rotational viewing with no translational movement permitted. The participants viewed a 3D stereo rendered scene in the head-mounted display that involved a fly-through of a virtual spaceport with twisting, turning, accelerating, and decelerating of the virtual camera. A screenshot of the scene is shown in Fig. 1. The virtual environment was created using a custom 3D rendering engine using C++ and OpenGL, written to be synchronized with the EEG recording process, all running on a Windows 8 machine with an Intel Xenon 2.6 GHz CPU and an NVIDIA GTX 1080 GPU. In addition to the HMD and EEG devices, we used a Thrustmaster joystick device, which the participants used to manually record their current level of cybersickness during the camera fly-through. The participants were instructed to indicate, by tilting the joystick in any direction, the magnitude of their sickness. They were told that no tilt indicated that they felt no sickness and that full tilt indicated extreme sickness. We have examined the correlation between the sickness reported by the participants and their EEG brain-wave recordings. We obtained University of Maryland Institutional Review Board (IRB) approval for our study on May 26th 2016, with IRB approval ID \(887087-1\).

Fig. 1
figure 1

A still from the virtual spaceport flythrough used in our cybersickness study

Full size image

3.1 Participants

We recruited 44 participants from our university campus and surrounding community for the user study, of which 31 were male and 13 were female, with an average age of 27 and standard deviation of 8 years. Every participant had normal or corrected-to-normal vision (self-reported). The study session for each participant lasted around 30 min. Due to technical problems associated with the EEG recording interface, we had to discard one participant’s data. We have used the EEG data from the remaining 43 participants for our analysis (Figs. 2, 3, and 4).

3.2 Experimental protocol

Each study participant was explained the entire procedure and how they would interact with the HMD and the input joystick mechanism. First, the EEG device was placed on the participant’s head and manually configured until the EEG device showed that all electrodes had registered good contact with the head. Second, the participant donned the HMD and their interpupillary distance (IPD) was adjusted so that the participant could comfortably see the 3D stereo rendering in the HMD. Third, the participant was given the joystick in their hand and explained that they had to push the joystick based on how cyber sick they felt. Each participant was explained each of the possible symptoms they may experience, for example, headaches, stomach awareness, nausea, vomiting, pallor, sweating, fatigue, drowsiness, dizziness and disorientation, and that if they experience any of them including any general discomfort to report that sensation, and its intensity, using the joystick. Finally, each participant was given 60 s to get used to and comfortable with the EEG, HMD, and the joystick. We believed that this one minute of acclimatization would be sufficient to distinguish the effects of cybersickness from the initial effects induced by VR exposure. Indeed, Fig. 9 shows that the EEG power spectra is statistically different between baseline (blue) and virtual flythrough (green) with \(p < 0.001\). To accurately simulate the normal experience of using VR, throughout the entire study session, the participants were standing while wearing both the EEG and HMD as well as holding the joystick in their hands. To minimize the risk of injury, the study was conducted in a safe and managed environment.

The duration of the fly-through condition was at 61 s. Although there have been studies that have used lengthy conditions for cybersickness, recent research indicates that shorter provocations are also effective in discerning the physiological effects of vection-induced cybersickness. Specifically, Fransson et al. (2019) have examined postural instabilities arising due to vection in a study which is designed similar to ours. In their study, they place participants in an HTC Vive VR headset and show them a 90 s VR movie Desert Ride Coaster, while they stand still. Within 90 s they report being able to record significant postural instabilities in participants. Similarly, Keshavarz and Hecht (2011) have shown that motion sickness experienced in the first minute is highly correlated with later motion sickness. Others have used even shorter duration trials for assessing vection-induced cybersickness. For instance, Weech et al. (2018) decided to use a trial duration of 30 s. Research has shown that a shorter duration provides a superior test–retest reliability (Le Clair and Riach 1996), and that durations significantly over a minute may be too lengthy for participants (Duarte and Freitas 2010). We observe a 0.49 correlation between SSQ and average joystick scores, with \(p < 0.05\), affirming that the SSQ scores and self-reported joystick scores, over the one-minute-long fly-through condition are consistent.

We first we took a baseline EEG reading of the participants after they were acclimated to being in a VR environment. During the baseline EEG recording, we instructed participants to slowly and precisely tilt the joystick back and forth intermittently in the same way they will tilt the joystick during the actual fly-through condition. During the baseline condition, the user location in the virtual spaceport was static, but they were allowed to rotate their view orientation by turning their head. The participants were asked to make slow, repeated, and deliberate head movements left and right, and then up and down while wearing the HMD and EEG devices to minimize any risk of injury and electrode separation. The main difference between the baseline and fly-through condition was the control the user had in changing the view of the camera. In the baseline condition, users had complete control of where and what they saw. In the fly-through condition, participants could still look around, but their location and heading continuously updated as they flew through the virtual environment. Our study design builds on how vestibular and visual mismatch elicits cybersickness (Cobb et al. 1999; Kolasinski 1995).

The participants were then re-instructed to use the joystick device to report their sickness levels and was then virtually flown through the scene, which lasted approximately one minute. Our pre-study trials showed that if a participant was at all susceptible to cybersickness, they would most certainly feel sick in the one-minute virtual fly-through. Restricting the flythrough to one minute kept the exposure time minimal for participant safety but long enough to record a satisfactory amount of data. Throughout the flythrough, each participant used the joystick to continuously log their self-reported level of cybersickness, with no tilt corresponding to no reported sickness, and full tilt as severe sickness.

3.3 Signal acquisition and pre-processing

As discussed earlier, we recorded the brain-wave activity using an Emotiv Epoc EEG with 14-channels sampling at 128 Hz. The name and locations of each of the nodes/channels The EEG headset uses a saline electrolyte solution on the contact heads. The raw data was acquired and saved to disk using the Emotiv Epoc C++ SDK which was integrated into our rendering program. This enabled the EEG recording and camera path to be synchronized for all participants. We used MATLAB with the commonly used EEGLAB for the EEG signal processing (https://sccn.ucsd.edu /eeglab/). The first part of signal processing involved importing each participant’s raw Emotiv EEG data into MATLAB and then into EEGlab for both the baseline and virtual flythrough recordings. Once the data is loaded, the mean power for each channel is calculated and subtracted from that channel’s data, centering the signals. Next, a high-pass filter with a cut-off frequency of 1 Hz was used in conjunction with a low-pass filter with a cut-off frequency of 50 Hz to remove any unwanted noise from the signals. We also manually inspected the filtered EEG signals for any recording anomalies, which can occur if a subject moves too abruptly or if an electrode temporarily loses contact. In our study, we found EEG recording anomalies for only one subject, and we simply removed their data from further processing. For this one subject, we found several instances where signals fluctuated with a large magnitude or were zero. After pre-processing, we exported the data into an EEGlab study package.

While participants flew through the space-port, they reported their cybersickness with the joystick, with more tilt indicating stronger sickness. The joystick was sampled at 90 Hz, the same as the frame-rate of the HMD. The cybersickness level is a score between zero and one, which is reported continuously in real time without interrupting the experiment.

3.4 Independent component analysis

Similar to previous work on EEG analysis (Chen et al. 2010; Delorme et al. 2007), we decompose our filtered EEG signals, for each subject, into independent components using independent component analysis (ICA) (Makeig et al. 1996; Delorme and Makeig 2004) using EEGLab. The intuition behind the use of ICA is that the observed EEG signals are the result of a mixture of sources throughout the brain and scalp, which are assumed to be independent, such as eye blinks, muscle movement, or other psycho-physiological stimuli, including cybersickness. EEGLab uses an enhanced version of the infomax ICA algorithm, whose goal is to minimize the mutual information among the data projections or maximize their joint entropy (Delorme and Makeig 2004). In our study, we apply ICA to the EEG recordings for each individual subject, resulting in 14 independent components per participant. From the calculated independent components, we cluster similar components using the built-in EEGLab K-means independent component clustering functionality. The idea is to cluster similar independent components so that similar underlying phenomena, such as spectral amplitude, phase, and coherence perturbations, are grouped together resulting in distinct clusters representative of eye blinks, noise, cyber sickness, and other phenomena.

Fig. 2
figure 2

Averaged scalp maps of clustered independent components. The colors indicate activity power within the scalp topographic map, with red indicating a high-level of activity power, and blue indicating a low-level. A black box outlines the scalp map best correlated with cybersickness (Color figure online)

Full size image
Fig. 3
figure 3

Names and locations of the 14 EEG electrodes in the Emotiv Epoc headset. The colors indicate activity power within the scalp topographic map, with red indicating a high-level of activity power, and blue indicating a low-level. This suggests that the majority of the power activity is centrally located within the scalp map (Color figure online)

Full size image
Fig. 4
figure 4

Virtual camera flythrough of the spaceport that each participant in our study experienced. Note how the above correspond to the self-reported cybersickness levels in Fig. 5

Full size image

3.5 Time–frequency analysis

During the study, participants continuously logged their current feeling of cybersickness using a joystick while virtually flying through the spaceport. We correlated the participants’ self-reported cybersickness levels with the ICA cluster power spectra. As the reported level of cybersickness changes, we hypothesized that the ICA power should also change at different frequencies relative to the strength or weakness of reported cybersickness. To calculate the time–frequency spectra, we used the EEGlab ERSPs (Event-Related Spectral Perturbation) function, resulting in an average of the power-spectrum density over time. The power-spectrum density is then converted into decibel power by the EEGLab.

4 Results

In this section, we review the results of our user study exploring cybersickness in virtual reality using EEG. First, we review the subjective sickness levels and symptoms as reported by each participant during and after the experiment. Second, we examine the results of the EEG analysis, showing a statistically significant difference between the averaged baseline and cybersickness EEG recordings. Third, we review the time–frequency spectral power graphs and compare them to the continuous self-reported sickness levels and show that there is a correlation between them.

4.1 Self-reported cybersickness

After the baseline EEG recording was taken, the EEG measurements during the virtual flythrough phase began. Each study participant was told that they would virtually fly-through the spaceport and if they felt any of the previously mentioned symptoms that they were to indicate their presence and strength by tilting the hand-held joystick device. We refer to this input as the participants’ self-reported cybersickness levels, whose average and standard deviation are shown in Fig. 5. In addition to the joystick information, each participant completed an SSQ form at the end of the study.

Fig. 5
figure 5

Self-reported cybersickness levels using joystick. The blue curve shows the average of all the participants’ self-reported cybersickness levels, with the standard deviation shown in gray around that curve (Color figure online)

Full size image

The highest peaks of the average of participants’ self-reported cybersickness levels, shown in the blue curve of Fig. 5, can be attributed to specific events that occurred in the spaceport fly-through (see Fig. 4). The first peak corresponds to a sudden burst in camera acceleration in close proximity to the surface of the spaceport. The second peak corresponds to a sudden free-fall off an edge of the spaceport. The third peak aligns with the sudden and hard pull-up of the camera after free-falling from the previous event. The fourth peak corresponds to the sudden acceleration upward after the initial camera pull-up. The final peaks correspond to a sudden deceleration of the camera as it comes to rest on the landing platform of the spaceport.

In addition to the self-reported information from the joystick, the Simulator Sickness Questionnaire (SSQ) scores from each participant were collected at the end of the session. The SSQ consists of 16 questions with 4 severity options, with values between 0 to 3, with 0 as none, 1 as slight, 2 as moderate, and 3 as Extreme. The average SSQ scores for each of the 16 symptoms and their variances are shown in Fig. 6. Based on the information from the graph it can be concluded that our participants primarily experienced varying levels of vertigo, dizziness, and general discomfort.

Fig. 6
figure 6

Participant Simulator Sickness Questionnaire (SSQ) scores after our study are shown. The plot shows the median, first and third quartiles (orange and grey respectively), with the minimum and maximum shown as error bars

Full size image

Our presentation of the raw SSQ scores along the 16 factors may initially the impression that the overall SSQ scores are low. We next calculate the weighted SSQ sub-scores for Nausea (N), Oculomotor (O), and Disorientation (D), as well as the Total Severity (TS), as shown in Fig. 7, using the well-established methodology of Kennedy et al. (1993). We observe the mean SSQ scores are N: 23.74, O: 23.09, D: 58.59, and TS: 36.44. These rate above 85th percentile for Nausea, above 80th percentile for Oculomotor, above 98th percentile for Disorientation, and above 90th percentile for Total Severity as per the comparative data reported by Kennedy et al. (1993). When we designed the 60-s flythrough, our goal was to induce cybersickness so that we could track it with EEG and these SSQ scores indicate the effectiveness of our virtual flythrough. We also note that for us the Disorientation sub-scores are in general greater than Nausea and Oculomotor sub-scores, which is consistent with previous findings (Stanney and Kennedy 1997) that cybersickness for virtual environments should show such a pattern (as opposed to other kinds of motion sickness in simulators or in weightlessness).

Fig. 7
figure 7

Participant Simulator Sickness Questionnaire (SSQ) weighted sub-scores for Nausea, Oculomotor, Disorientation, and Total Severity are shown here following the well-established methodology of Kennedy et al. (1993). The plot shows the median, first and third quartiles (orange and gray respectively), with the minimum and maximum shown as error bars

Full size image

Study participants self-reported cybersickness through both the joystick and the SSQ survey. From the distribution of SSQ and joystick scores, we see that the participants rated their level of cybersickness from mild to severe. This wide range of symptom intensities suggests that the brain-wave EEG data should be diverse and that not all users will be a part of the cybersickness-revealing independent clusters.

In Fig. 8, we show a comparison of the average level of self-reported cybersickness as reported through the joystick with the sum of reported SSQ sickness level for each participant. The two distributions shown are correlated with a statistically significant score of 0.49 using a Pearson Correlation.

Fig. 8
figure 8

A comparison of the average score as reported by the joystick with the SSQ sum for each participant. The SSQ score and the self-reported cybersickness using the joystick have a statistically significant Pearson Correlation \(r = 0.49\) and \(p = 0.0009 < 0.05\)

Full size image

4.2 Spectral differences

In this section, we compare the differences between the spectral frequencies of the EEG recordings of the baseline (green curve) and the virtual flythrough (purple curve).

To generate these curves, we rely on the resulting scalp maps from the clustered independent components as shown in Fig. 2. From the 14 generated clusters, we found one to be most representative of cybersickness as it had the most statistically significant difference between the baseline EEG frequencies and the virtual flythrough EEG frequencies and had a spatially plausible power concentration. For example, the power concentration is not spatially located around the eyes (eye movement or blinking) or the ears. Further, it represented a meaningful fraction of the participants, composed of 24 out of the 43 total (\(55\%\)) participants. We find it interesting that this is in general agreement with previous research by Stanney and Kennedy (1997) in which they noted that 30–40% of the participants in flight-simulator studies do not experience simulator sickness. These generated clusters, along with the one selected cluster, are shown in Fig. 2. The selected cluster, which to EEGLab is cluster 12, from this point forward to refer to it as cluster A. Figure 3 shows the selected cluster with the EEG node labels in more detail. The Emotiv Epoc uses 14 electrodes, AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4.

Figure 9 shows the mean component power spectra of the selected independent component cluster A for the baseline and virtual flythrough conditions. It is clear from the figure that there is a power increase across many frequencies for participants experiencing the virtual flythrough of the spaceport. In the component cluster spectra plot, we indicate where the EEG power changed significantly using paired t-tests. For the selected ICA cluster, we see that the difference between the baseline and virtual flythrough frequency spectra are statistically significant with (\(p \le 0.01\) for much of the frequency range), using EEGLab’s built-in paired t-test with Bonferroni-correction statistical analysis. Similar to previous work that studied motion sickness, we also see a power increase across many frequency bands for the virtual flythrough scenario compared to the baseline. Previous work has found that an increase in delta (Kim et al. 2005) and delta and theta (Chen et al. 2010) bands were indicative of motion sickness.

Fig. 9
figure 9

Comparison of the EEG power spectra between the baseline (blue) and virtual flythrough (green) for ICA cluster A. The paired t-test with Bonferroni-correction between the two spectra reveal \(p < 0.001\) for much of the frequency ranges

Full size image

The EEG spectral power differences, between the baseline EEG recording with the stationary scene and during the virtual flythrough, indicate that cybersickness can be detected using EEG. To be more specific, we have identified that an increase in spectral power, with respect to a baseline recording, is indicative of the onset of cybersickness. For both recording sessions, the participants used the EEG, the HMD, the joystick, and experienced the same environment while standing. The only difference was the camera motion during the virtual fly-through. We next look at the frequency spectra over time for the selected cluster, to examine when specifically a participant experienced cybersickness and correlate them with their self-reported cybersickness levels.

4.3 Time–frequency with user input signals

During the virtual flythrough of the spaceport, the study participants continuously recorded their current levels of cybersickness through a joystick device. The self-reported cybersickness levels for each participant are shown in Fig. 5 along with the average sickness level shown in black. We used time–frequency analysis to evaluate the EEG spectra changes across all participants with the self-reported sickness levels. The values for all frequencies, averaged over all users, for cluster A is shown in Fig. 10.

Fig. 10
figure 10

Time–frequency visualization of cluster A. The average self-reported cybersickness levels are shown below in red (Color figure online)

Full size image

The average self-reported cybersickness is shown below the time–frequency visualizations in red. Similar to the work by (Chen et al. 2010), we observe a correlation between the spectral power changes shown in the time–frequency plots, especially for the lower-frequency delta and theta bands, and the self-reported cybersickness levels from the participants. To assess the degree of correlation, we computed the correlation between the time–frequency band values and the average self-reported cybersickness information. The Pearson correlation r-value scores for each of the four frequency bands are presented in Table 1. Figure 11 compares each of the frequency bands with the average self-reported cybersickness levels over time.

Table 1 Correlations (Pearson R-values) between average ERSP values for the four frequency bands and the self-reported cybersickness levels
Full size table
Fig. 11
figure 11

Average over four frequency Bands for Cluster A compared with the average self-reported cybersickness (in green). The correlations between the different frequencies and the average reported sickness are shown in Table 1 (Color figure online)

Full size image

Our analysis shows that a statistically significant and high correlation exists for Delta, Theta, and Alpha bands for cluster A with the self-reported cybersickness information from the participants. This is perhaps best illustrated in the Time–Frequency plot with the self-reported cybersickness through joystick input for one of the participants, as shown in Fig. 12.

Fig. 12
figure 12

Visualization of the ERSP from a cluster A participant with self-reported cybersickness levels. Note how the changes in ERSP values, especially for Delta and Theta bands, align with the participant’s self-reported cybersickness

Full size image

4.4 External factors

We note that the high-correlations between the averaged time–frequency signals of the different EEG bands and the average joystick signal may be due to a confounding effect of increased cybersickness and the actual movement of the joystick. However, the joysticks movements were very sparse and were not sustained over long periods of time. The participants were instructed to tilt the joystick only when they felt their level of cybersickness changed. It has been shown in previous studies (Huang et al. 2007, 2008; Chen et al. 2010) that changes in spectral power as a result of finger and hand movements for sustained attention tasks diminished quickly, within the order of a few seconds. The effect would also result in a spectral change and rebound effect within that short period of time, which we do not see in our EEG signals. In addition, during the baseline recording, our participants held and moved the joystick to simulate the same effect and these do not appear in the EEG signals either. Therefore, any changes in the joystick would not have influenced the overall spectral frequency and power differences of our analysis.

Another external factor for consideration is head movement. Participants were instructed to freely look around the same environment that they would be placed in during the fly-through (the spaceport) in the baseline recording session. During the fly-through, the participants could also freely look around their environment as the camera flew through the scene. Therefore, any significant spectral differences between the baseline and sick condition are unlikely to be due to head movement.

Another potential limitation of our study is that the cybersickness provocation could be construed to be too short and the effects we are measuring may be influenced by emotional arousal of being in a virtual reality environment. A study (Gavgani et al. 2017) has reported that only about 50% of participants started to develop nausea (a major symptom of cybersickness) during the first minute of provocation. In our study we have tried to mitigate this by having an extra minute of acclimatization for the participants in the virtual environment. We then used this baseline to measure the brainwave differences during the virtual flythrough. However, this could be a concern that merits further study.

5 Conclusions and future work

Throughout the course of the study, we witnessed a wide range of reactions to the rendered stimuli. Some participants experienced minor discomfort, while others experienced moderate to high levels of cybersickness. Each participant was asked to briefly report what aspect of the experience made them the most cyber sick. They reported that the sudden changes in direction and velocity of movement made them feel ill compared to when the motion was smoother. In addition, they reported that the anticipation of where the camera was going to move heightened their reaction. Lastly, they expressed that if they were in control of the camera, as opposed to the camera being automatically moved, they might have felt less sick due to prior knowledge and mental preparation of what was about to happen. One observation that the test administrator made was that approximately \(70\%\) of participants would lean their bodies, with varying (in some cases, almost alarming) degrees of tilt, based on the motion of the camera. Approximately \(32\%\) of participants had previous experience with a head-mounted display.

In this paper, we have presented our findings of a user study with the goal of continuously measuring and quantifying cybersickness. In our study, the participants wore both an HMD and an EEG recording device, while being presented visual stimuli of a virtual flythrough in a spaceport. The recorded EEG data was decomposed using ICA to separate the underlying sources of the brainwave activity and eliminate noise. The independent components were then clustered across users for the purposes of comparing the EEG of those grouped users. Through independent component analysis and time–frequency spectral analysis, our findings suggest that a spectral power increase in the Delta, Theta, and Alpha frequency bands, relative to a baseline, strongly correlates to the presence of cybersickness. These results are similar to other studies on motion sickness that also found strong indicators in the delta and theta bands (Kim et al. 2005; Chen et al. 2010).

Our findings in this paper are just a first step to the many opportunities that present themselves in using EEG to study cybersickness in virtual environments. Some of the more important amongst these include a better understanding of the sources of cybersickness, the relationship of the duration of immersion to cybersickness, and the effect of age and gender on cybersickness. A number of cybersickness mitigation strategies have been studied over the last decade, but their evaluation has been largely based on questionnaires at the end of the immersive experience. As virtual environments grow in importance across a variety of applications (Krokos et al. 2018, 2019), an exciting direction of future work is in continuous evaluation of the effectiveness of cybersickness mitigation strategies, while the user is immersed in the virtual world. In our study, the participants were not asked to perform a task. It would be interesting to explore what effect if any, task performance has on cybersickness. Finally, it will be highly desirable, if at all possible, to move toward standards of assessing cybersickness and to use them to rate hardware (headsets, trackers, and displays) as well as the content (games, performances, and other immersive experiences).