1 Introduction

Sensorimotor adaptation is the “modification of a movement from trial to trial based on error feedback” (Bastian 2008). It takes place when already learned movements are performed repeatedly under changed conditions. One of the best known of these conditions in experimental psychology is the use of prism glasses, through which the world is perceived upside down, mirrored or displaced. The movements are initially erroneous but become more and more accurate over time. The adaptation of the movement is not only reflected in this increase in performance, but also in the decrease in performance when the prism glasses are removed. The difference to the initial performance level is called (negative) aftereffect (Harris 1965).

In virtual reality research, the term aftereffect is also used, but here it has a different meaning: It refers to any effect of a virtual reality (VR) that is observed after the participant has returned to the physical world. These effects can relate to the execution of movements, but also to well-being (also called simulation sickness or cybersickness, Stanney and Salvendy 1998). To distinguish the two meanings, we use the term "side effect" for the general effect of VR and the term “aftereffect” for the specific effect of sensorimotor adaptation.

Biocca and Rolland (1998) published the first study in which sensorimotor adaptation was systematically investigated in VR. The visual displacement was realised using a video see-through Head-Mounted Display (HMD). They found that people showed poorer performance after the visual displacement, but adapted over time and that there was an aftereffect after taking off the HMD. They concluded that research on sensorimotor adaptation is central to our understanding of how to better design VR.

However, sensorimotor adaptation in VR is not only an undesired consequence of an imperfect technology but can also be used intentionally. Rossetti et al. (1998) have shown that prism adaptation can be used as therapy in neglect patients. About 20 years later, a meta-analysis has found this form of therapy to be one of the most common and effective therapies for neglect patients (Yang et al. 2013). Because VR glasses offer advantages in handling compared to prism glasses, there are approaches to implement this form of therapy in VR (Cho et al. 2020; Gammeri et al. 2020).

Although there are several findings showing that sensorimotor adaptation occurs using video see-through HMDs or virtual environments (Biocca and Rolland 1998; Bruggeman et al. 2007; Groen and Werkhoven 1998; Hartman 2018; Lee and Park 2020; Littman 2009, 2011; Maksimovic et al. 2020), it is not yet clear whether the same processes as in prism glasses are involved. The question of whether prism adaptation can be fully transferred to VR is important for several reasons: First, VR as a research tool can be used to investigate further sensorimotor adaptation, because it offers the possibility for complex, but simultaneously controlled environments (Bohil et al. 2011). Second, when considering VR as the research objective, we should understand sensorimotor adaptation in VR better before using VR in different applications. Although VR technology is getting increasingly better, some researchers consider that a perfect simulation of reality will never be possible (Stoffregen et al. 2003), i.e. that a certain adaptation process will always take place. One should therefore not rely exclusively on technological development, but try to understand the adaptation process. For the intended use of sensorimotor adaptation in VR, knowledge is necessary to provide guidelines, for example, for virtual prism adaptation therapy, so that inconsistent results due to different treatment apparatus and treatment duration do not occur (Yang et al. 2013).

2 Interaction with the environment

Sensory discrepancies are more noticeable when a movement is performed goal-directed because in this case, errors are particularly relevant and must be corrected so that the movement intention can be achieved (see discussion about “active movement hypothesis” in Kornheiser 1976). The goal-directed interaction with the environment is thus a crucial factor for the adaptation process and is considered as one of the most powerful variable when examining VR adaptation (Welch 2002). In prism experiments, the relationship between the number of interactions and the aftereffect has already been investigated by Dewar (1970), Fernández-Ruiz and Díaz (1999), and Welch (1971b). The purpose of this study is to investigate whether the same relationship also applies to VR. We conducted an experiment in which participants performed pointing movements in a virtual environment. In the following, the hypotheses derived from prism adaptation research will be presented.

When goal-directed interaction is crucial for sensorimotor adaptation, the question arises as to how many of these are sufficient to cause an aftereffect. In the literature on prism adaptation after-effects have been reported after two to five interactions during prism exposure (Dewar 1970; Fernández-Ruiz and Díaz 1999; Welch 1971b). Therefore, we hypothesised that five pointing movements in a visual displaced virtual environment are sufficient to cause an aftereffect.

H1: Performing five pointing movements leads to a greater magnitude of the aftereffect compared to performing zero pointing movements during the adaptation phase.

Studies have shown that a higher number of interactions during prism exposure was associated with a larger aftereffect (Dewar 1970; Fernández-Ruiz and Díaz 1999; Welch 1971b). Welch (1971b) also reported that after thirty-five interactions there was no further increase in the magnitude of the aftereffect. Thus, we hypothesised that thirty-five pointing movements in a visual displaced virtual environment produce a greater aftereffect compared to only five pointing movements.

H2: Performing thirty-five pointing movements leads to greater magnitude of the aftereffect compared to performing five pointing movements during the adaptation phase.

Fernández-Ruiz and Díaz (1999) not only investigated the magnitude of the aftereffect as a function of the number of interactions, but also the persistence of the aftereffect after removal of the prism glasses. They found a positive association, i.e. with a higher number of interactions the aftereffect lasted longer. We hypothesised the same relationship in sensorimotor adaptation in a virtual environment.

H3: Performing thirty-five pointing movements leads to a greater persistence of the aftereffect compared to performing five pointing movements during the adaptation phase.

3 Methods

3.1 Participants

A total of 30 individuals (aged 22–29 years; 14 men, 16 women) were recruited to participate in this experiment. Eligibility criteria included healthy, right-handed individuals with normal or corrected-to-normal vision and no neurological disease affecting the upper body. All participants gave their informed consent prior to data collection and received course credits as compensation for participation. The study was approved by the research ethics committee of the Department of Psychology and Ergonomics at Technische Universität Berlin.

3.2 Apparatus

3.2.1 Physical environment

The experimental setup is shown in Fig. 1. During the experiment, the participants sat at a table with their heads positioned in a chin rest. A Head-Mounted Display (HTC-Vive, HTC Corporation 2016) was used to present the virtual environment. The participants wore closed-ear headphones to signal the beginning and ending of a pointing movement. The HTC Vive controller was attached to the participants' right arm using a finger orthosis. This allowed tracking of the index finger position without restricting the natural pointing movement.

Fig. 1
figure 1

Illustration of the experimental setup

Full size image

3.2.2 Virtual environment

The virtual environment is shown in Fig. 2. In this virtual environment, the participants also sat at a table. At this table, 50 cm in front of them, they saw a wooden board 30 cm wide and 60 cm high. On this wooden board, a target with a diameter of 3 cm appeared. After the pointing movement, feedback was given in the form of a green circle and the participants’ hand was displayed in the form of a virtual hand.

Fig. 2
figure 2

The virtual environment after the pointing movement

Full size image

3.3 Procedure

After reading an informed consent and agreeing to participate, participants completed a demographic questionnaire. The HTC Vive controller was then attached to the right hand and calibrated. The pupil distance was measured to calibrate the HTC Vive. After the participants had read the instructions and had the opportunity to ask comprehension questions, they entered the virtual environment and the experiment began.

The task in the virtual environment was to point as accurately as possible at a target. The position of the target could vary from its initial position by 3 cm both vertically and horizontally, resulting in nine possible target positions. The sequence of the target positions was predetermined. A trial was structured as follows: The start of the pointing movement began with a resting arm position on the table. After a signal tone, a target appeared on the wooden board and the participants reached out their hands to point at it. After the participants reached the wooden board, they received visual feedback in the form of a green circle and their virtual hand (see Fig. 2). When the participants pointed exactly into the centre of the target, the green ring completely filled the outer ring of the target. After the feedback became visible, the position of the virtual hand could no longer be corrected. The pointing position was held for one second. A second tone signalled the end of the pointing movement. The visual feedback disappeared and the participants put their arm back on the table. After one second the first tone signalled the start of the next pointing movement. The first and second signal tone differed in their frequency. Warning messages appeared if the participants did not adhere to the signal tones.

The experimental procedure was based on the prism adaptation paradigm (Kornheiser 1976) and is illustrated in Fig. 3. The experiment was divided into four phases: familiarisation, baseline, adaptation, and readaptation phase. In the familiarisation phase, the virtual hand was visible the whole time during the first 10 trials. During the next 20 trials, it was only visible after the pointing movement, as in all subsequent phases. In the baseline phase, the performance of the participants was recorded in order to use it as a reference value for the later performance.

Fig. 3
figure 3

Schematic illustration of the experimental procedure

Full size image

In the adaptation phase, the virtual environment was visually displaced to the right with a strength of 20 dioptres (or 11.31°). In order to test the hypotheses, participants were randomly assigned to one of three groups (10 participants in each group): The control group (CG), the little interaction group (LG) and the high interaction group (HG). The interaction groups were balanced by sex and age. The control group performed no pointing movements in the adaptation phase and thus had not experienced any visual displacement. The little interaction group performed five pointing movements and the high interaction group thirty-five pointing movements in the adaptation phase. Both interaction groups were informed in advance about the visual displacement. In the readaptation phase, all groups performed 30 pointing movements without visual displacement. The first target was positioned at the bottom right. This target position was not included in the adaptation phase in order to measure the aftereffect for a new target. During the entire experiment, a ten-second rest period was inserted after a maximum of 15 trials. In the end, the participants were debriefed and any remaining questions about the study were answered.

3.4 Design and data analysis

All analyses were carried out using R (R Core Team 2020). The magnitude of the aftereffect (Hypotheses H1 and H2) and the persistence of the aftereffect (Hypothesis H3) were analysed in dependence on the number of pointing movements.

There are different ways to operationalise the aftereffect (Prablanc et al. 2020). We defined magnitude and persistence of the aftereffect according to Fernández-Ruiz and Díaz (1999), as our hypotheses were largely derived from this work. For the calculation of magnitude and persistence of the aftereffect, we used the 30 data points in the readaptation phase and baseline corrected them for each person individually. Magnitude of the aftereffect was defined as the horizontal deviation at the first pointing movement, where the target position was new and participants had not yet been given feedback after the visual displacement was removed. Persistence of the aftereffect was defined as the sum of the absolute horizontal deviation of the remaining 29 pointing movements. Closed-loop pointing (i.e. with visual feedback) in an undistorted environment (i.e. when wearing neutral glasses or without glasses) after the adaptation phase is called washout in the prism adaptation literature. During washout, the measurement of the error is associated with the persistence or robustness of the aftereffect (Prablanc et al. 2020). To test for the hypothesised differences in magnitude and persistence we used a Linear Model (Fox 2016) as described in the results section.

4 Results

4.1 Magnitude

We applied the following linear model regression:

$${Y}_{i}= \alpha + {\gamma }_{1}{d}_{1i}+ {\gamma }_{2}{d}_{2i}+ {\varepsilon }_{i}$$

The three groups were dummy coded with the control group as reference. The equations for the three groups are therefore as follows:

$$\begin{gathered} Y_{i} = \alpha + \varepsilon_{i} \;{\text{for}}\;{\text{ the}}\;{\text{ control}}\;{\text{ group }}(d_{1i} \; = \;0,\;d_{2i} \; = \;0) \hfill \\ Y_{i} = \alpha + \gamma_{1} + \varepsilon_{i} \;{\text{for }}\;{\text{little}}\;{\text{ interaction }}\;{\text{group}}\;(d_{1i} \; = \;1,\;d_{2i} \; = \;0) \hfill \\ Y_{i} = \alpha + \gamma_{2} + \varepsilon_{i} \; {\text{for}}\;{\text{ high }}\;{\text{interaction}}\;{\text{ group}}\;(d_{1i} \; = \;0,\;d_{2i} \; = \;1) \hfill \\ \end{gathered}$$

Hypotheses H1 and H2 about the differences of the little and high interaction group, respectively, compared to the control group were formulated as statistical hypotheses about the parameters \({\gamma }_{1}\) and \({\gamma }_{2}\). Figure 4 shows the magnitude of the aftereffect for the three interaction groups. Since the horizontal deviation to the left was coded with negative numbers, a larger magnitude was equivalent to higher negative values. The statistical analysis revealed that the magnitude of the aftereffect in the little interaction group was significantly greater than in the control group (H1: \({\gamma }_{1}\) < 0), t(27) = −3.50, p < 0.001, \(\widehat{d}\) = −1.56. Furthermore, the magnitude of the aftereffect in the high interaction group was significantly greater than in the little interaction group (H2: \({\gamma }_{2}\) < \({\gamma }_{1}\)), t(27) = −1.77, p = 0.04, \(\widehat{d}\) = −0.79. The hypotheses H1 and H2 were thus confirmed.

Fig. 4
figure 4

Magnitude: Horizontal deviation in cm as a function of the adaptation group (CG = control group, LG = little interaction group, HG = high interaction group). Points represent individual values, bars represent group means, and error bars represent standard errors

Full size image

4.2 Persistence

We used the same linear model and the same coding scheme for persistence as for magnitude. Figure 5 shows the persistence of the aftereffect for the three interaction groups. The statistical analysis revealed that the persistence of the aftereffect was significantly greater in the high interaction group than in the little interaction group (H3: \({\gamma }_{2}\) >\({\gamma }_{1}\)), t(27) = 3.59, p < 0.001, \(\widehat{d}\) = 1.61. Hypothesis H3 was thus confirmed.

Fig. 5
figure 5

Persistence: Sum of the absolute horizontal deviation in cm as a function of the adaptation group (CG = control group, LG = little interaction group, HG = high interaction group). Points represent individual values, bars represent group means, and error bars represent standard errors

Full size image

In order to analyse group differences in the decrease in absolute horizontal deviation, we grouped the 29 trials into 6 blocks: Block 1 (trials 2 to 5), Block 2 (trials 6 to 10), Block 3 (trials 11 to 15), Block 4 (trials 16 to 20), Block 5 (trials 21 to 25), and Block 6 (trials 26 to 30). Because the number of trials per block was not constant, we no longer used the sum but the mean of the absolute horizontal deviation as dependent variable. We performed analysis of variance (ANOVA) with group (control, little, high) and block (1–6) as independent factors (we used Greenhouse–Geisser corrections). Figure 6 shows the progression over the six blocks for the three groups. The interaction between group and block was significant, F(4.86, 65.56) = 3.86, p = 0.004. Post hoc analysis revealed that the mean absolute horizontal deviation was significantly higher for the high interaction group in comparison to the other two groups in the first four blocks, the other Bonferroni-corrected pairwise comparisons were not significant (see Fig. 6).

Fig. 6
figure 6

Mean of the absolute horizontal deviation in cm as a function of blocks (Block 1 to 6) and the adaptation group (control group, little interaction group, high interaction group). Bars represent group means, and error bars represent standard deviation. All significant Bonferroni-corrected pairwise comparisons are indicated using asterisks (*p < 0.05, **p < 0.01, ***p < 0.001, ****p < 0.0001)

Full size image

5 Discussion

5.1 Relationship between number of interactions and aftereffect

The present study revealed that even five pointing movements in VR are sufficient to elicit an aftereffect. The number five was taken from prism experiments, which were consistent in finding an aftereffect (Dewar 1970; Fernández-Ruiz and Díaz 1999; Welch 1971b). It is also possible that fewer interactions will suffice. Indeed, Fernández-Ruiz and Díaz (1999) reported aftereffects after only three interactions and Dewar (1970) even after two interactions. This shows how quickly the sensorimotor system can adapt and that it is under constant maintenance (Held 1965).

Furthermore, the results indicate that interactions in VR are positively associated with the magnitude of the aftereffect. The little interaction group showed an aftereffect of about 30% of the visual displacement. The high interaction group showed an aftereffect of about 50% of the visual displacement. The concrete relationship is expected to be asymptotic rather than linear, as in prism adaptation (Welch 1971b). However, since we did not realise the same number of levels of the independent variable as Welch, this assumption would need to be investigated in further studies. In prism experiments, the aftereffect is on average 38% of the visual displacement with a maximum value of 73% (for review, see Facchin et al. 2019). As an answer to why the aftereffect is never as large as the adaptation, i.e. never reaches 100%, Facchin et al. (2019) experimentally demonstrate the so-called hand-centred aftereffect. Its direction is opposite to the total aftereffect and therefore reduces it accordingly (for details, see Facchin et al. 2019).

The third finding indicates that interactions in VR are positively associated with persistence of the aftereffect. The aftereffect lasted longer with thirty-five interactions compared to five interactions. Fernández-Ruiz and Díaz (1999) concluded for prism adaptation that once the adaptation process reaches an asymptote, further interactions only affect the persistence of adaptation. Yin and Kitazawa (2001) studied long-term effects of prism adaptation in monkeys. For 500 trials, the monkeys showed an aftereffect that did not differ from each other after 24 h and 72 h. They conclude that the decay of the aftereffect can be very slow and may last for weeks once it has consolidated. If this also applies to VR adaptation in humans, it would mean that especially prolonged VR experiences would result in a particularly persistent aftereffect.

Welch (2002) suggested two possible solutions to eliminate the aftereffect after VR use. First, he suggests establishing readaptation procedures before the user attends to other tasks. The results from the present experiment indicate that the longer the duration of VR use, the longer the duration of such readaptation procedures should be planned. Second, both adaptation and readaptation take less time the more times the same sensory discrepancy has been experienced. This effect is referred to as dual adaptation (Welch et al. 1993). Accordingly, users should systematically alternate between adapting to a virtual environment and readapting to the normal environment in order to shift from one to the other with little effort (Welch 2002). Perhaps a longer VR experience needs more alternations between VR and reality to achieve the same effect.

5.2 VR as a research tool

As already mentioned in the introduction, it is important to investigate the question whether the same processes underlie prism and VR adaptation. In the present study, the same relationship between the number of interactions and the aftereffect was demonstrated for VR adaptation as in prism adaptation. Nevertheless, when using VR glasses as a substitute for prism glasses, one cannot generally assume that identical results will be obtained. When comparing the aftereffects, for example, the question arises whether the VR aftereffect should be measured within the VR environment without virtual displacement or in reality.

Ramos et al. (2019) compared prism and VR adaptation and found that the VR aftereffect was larger than the prism aftereffect. The prism aftereffect was measured in reality after the prism goggles were removed. The VR aftereffect was measured within the virtual environment with the VR goggles on. This difference in the experimental setup could partly be responsible for the results. In a review paper Kornheiser (1976) wrote that the process of perceptual adaptation itself can be classically conditioned. This means that the more different the context is between the adaptation and readaptation phase, the smaller the magnitude of the aftereffect should be. Several studies show that simply removing the prism glasses is a discriminative stimulus and leads to a smaller aftereffect compared to replacing prism glasses with clear glasses (Kravitz 1972; Uhlarik and Canon 1970; Welch 1971a). In Ramos' experiment, the adaptation and readaptation phases were more similar in the VR condition than in the prism condition because the VR glasses were not removed, unlike the prism glasses. If in the VR condition the aftereffect had been measured in reality, the similarity between the adaptation and readaptation phases would have been much lower than in the prism condition. The similarity difference thereby depends on the fidelity of the VR environment. That fidelity has an impact on the aftereffect is suggested by the results of Norris et al. (2001). The authors have shown that more realistic representations lead to larger aftereffects than less realistic representations. In addition, they examined carryover effects from one representation during the adaptation phase to another during the readaptation phase. They found that less realistic representations during the adaptation phase resulted in smaller carryover effects than more realistic representations. In the present study, the aftereffect was measured within VR, thus excluding possible carryover effects into reality. However, for the application of VR goggles as a substitute for prism goggles, e.g. for virtual prism adaptation therapy, aftereffects outside VR are particularly interesting and should also be investigated in further research.

The effect of realistic body representation does not seem to be due to the fact that the error feedback is more direct in the realistic representation. Aziz et al. (2020) compared three types of feedback in three prism experiments: At the end of the pointing movement, either both the pointing finger and the target line were shown (direct error feedback, Experiment 1), or only the pointing finger without the target line (indirect error feedback, Experiment 2), or a representation of the pointing finger as a line and the target line (low-fidelity direct error feedback, Experiment 3). Although participants in Experiment 3 received direct error feedback, the aftereffect was significantly smaller than in the other two experiments, which did not differ significantly from each other. The realistic representation of the hand seems to be more important for adaptation than explicit error feedback.

In addition to fidelity, other differences are possible that influence VR adaptation. Anglin et al. (2017) compared visuomotor adaptation in reality with a simulated environment viewed through an HMD. In the VR environment, participants made greater use of cognitive strategies; in reality, adaptation occurred more strongly through implicit learning. The authors concluded that the VR environment attracted more attention, which in turn may have an impact on the adaptation process. They raise the question of whether this difference is maintained as familiarisation and experience with the VR environment increases.

5.3 VR as a research object

So far, we have discussed what differences may exist when the same prism paradigm is applied in VR. When VR is no longer viewed as a research tool, but as a research object, many more questions arise. The experiences that can be had with VR glasses far exceed the possibilities of prism glasses. Thus, sensory discrepancies may emerge that could not previously be investigated without VR. An example is body-related illusions, where individual body parts change their proportions (Banakou et al. 2013; Kilteni et al. 2012). Another example is interaction techniques that intentionally use sensory discrepancies to make a particular interaction possible. In redirected walking (Nilsson et al. 2018), for example, motion, in reality, is decoupled from motion in VR to the extent that the user can be directed in a specific direction. Redirected walking is a good example of how some discrepancies in VR can only be experienced through interaction. In contrast, with prisms, discrepancies are perceptible even without interaction. You can see immediately when you put on the glasses that the environment is displaced. In the present experiment, the visual displacement was also immediately apparent, as we wanted to replicate the results from the prism experiments in VR. It is quite possible that the relationship between the number of interactions and the magnitude and persistence of the aftereffect is stronger when the discrepancy is tied to the interaction.

Finally, VR can be used to simulate naturally rearranged sensory environments, i.e. to adapt to these environments in VR before actually entering them (e.g. preflight adaptation training, Welch 2002). For this to be successful, distortions, which need not only be visual, should correspond as closely as possible to those of the real world.

In addition to intentional sensory discrepancies, unintentional errors in VR can also occur. They do not have to remain stable over the VR usage period. Hartman (2018) has shown that the unpredictability of visual rotation in VR has an influence on the adaptation of pointing movements. Unintentional errors can also be multidimensional, i.e. visual deviations can occur with respect to several dimensions simultaneously. People can successfully adapt to such multidimensional distortions, but the adaptation process is more difficult and requires more time. With very complex distortions, the likelihood of motion sickness increases (Littman 2011). The consequences of intentional and unintentional sensory discrepancies in VR should be further considered in future research in order to establish appropriate VR design principles.