1 Introduction

1.1 Background

Stroke can cause many neurological impairments which severely reduce patient ability to perform activities of daily life (ADL): approximately 30% of patients with arm paresis do not regain significant dexterity after 6 months [23, 24]. Current therapy techniques are dominated by occupational and physical therapy, which focus on guided limb manipulation and task-oriented exercises. Systems employing virtual reality (VR) technology build on this methodology by increasing the range of possible tasks, partly automating and quantifying therapy procedures, and improving patient motivation using real-time task evaluation and reward [12, 14, 15]. The feedback can be provided either after a task in the form of scores, or during the task using dynamic biofeedback visual and auditory cues [16]. Some systems also provide physical assistance with movement and/or simulate haptic feedback [7, 13, 17, 18, 20, 21, 26, 34].

Besides its potential to trigger external stimulation, we hypothesize that VR can additionally induce use-dependent plastic changes in response to internal stimulation of higher-motor cortical areas. This so-called VR-based interactive visuo-motor intervention is based on the idea that stimulation of the action processing system in turn activates downstream cortical areas involved in movement execution. Here, a population of “mirror neurons” [6] plays a key role since these neurons discharge during both action execution and action observation or imagery [3, 5, 10]. With the recruitment of a widespread movement network normally involved in movement execution, VR-based visuo-motor therapy offers potential for specifically enhancing functional recovery.

1.2 Mirror neurons and limb mirroring

We hypothesize that a system combining two elements: movement observation with intent to imitate and visualization of mirrored movements of the non-paretic limb: may optimally induce cortical plasticity and functional recovery in acute stroke patients. The first element is based on the observation that mirror neurons discharge during goal-directed hand actions and also during observation of another individual performing a similar action. It has been proposed that mirror neurons constitute a vocabulary of hand actions [28]. Their activation leads to recruitment of functionally interconnected cortical structures coupling action execution and observation. The execution–observation system has also been found in humans [3, 5, 6]. Moreover, there is evidence that action observation may also facilitate motor activity [11] and induce cortical plasticity [30]. In addition to action execution and observation, mirror neurons and motor planning areas are known to be activated during voluntary mental motor imagery, which selectively modulates muscle excitability [9]. Recent reviews [4, 29] and studies of motor imagery in stroke rehabilitation show some potential for cortical reorganization in injured sensorimotor areas [19, Gaggiolo et al. 12] and behavioral performance improvements [31].

The second element of our rehabilitation concept: visualizing mirrored movements in the non-paretic arm, is based on Ramachandran’s work on patients with phantom limbs [27], applied using real mirrors to chronic stroke patients [1]. These studies used mirrors placed along the centerline of the patient’s body, so that viewing the reflection of the non-paretic arm in a mirror gave the patient the impression that their paretic arm was able to move. In mirror therapy, cortical representations of hand visual configuration and movements are lateralized contralateral to the limb performing the action [25]. If the visual input provided to a test subject is the mirror image of an upper limb action being performed, the activation switches to the ipsilateral side [8]. This phenomenon could possibly be exploited to stimulate a damaged region of motor cortex by using the non-paretic limb to control a visual representation of the paretic limb. Such a stimulation paradigm may serve to accelerate recovery by recruiting circuits projecting to the affected area. In our system we replace the real mirror with its VR equivalent, allowing us to apply generalized visualization mappings that are not possible with normal mirrors (see following section). The potential advantage of generalized mappings lies in the possibility of specifying different contributions from the paretic and non-paretic arms and hands to the movement of each virtual limb (Table 1).

Table 1 Sample parameter settings for linear mapping function
Full size table

2 Virtual reality visuo-motor therapy system

Our therapy system (Fig. 1) is based on the Torque multi-user 3D gaming environment (GarageGames, Oregon, USA). In the current prototype we are using 3D digital compasses (Honeywell) for arm position input and an 80 cm wide-screen LCD television for audio–visual output. This configuration has been selected to strike a balance between accuracy of tracking, effective user stimulation, patient acceptance and cost.

Fig. 1
figure 1

Overview of VR therapy system

Full size image

The patient is seated in a normal chair or wheelchair at a table facing a monitor, with their arms on the table in front of them (Fig. 1). The image on the monitor shows two arms in the same orientation and relative position, resting on a flat surface representing the table. The movements of the patient’s real arms are transferred to the virtual arms in real time. This close correspondence between the real and virtual arms in terms of position, relative orientation and movement is designed to optimally stimulate the patient to treat the virtual arms as their own during the therapy session. To reinforce the illusion of ownership, the color of the sleeves on the virtual arms can be set to match the clothes that the patient is wearing.

The mapping of the measured left/right limb pose P on to the virtual limb pose P V is determined using a mapping function f with four mapping parameters (α) for the arm and four parameters (β) for the hand/fingers (Fig. 2):

$$ \begin{array}{*{20}c} {{P = }} & {{{\left[ {\begin{array}{*{20}c} {{P_{{\text{L}}} }} \\ {{P_{{\text{R}}} }} \\ \end{array} } \right]}}} \\ {{{\rm A} = }} & {{{\left[ {\begin{array}{*{20}c} {{\alpha _{{{\text{LL}}}} }} \\ {{\alpha _{{{\text{LR}}}} }} \\ \end{array} \begin{array}{*{20}c} {{\alpha _{{{\text{RL}}}} }} \\ {{\alpha _{{{\text{RR}}}} }} \\ \end{array} \begin{array}{*{20}c} {{\beta _{{{\text{LL}}}} }} \\ {{\beta _{{{\text{LR}}}} }} \\ \end{array} \begin{array}{*{20}c} {{\beta _{{{\text{RL}}}} }} \\ {{\beta _{{{\text{RR}}}} }} \\ \end{array} } \right]}}} \\ {{P_{{\text{V}}} = }} & {{f{\left( {P,{\left[ {\begin{array}{*{20}c} {{A_{{1,*}} }} \\ {{A_{{2,*}} }} \\ \end{array} } \right]}} \right)} = {\left[ {\begin{array}{*{20}c} {{f(P_{{\text{L}}} ,\alpha _{{{\text{LL}}}} ,\alpha _{{{\text{RL}}}} ,\beta _{{{\text{LL}}}} ,\beta _{{{\text{RL}}}} )}} \\ {{f(P_{{\text{R}}} ,\alpha _{{{\text{LR}}}} ,\alpha _{{{\text{RR}}}} ,\beta _{{{\text{LR}}}} ,\beta _{{{\text{RR}}}} )}} \\ \end{array} } \right]}}} \\ \end{array} $$
Fig. 2
figure 2

Schematic of mapping from real to virtual space. (P L, P R), left/right limb pose in real world; (P VL, P VR), left/right limb pose in virtual world; α, β, mapping parameters; f, mapping function. Crosses indicate limb anchor points to which the mapping function is applied

Full size image

The mapping function enables a variety of control scenarios to be supported, e.g., a patient with a paretic right limb may benefit if the real left limb assists with moving the virtual right limb to enable easier task success and thus more positive reinforcement. This mapping can be seen as a generalization of mirror therapy [1] applied to VR.

Table 1 lists some commonly used parameter settings for a linear mapping function f. Non-linear mapping functions may be used to deal with boundary conditions related to movement range, task-oriented movement assistance (e.g., “snapping” towards nearby target objects) and smoothing of jerky, poorly controlled movements using the left/right mappings, the control of the movements of the “mirrored” arm can be gradually shifted from the intact arm to the paretic arm as the patient recovers, possibly accelerating further the speed of recovery.

In a normal control mapping, patients observe virtual representations of the movements they make on the screen. If they cannot directly control the movements of a virtual limb by moving the corresponding real limb, whether due to arm paresis or the use of a particular control mapping, they are still instructed to attempt to imitate the movements they see on the screen. It is this observation with intent to imitate that we hypothesize to be the optimal way of stimulating the action recognition system. The visibility of the paretic or non-paretic arms can also be switched-on or -off by the therapist depending on the particular task.

The rehabilitation scenarios provide a graded training programme of goal-oriented reaching and grasping exercises. The initial scenarios we are testing are:

  • Hitting: the patient must intercept virtual balls moving along the table towards him/her. Adjustable parameters are: ball speed, interval between successive balls, lateral left/right dispersion of ball start positions and the probability distribution of the ball start positions.

  • Catching: as for Hitting, with the additional constraint that the patient must “catch” the balls by holding the relevant hand open for the ball intercept and closing it within a certain interval after the ball has been intercepted; otherwise it is registered as a “drop”. Additional parameters specify the time within which patients must close their hands to register a “catch” event.

  • Grasping: as for Catching, but after catching the ball the patient must move the ball to a movable target location and then release it. Partial success in each phase (intercept, catch, release) scores partial points.

3 Usability testing and pilot study

The VR motor therapy system is undergoing usability testing to assess the enjoyment and ease of use of the system, and a clinical pilot trial to assess its potential efficacy. These initial results guide the design of a full-scale clinical study, currently in preparation. Each test is described in more detail below.

3.1 Usability testing

During the technical development, we tested the usability of the system with naïve healthy subjects recruited from an academic conference and patients at a children’s hospital with various neurological ailments. In these test trials, each subject played one or a few games (about 3 min per game). They were asked for feedback about their enjoyment of the game scenario, the ease of use of the input devices and how they would like to improve the game. We also observed how the users interacted with the system to identify weaknesses in the game software, the sensors (construction of the data gloves, difficulties with putting the gloves on or off, etc.) and the psychological effects of the game (patient attention, concentration and motivation).

3.2 Clinical pilot

The clinical pilot study in progress at the Neurology Department of the University Hospital Zurich aims to assess whether our therapy system has the potential to enhance functional recovery, in preparation for a future full clinical study. All the clinical procedures have been approved by the responsible institutional ethics committees and all participants gave written informed consent. Patients with moderate to severe hand paresis meeting the entrance criteria (first ever stroke, cortical or cortical–subcortical stroke, age 18–80 years) are admitted into the trial during the first week after stroke onset. After an initial clinical and functional assessment, patients receive medical treatment and individual physiotherapy including deficit-dependent and ADL training tasks. This therapy consists of basal task- and ADL-oriented physiotherapy consisting of several modules such as vital (e.g. cardio-pulmonary, etc.), static (e.g. posture, position, etc.), mobility (e.g. transfer, gait, etc.), and upper extremity functions. In addition to the normal therapy sessions, patients experience our experimental therapy during one 45-min session per day on 5 days per week over a maximum of 5 weeks. In each session the therapy parameters are set individually by the therapist for each patient to maximise motivation by maximising difficulty, while keeping game scores relatively high (above approx. 85% of the maximum possible score). All patients are assessed weekly during the treatment phase with ADL-oriented scores, including the Chedoke Arm & Hand Activity Inventory (CAHAI) [2] and the motor activity log (MAL) [33].

4 Preliminary results

4.1 Preliminary results: usability testing

In the usability trials, all healthy subjects (N = 19, age 25–36 years) and children’s hospital patients (N = 5, age 7–14 years) quickly learned to use the system to perform the task. For the initial settings (1 ball every 2.5 s, low speed), most subjects could intercept 70–100% of the balls. They had more difficulty with different mapping parameters (following, mirroring). User acceptance of the system was high (anecdotal and questionnaire responses, numerical scale). Most patients expressed a desire to use the system on an ongoing basis. Children requested various improvements to hold their attention, such as background music, better graphics, and stronger storylines.

4.2 Preliminary results: clinical pilot

Six acute stroke patients with moderate to severe arm and hand paresis have completed 3–5 weeks of training using our system. As a group they improved in neurological outcome tests (mean ± standard deviation CAHAI increase 28.4 ± 18.5, mean MAL increase 33.3 ± 21.5), reflecting functional recovery of the paretic arm. Although the aim of this initial study was not to establish the therapeutic efficacy of our system, it was important to demonstrate that our therapy does not have adverse effects, and that it has potential efficacy. These preliminary measures show that our therapy has not prevented patients progress, and suggest that it might add to the efficacy of traditional physiotherapy. To further validate the therapeutic efficacy of our system, a full study with control patients is necessary. Collection of data from control patients is planned in the coming full-scale clinical study.

4.3 System as assessment tool

In addition to assessing patients using clinical scales, we analyzed data collected from our therapy system to determine its potential for use as a simultaneous therapy and assessment tool. The game data can be analysed at several timescales: individual events within each trial, summary information for each trial and across several trials and sessions. Figure 3 shows summarized data from one trial with a right-handed patient with left hemiparesis, with the fates of each of the balls (touched with left hand, touched with right hand, or missed). Most of the “miss” events occurred at the extremities of movement on both sides, where the patient is forced to make a large outwards forearm rotation at the elbow. A greater proportion of the balls were missed on the left side than the right side, indicating reduced mobility of the left arm possibly due to hemiparesis, handedness or both. Figure 4 compares the accuracy of ball interception near the beginning and after the end of therapy. In this patient, the non-paretic left arm was more consistent in its interception offsets both before and after therapy, and was much more accurate than the right arm for balls which came along the midline. The interception offsets tended to increase with the eccentricity of the incoming ball. The paretic right arm showed no discernible change in interception accuracy for balls away from the midline, but it did register an improvement for balls near the midline. In Fig. 5 the improvement in performance can be seen of a sample patient over multiple sessions, with the difficulty (speed, rate of ball appearance) increasing over the course of treatment while the overall score remains constant. The performance increase obtained during therapy persisted at least until the 3-month retest. The relative contribution of paresis and handedness to left/right performance imbalances and overall behaviour patterns can only be judged after longitudinal intra-individual testing during the course of recovery and from inter-individual comparisons.

Fig. 3
figure 3

Sample histogram of patient performance (hit/missed balls) during a single trial of the Hitting therapy scenario. The patient had partial paresis of the left arm. Each trial consisted of 50 balls

Full size image
Fig. 4
figure 4

Comparison of accuracy of interception for hit balls near the beginning (Session 3) and 3 months after the end (Session 25) of treatment. Triangles represent error in hand position for intercept events to the left (white background), to the right (light grey background)and along the centerline (dark grey background). Left/right pointing triangles represent intercept events using the left and right hand, respectively

Full size image
Fig. 5
figure 5

Improvement of an acute stroke patient over time as measured by increase in scenario difficulty (ball speed and ball appearance interval) while maintaining constant performance (balls not missed). Each session consisted of 4–6 trials. The last point in each graph indicates the results for the 3-month retest

Full size image

5 Discussion

Our results show that our visuo-motor stroke rehabilitation method combining action observation with goal-directed movement imagery is promising for providing effective rehabilitation based on validated neuroscientific hypotheses. Patients generally accept our therapy system and are motivated to use it. The preliminary clinical results show that they may benefit from the therapy, but further data from both test and control patient groups is required to allow population-level comparisons. We are optimistic that our system will show patient benefits along similar lines to “normal” mirror-therapy, e.g. [1], but with further benefits such as less therapist intervention and higher patient motivation. In addition, our system may provide improved efficacy of rehabilitation by enhancing patient concentration through the use of the goal-oriented tasks. Analysis of data collected from therapy sessions also suggests that our system can be used as both a therapy and assessment tool. However, more work is needed in defining and calibrating standard tests using the game infrastructure to ensure reproducible results. Future work will focus on further patient testing in acute and chronic stroke patients, development of standardized assessments using the system, and separate investigations into optimizing the patient stimulation conditions to maximize the potential benefits of the system.