The home-training system consists of an HMD with integrated cameras connected to a personal computer running custom-made software (Fig. 1).
The HMD is based on a commercially available display (Z800 3DVisor, eMagin, Bellevue, WA) using organic light-emitting diode technology and providing flicker-free images with a resolution of 800 × 600 pixels per eye. The optics of the HMD provide a horizontal field of view (FOV) of 32° and correct geometrical distortions and aberrations. The two displays can be shifted independently in a horizontal direction to adapt to different interocular distances. Two 1.3-megapixel cameras (UI-1642LE-C, Imaging Development Systems, Obersulm, Germany) were mounted on the display using a custom-made fixture made of aluminum. The camera positions could be adapted to the interpupillar distance and the declination angle according to the users’ needs. The camera optics (Lensagon BM3516ND, Lensation, Karlsruhe, Germany) were chosen for low geometrical distortion and low chromatic aberration to ease postprocessing. This setup, in principle, allows stereoscopic 3-D presentations, but this feature was not used in the present version of the training system, which is based on a simpler 2-D version.
Controlling computer and software
The software was implemented on a personal computer (PC) running a Windows XP Professional operating system. The image data of the camera system were transferred to the PC via the universal serial bus (USB). Then a processed virtual environment was integrated with the image by the software. The combined output was postprocessed and shown on the HMD. The core of the software consists of computer-vision algorithms, which allow the separation of body parts from a unicolor background and detect several hand features like the positions of cardinal points, such as the fingertips (see Bach et al., 2010). The system is capable of either presenting the mirrored limb alone in front of a neutral background or as an overlay of the image captured by the cameras. In principle, the software also allows for separate manipulation of left and right images in order to provide a 3-D presentation. Due to methodological and technical considerations, we did not yet use this feature in the present setup and only processed and displayed images from the left camera to both eyes. However, 3-D will be implemented in future versions. For user interaction, a computer mouse and a foot pedal were connected via USB. The possibility of connecting the computer to the Internet permits the automatic transfer of the training data to a central server.
The computer system is set up such that the training software automatically starts after powering up the computer. This eliminates the possibility of system failures due to operating errors. Before the training task begins, the software presents a screen with questions to be answered by clicking on radio buttons with a mouse. With this feature, external measures for monitoring training success—for example, pain ratings—can be obtained and stored along with the training parameters. Then an FOV calibration is performed, ensuring that the user is seated in a position in which only the unicolor background is captured by the cameras. After these preliminary steps, the training tasks start.
The training tasks
All tasks are based on the general concept of capturing the image of one hand, cropping it from the background, and presenting it in a horizontally mirrored position, mimicking the contralateral hand.
Finger flexion task
The finger flexion task is a more sophisticated version of the hand-closing/opening task used in most mirror-training implementations. Specified fingers have to be moved individually, putting more focus on fine motor skills and yielding a more precise vision–motor coupling.
Participants look in the direction of one hand while seeing a mirrored image mimicking their other hand (Fig. 2a). Participants are instructed to find a comfortable posture for viewing their hand, avoiding unnecessary straining of arm and hand. Once a comfortable posture has been achieved, participants confirm this by pressing a foot pedal. Fingertip positions are then automatically captured by the software, and the task starts.
Open circles surrounding the fingertips are displayed. Each trial of the finger flexion task consists of two phases. In the memorizing phase, participants are required to keep their fingertips within the boundaries indicated by the open circles and watch these circles being filled by green color in a random sequence. The number of fingers, which is indicated in this sequence, is determined by previous performance (see below). Then, in the reproduction phase, participants are to flex individual fingers in the sequence presented in the memorizing phase. As soon as a fingertip leaves its boundary, the circle turns yellow. This allows participants to quickly grasp whether they have indeed flexed the intended finger only. Participants have to keep the finger flexed—and the fingertip outside the boundary—for at least 1 s. Then, the response is accepted, indicated by the boundary circle turning green, and participants can extend their finger so that the fingertip is situated in the boundary again. If the participant manages to flex his or her fingers in the correct sequence, a trial is successfully completed, and a smiley, along with some words of praise, appears. Flexing fingers out of sequence (including accidental flexion of more than one finger at once) immediately cancels the trial and triggers the display of a frowney, along with some uplifting words.
Immediately after each trial, the next trial is started, as long as the predefined duration of 180 s (excluding time for calibration and feedback screens) has not been exceeded. After a successful trial, the length of the finger flexion sequence is increased by one more item; after an unsuccessful trial, its length is reduced by one item. After the last trial, a clock icon appears, along with a message stating that the training time is over.
Hand posture task
In the hand posture task, patients have to combine mental imagery with extensive movements of the whole arm and fine movement skills of the fingers.
Target hand postures are presented as white silhouettes, similar to an inverted shadowgraphy game (Wikipedia contributors, 2012b). The participants’ task is to move their mirrored hand into a posture fitting the silhouette. In order to solve the task, a predefined amount of overlap between the arm projection and the silhouette has to be achieved. At the right side of the display, a color-coded scale indicates the amount of overlap. Once the overlap criterion has been reached, the trial is solved, a smiley along with some words of praise appears, and then the next trial begins. In total, the set consists of 41 different silhouettes of varying difficulty, ranging from a closed fist to very complex forms. Each training session starts with an easy trial; then silhouettes are picked at random by the software. Participants are given a maximum time of 30 s to complete one trial. If they do not succeed within this period, the trial is counted as unsuccessful, and the next trial starts. This feature was included to prevent frustration arising from being “stuck” with very difficult silhouettes for the whole training period. New trials are started until the net training duration of 180 s has been exceeded. This means that if a new silhouette appears after a net training duration of 150 s, the user still has 30 s to finish the task. This can extend the training time for a maximum of 29 s, depending on the starting time of the last silhouette.
Finger-guided Snake game
This task focuses on moving the hand in space with shoulder and elbow movements while the finger and wrist are kept mostly rigid.
In this classical Snake game (Wikipedia contributors, 2012a), the participants’ task is to guide a virtual snake to virtual cookies. The snake follows the extended index finger of the mirrored hand. The speed of the snake can be controlled via the distance between snake and finger: The greater the distance, the faster the snake moves. Care has to be taken that the snake does not “bite” itself: Its “head” must collide neither with its “tail” nor with the screen’s borders. With each successfully “eaten” cookie, difficulty is increased by elongating the snake. After unsuccessful trials—the snake “biting” itself or moving outside the display boundaries—difficulty is decreased. New trials are started until the net training duration of 180 s has been exceeded.
In the ball-grasping task, task, elbow and shoulder movements, as well as fine movement skills of the thumb and index finger, are trained.
The goal is to grasp a ball by forming a “C” with the thumb and index finger and carry it to a quadratic target area, where it has to be slightly squeezed in order to finish the trial. As in the other tasks, participants perform movements with one hand while seeing its mirrored image. Each contact with the ball induces movement, resembling a rubber ball: After being pushed, it rolls away, and even if the ball has been caught, it may “wriggle” out again if the thumb and index finger are held too far apart. If the ball is squeezed too tightly or if it is pushed too hard, it changes color as a warning signal; if the force becomes too high, it eventually pops. New trials are started until the net training duration of 180 s has been exceeded.