1 Introduction

The number of persons who has handicap or paralysis in the upper limb is increasing with year. As one of the reason, an increasment of the survival rate in stroke patients can be considered. In Japan, the survival rate was doubled in the past five decada. This results in that patients would have paralysis in the after-effect of the stroke. However, the paralisis has a variety. Thus, evaluation method for the upper limb function has been modified with the variety. As examples of the evaluation methods, there are two methods. The first one is manual function test (MFT) which aims for the comparative large number of handicapped person with upper limb [1]. The second one is simple test for evaluating hand function (STEF) which aims to evaluate the ability of agility of upper limb. All the tests are performed under an examiner and evaluated with judement of the examiner and performance time.

These test has four problems. First, the reliability is not high, because the performance is scored by the examiner visually. Second, the management of the data is tough task, beause the performance results is handled by manual. Third, comsumption labor and time in an occupational therapist is high, because they are required to example the test procedure equaly for all the patients. Fourth, the mental effect can happen in patients, because some patients strongly consider the relathionship between them and occupational therapists.

To solve these problem, an evaluation system for rehabilitation with virtual reality (VR) has been researched, recently. Norio et al. developed telerehabilitation system based on VR with multi-sensory feedback [2]. Toshiaki et al. proposed a training system of upper limb with virtual emvironment [3]. These studies can solve the previous problems, however ignore grasping motion of the upper limb. Therefore, this study aims to develope a evaluation system for upper limb motion which is performed on the personal computer screen. In addtion, this study automated STEF which requires to score the grasping motion.

2 Preliminary

2.1 STEF

STEF is performed under one occupational therapist, and can evaluate agility and position of upper limb in comparing with normal person. In STEF, patients moves multiple objects from one position to another position. The motion can help to diagnosis the symptoms of the patients and improve rehabilitation and daily motion after this. STEF consists of ten examinations and uses ten instruments Fig. 1 [4]. One examination is scored from zero to ten ponts. Full marks means 100 points. The targets coveres all the patients who have upper limb function disorder regardless of the sympton types. However patients who has serious disorder cannot move objects, then the patients are excluded from the target.

Fig. 1.
figure 1

Instruments of STEF

Full size image

One examination to be considered in this study is exampled. The examination uses five big balls. Figure 2 shows a platform used in STEF. Each character from A to H means places where some objects move to by patients. In case of right hand evaluation, all the balles are placed on region B at first. Then, a patients moves five balls from region B to region A one by one (and vice versa).

Fig. 2.
figure 2

Platform of STEF instruments

Full size image

2.2 Leap Motion Controller

This study uses a motion sensor (Leap Motion Controller, Leap Motion Inc.) as shown in Fig. 3. This sensor focuses on only finger and hand motions. The size of the sensor is 30 mm (depth) × 80 mm (width) × 12.7 mm (height). The sampling speed is 90 to 150 fps.

Fig. 3.
figure 3

Leap Motion Controller

Full size image

This sensor can detect hand motion with resolution of 0.7 mm [5]. The detective area is 500 mm in one edge and spread out with upper position when the sensor is placed on the table as shown in Fig. 4. The range of height is from 25 mm to 600 mm [6]. Thefore, this study employs 500 mm as hand motion distance, which is corresponding with actual examination. The coordinate system is as follows. X, Y, and Z axes are right direction (positive), upper direction (positive), and drawing direction (positive), respectively. Although Leap Motion Controller can trace hand motion with the highest accuracy among the motion capture devices, the error along Y axis is large. Therefore, this study focuses on only X and Z axes motion.

Fig. 4.
figure 4

Detective area of leap motion controller

Full size image

3 Method

Subjects sit on a chair in front of a PC screen as shown in Fig. 5. Leap Motion Conroller is placed on a desk which is in front of the screen. The hand of the subjects is displayed on the screen with virtual ball objects. The subjects moves the virtual ball object from one place to another place on the screen. The grasping condition of the hand is recongnized by Leap Motion Controller and immeiately reflected on the virtural environment.

Fig. 5.
figure 5

Measurement environment

Full size image

The starting condition on the virtual environment is shown in Fig. 6. This is an example for left hand examination. Five virtual ball are placed on the left. A grey line can be seen in the right. The subjects should move the all the ball over the line one by one. The blue box (center) is the start button on this panel. When pushing the start button the measurement starts.

Fig. 6.
figure 6

Starting conditions (Color figure online)

Full size image

In this system, subjects feels difficulty to comprehend the relathionship between ball and hand, because the ball is virtual object and cannot tell the contacting feeling. To overcome this problem, this study employs color coding to tell the hand condition for the subject as visual feedback. Here, the hand condition can be classified into three situations: non-contact, contact, and grasp. Each condition is colored seperately as shown in Fig. 7.

Fig. 7.
figure 7

Hand condition to the virtual objects. (Color figure online)

Full size image

The five virtual objects are also colored as red, deep green, blue, yellowish green, and violet. When moving the virtual object over the goal line (grey line), color of the virtual object is changed as water to make it easy to understand the achievement. When finishing movement of all the virtual objects over the goal line, the measurement time was recorded automatically. The unit of measurement time is second.

The hand condtion can be recognized by considering the coordinates of the hand and balls. Here, the hand coordinate is F x , F y , and F z . The center coordinate of the ball is M x , M y , and M z . The distance d between the finger tip and the ball is calculated by Eq. (1).

$$ d = \sqrt {\left( {F_{x} - M_{x} } \right)^{2} + \left( {F_{y} - M_{y} } \right)^{2} + \left( {F_{z} - M_{z} } \right)^{2} } $$
(1)

When d is less than threshold value th, the hand is recognized as contacting the ball. th is set to be the radius of the ball. Leap Motion Controller can diagnose the bending angle of the fingers. When two or more fingers bend and contact the ball, the hand is recognized as grasping the ball.

4 Experiments

The number of subjects was six (Sex: six males, Age: 25 ± 3, Dominant hand: all right-handed). The evaluation method is to score the speed to move the objects as actual evaluation. In this study, the measurement time was compared. Ten trials were performed in left and right hands.

5 Results

The results of the measurment time is shown in Table 1. This results indicate that the right hand has higher agility than left hand.

Table 1. Measurement time in left and right hands.
Full size table

6 Discussions

This study assigned ten trials to subjects. The performance time was improved with trials in all the subjects. The reason is a learning ability of the subjects for appropriate hand motion in this system. Therefore, it might be better to assign a constant training time for the subjects.

The measurement time of the right hand is faster than left hands in all cases. Perhaps, dominant hand effects this results. However there are no data to conclude that. Therefore, we have to apply this system to left-handed person.

7 Conclusion

This study proposes an automated system for STEF with Leap Motion Controller. The proposed system was applied to healthy six subjects. The measurement time is automatically recorded. Then, the proposed system could judge the hand condition which cannot be realized in conventional systems. Therefore, the proposed system is usefull for occupational therapists. As a future work is to apply this system to the handicapped person.