Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Wearable sensors have emerged as an important tool in human computer interaction. Physiological signals and human movements that can be captured by wearable sensors, such as electrooculogram (EOG) [1], electromyogram (EMG) [2], electroencephalogram (EEG) [3], and lip motions [4], have been proposed for controlling various types of robots. Nevertheless, few studies have reported the use of wearable sensors for the control of surgical robots, which requires high precision and an ergonomic approach.

We have previously developed a surgical robot to perform advanced endoscopic procedure such as endoscopic submucosal dissection (ESD) [5]. ESD, which is a skillful endoscopic technique that allows en bloc resection of early stage gastrointestinal cancer for reducing the risk of residual cancer, involves the following working steps: marking, injection, cutting mucosa, dissection, proceeding dissection and complete resection [6]. This procedure is scar-less and effective. However, the procedure is also technically demanding and have a high risk of perforation [7]. Therefore, to assist surgeons and endoscopists to complete ESD, we have developed a robot with two arms, a gripper for lifting and a knife for dissecting tissues.

In this study, we reported the use of wearable sensors to recognize hand gestures and arm motions for intuitive control of this surgical robot. This novel way for controlling surgical robot is achieved by using flexible surface EMG bipolar electrodes and inertial measurement unit (IMU) to control discrete actions and positioning the robot respectively. For comparison, a standard motion controller is also used as the benchmark.

2 Surgical Robot and Controller Design

Figure 1 shows the surgical robot used in this study. The two robotic arms are designed with a total of nine degrees of freedom. Each arm is with a continuous structure and attached with a small surgical tool as the end effector for tissue lifting and dissection respectively.

Fig. 1.
figure 1

The surgical robot

Full size image

Two types of controllers are used in this study: (a) a standard motion controller and (b) a wearable sheath that collects EMG and arm motions of the user. As shown in Fig. 2, the standard motion controller is designed with a 2-DOF joystick that controls the continuum section of the robotic arms, a rotational knob that controls the tilting angle of the gripper or the hinge joint of the dissector, as well as a scissor-liked knob that controls the opening and closing of the gripper. The robot arms can be translated by push buttons.

Fig. 2.
figure 2

Two controllers used in this study: (a) a motion controller and (b) a wearable sheath with EMG electrodes and IMU sensors.

Full size image

2.1 Design of the Wearable Controller

The wearable sensory sheath in this study was designed with eight pairs of bipolar electrodes and four IMU sensors. Four bipolar electrodes to record forearm EMG and two IMU sensors to detect the motion (rotation and translation) of each arm of the subject. Figure 3 shows the overview of the control method. EMG features were extracted to control the locking function of the gripper and dissector, while the IMU sensors, which measure velocity, orientation, and gravitational forces of the operator’s arm, were used to position the robotic arms.

Fig. 3.
figure 3

Flowchart of the signal processing of the wearable controller

Full size image

E-textiles materials sewn in the inner surface of the wearable sheath were used as dry electrodes for capturing EMG. Each electrode had a conducting area of 300 mm2 and each pair of electrode has a center-to-center separation of 20 mm. As shown in Fig. 2(b), the four pairs of electrodes were arranged as follows: (1) one pair at the crest of the wrist flexor; (2) one pair at the crest of the wrist extensor; (3) one pair at the anterior distal end; and (4) one pair at the posterior distal end. Two IMU sensors were placed along the same line and parallel to each other, on the upper arm and forearm of the user respectively.

2.2 EMG Signal Recognition and Motion Mapping

Six subjects were invited to participate in this experiment. Each subject was asked to participate in a training session, during which he or she was asked to perform four motions: hand opening, hand closure, wrist flexion and wrist extension. These four postures were used to map to the discrete motion of the robot, as shown in Table 1. The subjects were asked to maintain each posture for five seconds, then relaxed for the next five seconds, and repeated the same for 5 times. The recorded EMG in the training session was used as the reference during real-time control. Mean absolute value (MAV), zero crossing (ZC), slope sign changes (SSC) and waveform length (WL) were selected as the EMG features in our study, since they have been widely verified previously in EMG prosthesis control [8, 9]. The window size for extracting features is chosen to be 150 ms. The linear discriminant analysis (LDA) [10], which finds a linear combination of features to characterize or separate two or more classes of objects or events, was used as the decoding method to classify these extracted features.

Table 1. Discrete motion control by EMG
Full size table

2.3 IMU Signal Recognition and Motion Mapping

The IMU sensor consisted of a three-axial accelerometer, a gyroscope and magnetometers. The IMU signals were measured continuously and processed as an independent dataset to control the rotation, translation and bending of the continuum section of the two robot arms. Two IMU sensors per arm were placed on the upper arm and forearm of each subject. By analyzing the relative motions of the upper arm and forearm, the arm movements of the user were mapped to the continuum sections of the robot arms, governed by the following equations:

$$ {\text{d }} = {\text{ K}}_{ 1} { \sin }\upalpha $$
(1)
$$ {\text{K}} = {\text{ K}}_{ 2}\upbeta $$
(2)
$$ \uptheta \, = {\text{ K}}_{ 3}\upgamma $$
(3)

where d is the translation distance of the robotic arm, θ is the bending angle of the continuum section of the robotic arm, K is the curvature of the continuum section of the robotic arm, K1, K2, K3 are the sensitivity constant of the user’s behavior obtained during the calibration section, α is the angle formed by the upper arm with the vertical plane, β is the angle formed by the forearm with the horizontal plane, and γ is the rotation angle of the forearm.

In order to familiarize the user with the working space environment and the mapped motions of the robotic arms, a calibration procedure was setup in the beginning. The user was asked to keep the initial position at the beginning of the calibration to ensure the mapping is effective. EMG signals of the user were also sent to the computer for real-time classification.

3 Subjects and Experimental Setup

Six subjects (aged 20–30 years old) participated in this experiment, where they were asked to control the surgical robot using the standard motion controller and the wearable sensory sheath in randomized order. With each controller, each subject was asked to perform two designated tasks: (1) to control the dissector to mark a dot in each and every circle shown along a straight line; and (2) to control the lifter to lift a weight from one location to another. The experimental setup is shown in Fig. 4.

Fig. 4.
figure 4

Experimental setup

Full size image

After the calibration session, each subject is allocated 3 min to practice on a virtual platform as well as with the surgical robot to familiarize himself with the mapping. Between two sessions of the experiment, a rest of two minutes was given to each subject to avoid mental and muscle fatigue.

To compare the two control methods, i.e. using the controller and the wearable sheath with EMG and IMU sensors, the accuracy of the EMG classifier, the time to finish each task and the offset distance from each designated path were recorded. A scoring system is defined as follows: A user is considered to achieve 100 % accuracy (i.e. 100 marks) if and only if he or she can position the dissector to all designated positions. For every mark outside a designated position or any missed position, 10 marks were to be deducted.

4 Results

Figure 5 shows a typical recording of EMG of a user. For all subjects, the pattern can be clearly identified for each posture. A cross validation of the classifier in the training session achieved an overall accuracy of 99.2 ± 0.3 % for all subjects. The accuracy is considered to be sufficient to achieve a robust control during the testing session.

Fig. 5.
figure 5

Typical recording of the 4-channel EMG of a user

Full size image

Table 2 reports the time of completion of each task using the motion controller and the wearable sensory sheath.

Table 2. Time of completion of each task in minutes using (a) the motion controller and (b) the wearable sensory sheath.
Full size table

Figure 6 shows the marking results for positioning the dissector at specific locations. The average marks attained by all subjects to complete the task using the motion controller and the wearable sheath were 83 ± 15 and 83 ± 14 respectively. The average marks of lifting a weight to specified position using the traditional motion controller and the wearable sheath were both 100.

Fig. 6.
figure 6

Positioning accuracy of the dissector of all subjects

Full size image

5 Discussion

The average times of completing the task on positioning the dissector using the motion controller and the wearable controller were 2 min and 4 min respectively. The average times for lifting a weight from a specified position to another by either control interface were similar. On the other hand, the time required by the wearable sheath as the controller is longer than the regular motion controller. The results suggested that it is possible to use wearable sensors to control surgical robots with similar precision as regular motion controllers. Two major factors are needed for further considerations: (1) the relative motion between wrist motion and forearm rotation; and (2) the control habit of users.

5.1 Relative Motion Between Wrist Motion and Forearm Rotation

The current design uses the user’s EMG signal to control the opening/closure of the gripper and the elevation/lowering of the dissector hinge joint by the wrist flexion/extension postures. Meanwhile, the left/right motions of the continuum section of the robotic arm were controlled by the rotation of the forearm. The two sets of movements induced cross-talks and can be a reason for the difficulties in the control using the wearable sheath. Therefore, the motion mapping of the datasets must be further refined, especially in understanding the correlation of the signals in different situations. In addition, increasing the number of EMG electrodes can further improve the precision.

5.2 Control Habit of Users

The traditional motion controller is designed with a one-to-one mapping, which is more definite. Although the control via the wearable sensory sheath provides a user-specific control method by analyzing the motion of the operator’s arms, the subject is required to learn a designed dataset. He is required to familiarize himself in controlling his muscles with the same EMG pattern so that it can be repeatedly produced when he controls the robot. Nevertheless, the wearable controller is designed by a user-specific model and therefore, it has the potential to be adaptive to different users’ behaviour if properly calibrated and setup.

6 Conclusion

In this study, we demonstrated a novel user-specific control interface in controlling a surgical robot using wearable sensors. We used e-textile electrodes instead of gel-like electrodes to collect EMG such that the sensory sheath is reusable. This arrangement can be developed into a new way of human computer interaction. When compared to traditional joystick-like motion controller, the wearable controller is able to achieve similar accuracy; however, the time of completing the same task is longer than the time required by the regular motion controller. Since human beings are used to control things through push buttons and joysticks rather than controlling things virtually, they may take a longer time to adapt to this approach. Nevertheless, the new approach allows users to move their hands freely and flexibly. The control method is also user-specific.