Introduction

Wheelchair mounted robotic arm (WMRA) is a typical assistive robot to help the elderly and disabled to take care of themselves [8, 13, 28, 29]. WMRA is welcomed widely because it can greatly expand the activity space of the users by integrating the mobility of the wheelchair with the flexible manipulation capability of the robotic arm. However, due to the physical or cognitive deficiencies of the elderly and disabled, it is difficult or impossible for them to flexibly control the WMRA, and they still wish the robot can follow their instructions to semi-autonomously complete various tasks in the home environment [1].

Compared with the structured environment faced by industrial robots, the daily home environment faced by WMRA is a typical unstructured environment, full of various dynamic and random events. Additionally, the tasks to be completed are diverse, and the size or the position of the target object often change. In this situation, the traditional programming approach which is designed for the structured environment is no longer suitable to deal with the enormous daily living tasks. Therefore, the programming by demonstration (PbD) approach (or learning from demonstration, imitation learning) [2, 4, 10, 11, 22] was proposed by researchers to solve this problem. The PbD approach allows the robots to learn relative motion skills directly from the offered demonstration cases. Based on the PbD approach, many skill learning approaches, such as Dynamic Movement Primitive [12, 26, 30], Gaussian Mixture Model and Gaussian Mixture Regression [5, 6, 20], have also been developed to learn related skills and accomplish similar tasks in new environment autonomously. The learned skills can also be integrated with the tactile sensing system [17, 19] to complete related tasks more accurately and efficiently. Additionally, the learned movement skills can be used to build a related skill library and reused flexibly to accomplish complex, multi-step tasks with the help of finite state machine [18, 22]. Whereas, how to easily and quickly record the demonstration information of various daily living tasks is still the most urgent problem to be solved. This is the precondition to carry out a series of follow-up related studies.

The traditional demonstration information recording approach generally directly records the related motion trajectories information of the end effector during the task demonstration process, which is called the direct demonstration information recording approach. Although this approach can record the demonstration information of related tasks, the robot cannot effectively distinguish the effective operations and mis-operations during the task demonstration process. Recorded demonstration information also contains a lot of redundant operation information or mis-operations. In actual operation, even demonstrators who are proficient in controlling handles often need to make multiple attempts to obtain relatively satisfactory demonstration trajectories. The mentioned problems have brought great mental burden and operational difficulty to the demonstrators.

To solve the above questions, this paper proposes the key-point-based PbD recording approach to obtain relative demonstration motion information. The main contributions of this paper can be summarized as follows: (1) Compared with the traditional direct demonstration information recording approach, the key-point-based PbD approach can not only improve the quality of the acquired demonstration trajectories, but also significantly reduce the difficulty and mental burden of the demonstrator; and (2) more importantly, the demonstration motion information acquired by the proposed approach can make sure the robot effectively completes the relative household task, avoiding the task reproduction failure problem easily caused by too many mis-operations in the past. The rest of paper is organized as follows. First, the JACO robotic arm is selected to build the WMRA robot based on the identification and systematic analysis of the daily living tasks of the assistive robot. Next, the key-point-based PbD recording approach is especially proposed for the JACO robotic arm to quickly obtain the demonstration information and the corresponding interface of the proposed approach is designed to simplify the whole operation process. Then, the demonstration evaluation approach is proposed to comparatively evaluate the key-point-based PbD recording approach and the traditional direct recording approach. Finally, three typical tasks are carried out to validate the proposed approach. This study can not only facilitate the end-users to teach related tasks and quickly acquire the corresponding demonstration information, but also lay a good foundation for the assistive robot to learn the motion skills and reproduce the daily tasks that existed in the daily home environment.

Analysis on daily living tasks of assistive robot

Requirements of defining tasks of an assistive robot

Proper identification and classification of the daily living tasks which are in need of WMRA’s assistance are critical to improve the WMRA’s performance in executing semi-autonomous tasks with end-users. Over-definition of daily tasks can result in the system being redundant and complex, even missing the best solution to solve the current task. Meanwhile, under-definition of daily tasks can directly lead the robots to fail to meet the users’ basic needs of daily life. So, this paper proposes the following factors that should be mainly considered while defining the tasks of assistive robots.

  1. 1.

    Operability. Operability is the primary requirement to be taken into consideration when defining tasks. With the limitation of the robotic arm’s motion performance, successful completion of the tasks requires that the tasks defined should be within the robotic arm’s reachable space and operation ability.

  2. 2.

    Priority. The best solution to the defined tasks must be adapted to the assistive robots. With the continuous promotion of smart home technology, many tasks that originally need robotic assistance can also be completed efficiently and conveniently with smart home products, such as switching on and off the indoor light. Therefore, assistive robots should give priority to the tasks that cannot be replaced by other intelligent products.

  3. 3.

    Uniqueness. The tasks defined should be unique. This requirement mainly considers the purpose level of the tasks. For instance, taking a water cup from a table or a cupboard is the same kind of task. The defined task is a general term for a class of tasks with similar characteristics, not just limited to specified tasks. Uniqueness can avoid the redefinition of the required tasks and reduce the complexity of the assistive robot system.

Daily living tasks of assistive robot

According to the abovementioned requirements that should be considered in the definition of tasks, the daily living tasks of assistive robots as shown in Table 1 are determined on the summary of the relative previous studies [3, 9, 15, 23, 25, 27]. The specific contents and primary requirements to complete the task are also elaborated in Table 1.

Table 1 Daily living tasks of assistive robot
Full size table

In the daily living tasks listed in Table 1, task 1 involves the grasping of various objects in different environments. The targets are objects of various shapes or containers filled with liquids, and the robotic arm is required to maintain steady motion and smooth trajectory while performing the tasks. Tasks 2 and 5 are tasks related to users themselves. The primary requirements of these two tasks are to ensure the safety of users and steady movement in turn during the execution of the tasks. Tasks 3 and 4 are related to the structured objects in the environment. The motions of the robotic arm are restricted by the surrounding environment. These two tasks require precise motions of the robotic arm to meet the constraints of the surrounding environment.

Task classification and its typical task

Based on the above analysis of the tasks, this paper divides the daily living tasks into three categories according to the different targets and constraints of the tasks. The detailed categories are shown below. (1) Tasks related to “free objects”. The free objects are relative to the objects in a structured environment. It usually can move along more than one direction; (2) Tasks related to “structured environment”. The motion of the object is constrained by the surrounding environment. Usually, the object can only move along the specified direction; (3) Tasks related to “users themselves”. These tasks usually have close contact with the users themselves. Safety and stability are the primary requirements for accomplishing these tasks. Based on this consideration, this category of tasks is treated as a separate category. “Holding water glass”, “Opening door”, and “Eating” are chosen as the respective typical task of the above three categories for the following study of the learning from demonstration approach and verification of the demonstration interface. It should be noted that the tasks related to “users themselves” are generally comprehensive tasks that include the other two types of tasks. For example, the task of grasping a cup from the table for drinking water contains the following two tasks both “free objects” and “users themselves”. The task classification and its typical task are shown in Table 2 in detail.

Table 2 Task classification and its typical task
Full size table

Demonstration approach for WMRA

Construction of the WMRA-assistive robot

WMRA is a kind of assistive robot which is constituted by a multi-DOF, lightweight robotic arm with an electric wheelchair integrated with other sensors to assist the elderly and disabled to live independently without any help. The robotic arm is the core component of WMRA. Especially, based on the analysis of the daily living tasks of the assistive robot, the safety of users, the difficulty of manipulation, the accuracy of the movement, and the weight and appearance of the manipulator are all key factors should be considered when selecting the commercialized robotic arm. For the above reasons, the JACO robotic arm produced by Kinova Company of Canada [7, 21, 24] is selected to build the assistive robot in this paper.

The JACO robotic arm is a 6-DOF serial robotic arm. It is made of carbon fiber with its weight of only 5.3 kg, fully meeting the requirements of lightweight. The six joints and three fingers of JACO are directly driven by DC servo motors. The JACO robotic arm with three fingers can easily complete any dexterous movement. The maximum load is 2.5 kg and its reachable range is 90 cm, which also meets the requirements of most daily living tasks [24]. The lightweight robotic arm’s low moment of inertia and JACO’s unique safety protection area during controlling can ensure the safety of users. The 25 W average power of JACO can ensure its better endurance while sharing the same power supply with the electric wheelchair. At the same time, the harmonic reducers are equipped in each joint of the JACO robotic arm to ensure the motion accuracy of the robotic arm, and the torque sensor, temperature sensor, and current sensors are also equipped to monitor the movement state of the robotic arm in real-time.

Additionally, the electric wheelchair produced by Vermeiren Company of China and the Xtion camera produced by Asus is separately selected here. The WMRA constructed in this paper is shown in Fig. 1.

Fig. 1
figure 1

Physical photo of WMRA-assistive robot

Full size image

Selection of JACO demonstration approach

The demonstration approach is the platform for communications between the users and the robot. The selection of demonstration approach not only affects the difficulty of the demonstrator’s operation, but also affects the collection and processing of motion information. For the JACO robotic arm, there are three main approaches besides the traditional programming approach [4].

  1. 1.

    Direct recording of human motion. This approach can capture human motion information through motion tracking systems [14, 16], which is mainly based on vision exoskeleton or other wearable sensors. Then, related motion information is used to control the motion of robotic arm. Although this approach allows the user to demonstrate various actions conveniently, the robot might fail to reproduce the task due to the sensors’ errors and different structures between the robot and the user.

  2. 2.

    Kinematic demonstration. The demonstrator moves the robotic arm directly to accomplish specific actions. This approach is simple and convenient to operate, and it conducts the demonstration process on the robot itself, thus avoids the joint mismatch problem caused by the structural differences between the robot and demonstrator. However, this approach requires more degrees of freedom (DOF) to achieve the precise control of the specified joint. For example, although only one degree of freedom is involved, two hands are simultaneously needed to control the motion of the robot’s elbow joint. Therefore, this approach is difficult to control the multi-DOF robotic arm.

  3. 3.

    Teleoperation mode. The demonstrator utilizes the handle or other special equipment to control the motion of the robot. With the tactile sensor, the demonstrator can also control the force accurately during the demonstration process. This approach collects the original motion directly through the robot’s own sensors. It not only avoids the motion matching problem between the robot and the demonstrator, but also allows users to manipulate the robot remotely. However, this approach requires the users to learn the operations of related teleoperation devices in advance. Additionally, the teleoperation equipment for multi-DOF robots is extremely complex.

As to the JACO robotic arm, the default operation approach mainly includes handle control, kinesthetic demonstration, and traditional programming approach. Although the direct recording of human motion is simple and convenient, it is not considered in this study because it cannot ensure the effective completion of tasks. Considering the application situation of the assistive robot, the characteristics of the application crowd, and the advantages or disadvantages of the related demonstration approaches, the handle control approach is chosen as the related demonstration approach. This approach is simple to understand and operate. It has solved the problem of joint mismatching, and avoided the re-learning of related operation devices. This approach is especially suitable for non-professionals to teach robots in the daily home environment.

Demonstration information recording approach

The traditional demonstration information recording approach generally obtains the task’s relative information by directly storing the motion trajectories, while the effective operations and wrong operations cannot be distinguished appropriately during the demonstration process. Therefore, this approach requires the accurate operations of the related tasks to avoid the mis-operations in the task reproduction or failures in achieving the operational goals. In the real scene, satisfactory demonstration operations might require many trials to be carried out, while erroneous operations still exist in the obtained demonstration trajectories. For example, a satisfactory grasping posture can only be obtained through continuous adjustments between the robotic arm and the target object’s position. These repetitive adjustments bring more mental burden and operation difficulty to the users.

Therefore, the key-point-based PbD recording approach is proposed in this paper to solve the above problems that existed in the demonstration information recording approaches. The key-point-based PbD recording approach only requires the user to selectively record the key points in the demonstration process. The key points are selected according to the function switches of the handle, position constraints of a specific task, and their own usage habits. More importantly, the user can select the appropriate key points after multiple adjustments without any effect on the recorded task demonstration information. This approach can greatly simplify the demonstration process and simultaneously reduce the operational burden. Moreover, it can make sure the robot can successfully utilize the demonstration information to accomplish related daily living tasks even during the learning process, thus facilitate the popularization of assistive robots.

The key-point-based PbD recording approach of JACO robotic arm includes two parts: the learning from demonstration stage and the motion reproduction stage. (1) In the learning from demonstration stage, the robotic arm moves from the initial position under the control instructions of the handle, so as to accomplish any specific task. During this process, they record the key points in the task demonstration process as well as the relevant gesture information and gripper’s information; (2) In the motion reproduction stage, the JACO robotic arm reads the collected key points information in turn and completes the path planning according to the built-in interpolation algorithm. Then, the corresponding path plan is converted into the control signal that drives the robotic arm and reproduces the learned task. During the task reproduction process, the angles of each joint, Cartesian Coordinates of the end effector, and the gripper’s information can all be obtained as the original demonstration trajectory information according to the preset sampling time interval. The specific demonstration approach flowchart of JACO robotic arm is shown in Fig. 2.

Fig. 2
figure 2

Key-point-based PbD recording approach

Full size image

Design of demonstration interface

To facilitate the operation process of the key-point-based PbD recording approach and reduce the manipulation burden of the demonstrator, a special demonstration interface is developed in this paper by using the library function of the JACO robotic arm and some other related functions. In the development of this demonstration interface, the following functions are mainly used: “MyGetCartesianPosition’’ function is used to obtain Cartesian coordinate of the robotic arm’s end, “MyGetCartesianCommand” function is used to obtain Cartesian command information of robotic arm, “MySendAdvanceTrajectory” function is used to input the relevant Cartesian coordinate and control the robotic arm to move. The JACO interface developed in this paper is shown in Fig. 3 in detail.

Fig. 3
figure 3

Demonstration interface of JACO robotic arm

Full size image

This demonstration interface mainly includes the following five parts.

  1. 1.

    Demonstration instructions part. This part is the core part of the interface, and mainly realizes the learning and reproduction of related tasks. Especially, when JACO robotic arm moves to the key point position, the key point information can be recorded by clicking the “Acquiring demonstration key point” button. The relevant cartesian coordinate information and the gripper’s state information are simultaneously displayed in the state monitoring part. When the whole task is completed, the JACO robotic arm will return to the default position by clicking the “Reset” button. When the “Output demonstration trajectory” button is clicked, the JACO robotic arm will pass through the collected key points in turn and perform the relevant movement with the built-in interpolation algorithm. Besides, the teaching interface can simultaneously output the number of key points in the demonstration trajectory and save them to the specified “Demo1-1.txt” file.

  2. 2.

    Parameter setting part. In this part, the interface first displays the default translation speed of JACO robotic arm, and the user can further modify the translation speed and rotation speed of the robotic arm. Besides, the user can also modify the sampling interval to control the stored trajectory points of related tasks.

  3. 3.

    System button part. This part mainly includes two buttons: “Emergency stop” button and “Exit” button. The “Emergency stop” button enables the system to stop the current operation at any time, thus eliminating the adverse consequences of mis-operations. The “Exit” button enables the system to withdraw from the current interface. In addition, this part also shows the system’s current time.

  4. 4.

    Task reproduction part. In this part, the system can directly read the saved demonstration files, thereby reproducing the stored demonstration tasks. The user only needs to give the saved file’s name, such as “Demo1-1.txt”, and click the “Task reproduction” button, then the JACO robotic arm can automatically judge the number of teaching points, read the relevant path points, perform a series of related interpolation operations, and reproduce the teaching tasks. It is worth pointing out that this instruction only reads the stored demonstration task information and does not require any demonstration operation of the task.

  5. 5.

    State monitoring part. This part is mainly used to display the JACO robotic arm’s Cartesian coordinate information and the gripper’s state information when collecting key points of the demonstration process and carrying out the task reproduction process.

With the help of this demonstration interface, users can obtain the demonstration information of tasks by only clicking the corresponding buttons and inputting relevant information. The convenient operation enables even non-professional programmers to acquire the demonstration information of specific tasks.

Evaluation of demonstration information recording approach

The demonstration information recording approach not only directly affects the operation complexity of the demonstrator, but also seriously affects the quality of the acquired demonstration trajectories and the popularization use of the assistive robot. As to the evaluation of demonstration information recording approach, this paper mainly focuses on the comparison between the proposed key-point-based PbD recording approach and the direct demonstration information recording approach, starting with the demonstration trajectories and the demonstration process respectively to carry out the relevant analysis and evaluation. Here, we assume each factor accounting for 50% influence weight. It needs to be pointed out the evaluation of the demonstration trajectories can also select the best demonstration trajectory from multiple demonstration trajectories of the same task, thereby laying the foundation for the robot to learn and generalize related tasks in a new environment.

Evaluation of demonstration trajectories

A fine demonstration trajectory is a prerequisite for the successful application of learning from demonstration approach. It is also the basis for the assistive robot to independently complete related tasks in the new environment. If a demonstration trajectory contains a large number of mis-operations or adjustment operations, it will inevitably lead to an increase in the energy consumption of the robot arm during the task demonstrating process, and seriously affect the quality and efficiency of the subsequent task reproduction, sometimes even cause the robot cannot successfully reproduce the demonstrated tasks.

Considering the fact that many tasks themselves including a series of tiny operation actions, such as mixing tasks, writing tasks, et al., and the randomness of the operation actions when the user manipulates the robot to accomplish the task, the traditional trajectory evaluation approach is no longer suitable in this situation. The quality of the trajectory has to be indirectly evaluated from other aspects of the demonstrating process. At the same time, considering that the JACO robotic arm of the assistive robot shares the same power source with the electric wheelchair, it is necessary for the robotic arm to consume as little energy as possible during the process of completing various tasks. Therefore, the paper seriously considers the above factors and proposes to evaluate the quality of demonstration trajectories from the aspect of the lowest total energy consumption of the motors at each joint of the robotic arm.

During a period of time T, the JACO robotic arm starts to move continuously from the initial position and completes a specified task under the control of the user. Then, the total energy consumption of the robotic arm during this process is equal to the total sum of the nine motors at the joints and fingers. The energy consumption can be calculated by formula (1).

$$ E = \sum\limits_{i = 1}^{9} {\int_{0}^{T} {P_{i} (t){\text{d}}t = } } \sum\limits_{i = 1}^{9} {\sum\limits_{j = 1}^{N} {UI_{i} (j)\Delta t} } \,, $$
(1)

where E is the total energy consumption of the JACO robotic arm; \(P_{i}\) is the power of the i-th motor; \(U\) is the voltage value of the motor, which is 24 V in this situation; \(I_{i}\) is the current value of the i-th motor, which can be collected directly during the demonstration process; \(\Delta t\) is the sampling time interval of the system, which can be set by the actual user of WMRA; N is the total sampling points number contained in a demonstration trajectory.

It should be pointed out that due to the differences in the usage habits of the demonstrators, the obtained demonstration trajectory is not necessarily the optimal motion path for completing the task in theoretical calculations, but it is the motion trajectory which is most in line with the user’s usage habits. At the same time, due to the uncertainty of human daily actions, even if the same user strictly follows the same action sequence instruction of the task for repeated teachings, the recorded multiple trajectories are only similar in shape and style to each other, but the actual motion information is still different from each other. In this paper, the multiple teaching trajectories are evaluated according to the lowest energy consumption standard, and the better trajectory can be selected from the multiple teaching trajectories that meet the user’s own usage habits for the learning and generalization of subsequent teaching tasks.

Evaluation of demonstration process

In addition to the evaluation of the quality of demonstration trajectories, it is also necessary to evaluate the demonstration process. If the demonstration information acquisition operation is cumbersome, time-consuming, and laborious, this will not only increase the user’s operating burden and mental pressure, but also severely restrict the popularization and use of the approach. For this reason, this paper mainly evaluates it from the following two aspects, the time of teaching operation and the number of times of task teaching. Here, we also assume each factor accounting for 50% influence weight.

  1. 1.

    Demonstration operation time T. The time T refers to the time it takes for the demonstrator to obtain complete and usable trajectory information of the relevant task from the beginning of the teaching. As to the key-point-based PbD recording approach, the operation time includes the time spent in the teaching phase and the reproduction phase shown in Fig. 2. Additionally, this approach can directly reproduce the acquired demonstration trajectory information to verify the validity of the trajectory information. As to the demonstration information directly recording approach, the operation time includes not only the time spent in the teaching operation process, which contains the effective operations, mis-operations or reciprocating adjustment actions in the trajectory, but also includes the time wasted in repeated teaching. Additionally, it includes the time consumed for task reproduction, so as to ensure that the recorded demonstration trajectory information is available.

  2. 2.

    Number of demonstration N. The number N refers to the demonstration times the demonstrator needs to carry out so as to obtain a complete and successful trajectory. It should be pointed out that when the robotic arm touches other objects during the demonstration process or performs multiple reciprocating adjustments and still fails to obtain a satisfactory demonstration trajectory, it needs to be re-taught. Normally, if the teacher uses the traditional demonstration information directly recording approach to obtain the demonstration information of some complex daily tasks, it often requires multiple demonstrations to obtain a satisfactory result; while the key-point-based PbD recording approach generally only needs to carry out once demonstration operation.

Robot experiments

To meet the requirements of most people’s right-hand usage habits, this paper sets the JACO robotic arm to the right-hand operation mode and fixes it on the right front of the electric wheelchair. The JACO robotic arm and the electric wheelchair share the same 24 V power supply. Meanwhile, the JACO robotic arm is connected to the computer through a data cable to complete the motion control and related motion information collection.

This paper recruits three volunteers with sound athletic ability to carry out the experiments. Three volunteers are two males and one female worked as the demonstrators. They have to complete the demonstrations of three typical household tasks of holding water glass, eating and opening cupboard doors and are required to obtain the related demonstration information with the key-point-based PbD recording approach and the demonstration information directly recording approach respectively. Finally, the results of the two approaches are comprehensively compared and evaluated from the demonstration trajectories and the demonstration process.

It should be noted that the three volunteers recruited in this paper are all right-handed, and they have never manipulated the JACO robotic arm before. Before carrying out the experiments, the three volunteers received about half an hour of training on the basic operation of the handle, and each volunteer performed about 2 h of operation practice per person, so as to lay the foundation for the follow-up experiments to be carried out. The half an hour training content did not involve any information of the experiments.

Experimental task description

Holding water glass task

This task requires the volunteer to manipulate the JACO robotic arm to take the cylindrical water cup from the initial position to the target position. Meanwhile, the demonstration motion information and the related object position information are required to record. The specific experimental scene is shown in Fig. 4. The electric wheelchair is parked and fixed in front of the table. Three positions marked with 1/2/3 are chosen on the right front of the wheelchair as the initial position. Another position marked with 0 is chosen in front of the user as the target position. Most importantly, the above four positions are all required to be within the reach of the robotic arm. The volunteers have to carry out the experiment in accordance with the same action sequence to ensure the consistency of multiple task demonstrations. It should be noted that although this type of object handling task seems simple, it is a difficult problem that the disabled and the elderly often face in their daily home life and urgently needs the assistance of robots.

Fig. 4
figure 4

Demonstration scene of holding water glass task (initial positions: 1, 2, 3; target position: 0; demonstration 1: from 1 to 0; demonstration 2: from 2 to 0; demonstration 3: from 3 to 0)

Full size image

Eating task

The volunteer has to first control the robotic arm to grab a rice scoop placed arbitrarily on the table; second, move to the designated rice bowl and scoop the porridge; third deliver it to the user’s mouth and accomplish the eating action; finally, put down it to complete the task demonstration. The specific experimental scene is shown in Fig. 5a. The position of the wheelchair remains unchanged which is the same as task 1. Three positions marked with 1/2/3 are chosen in front of the wheelchair as the spoon initial position with the requirement that the spoon handle has to be placed horizontally left and right. The rice bowl is placed at a fixed position on the left front and the position marked with 0 is chosen as the spoon placed position after the experiment. Additionally, this paper uses water instead of actual porridge to prevent food from decay.

Fig. 5
figure 5

Demonstration scene of eating task (initial positions: 1, 2, 3; target position: 0; demonstration 1: from 1 to 0; demonstration 2: from 2 to 0; demonstration 3: from 3 to 0)

Full size image

In addition, to facilitate the JACO robotic arm to grasp the rice spoon for related operations, this paper fixes the spoon handle on the side of the cylinder while keeping the spoon level, as shown in Fig. 5b. This cylindrical rice scoop structure does not require the flexible operation of the fingers and only needs to be simply held by the robotic arm, thereby avoiding the disadvantages of inflexible fingers of the under-driven fingers.

Opening door task

This task requires the user to open the common cabinet door through the robotic arm at different wheelchair parking positions. The cabinet doors are connected by fixed hinges with only simple door handles installed on the surface of the door instead of common door handles. In the specific experimental scene, the electric wheelchair faces the cabinet door and stops at three arbitrary positions marked with 1/2/3 in turn. The cabinet door is required to be in the right opening manner and the position of the door handle is within the working range of the robotic arm. The shape of this kind of door handle is generally a slender arch and the handle is installed close to the door surface. The inner surface of the handle is only about 15 mm from the door surface, which makes it difficult for the under-actuated fingers to directly grab the handle to complete related operations action.

For this reason, this paper adds an extra cylindrical pendant with a diameter of about 80 mm connected by a flexible rope to the door handle, which is shown in Fig. 6a. After this modification, the robot can indirectly complete the door opening task by grabbing the cylindrical pendant, which is shown in Fig. 6b. The pendant is cylindrical and can swing from side to side, reducing the difficulty of the grasping process. This indirect grasping approach through flexible ropes not only facilitates the robot’s grasping of the pendant, but also effectively avoids the robotic arm or the cabinet door damaged by the manipulation errors. However, this flexible connection method can easily lead to inconsistencies and pauses in the opening movement process.

Fig. 6
figure 6

Demonstration scene of the opening door

Full size image

The above three household tasks require each volunteer to use the key-point-based PbD recording approach and the demonstration information directly recording approach to carry out the related demonstration three times at each position, and to record the number of demonstrations, operation time for successfully completing the task, and the position information of the task-related objects in the robotic arm coordinate system. At the same time, during the demonstrating process, the Cartesian trajectory information of the end effector and the current information of each motor is sampled at a sampling interval of 100 ms. Here, the trajectory information of the end effector in the demonstration trajectory is expressed in the form of three-dimensional position coordinates and X–Y–Z Euler angles in the Cartesian coordinate system.

The evaluation of demonstration trajectories is done indirectly through the total energy consumption of the robot arm in the teaching process. The specifically related calculation is shown in formula 1. Since the input voltage of the motors at each joint of the robotic arm and the preset sampling time interval remains unchanged, the total energy consumed by the robotic arm in the teaching process is only related to the total current value of the motors at each joint. For this reason, this paper directly samples the current information at each joint for the evaluation of demonstration trajectories. In addition, in order to reduce the difficulty of the demonstration operation and reduce the number of operating errors, the finger movement is controlled separately when the end of the robot arm is stationary during the experiment.

Experimental results and analysis

Statistics of experimental results

According to the design of the abovementioned experimental program, three volunteers are recruited and they have to respectively use two different demonstration information acquisition approaches to carry out the experiment at three different positions in turn. At each position, each volunteer has to repeat the same experiment three times. For any task, 54 times demonstration experiments are required. Among the recorded demonstration information, the number of demonstrations, time of demonstration operation, and record of current information are mainly used to evaluate the demonstration information acquisition approach. Additionally, the Cartesian coordinate information of the end effector and the finger movement status information, and position information of task-related objects are mainly used for the learning and reproduction of demonstration tasks.

Just randomly take the holding water glass task for example. This paper only considers trajectories obtained by the same volunteer so as to eliminate the differences between different users’ operations. At each different position, only one trajectory is randomly selected from the three repeated demonstrations for comparative analysis. Moreover, demonstration trajectories obtained at three different positions marked with 1/2/3 are respectively represented as trajectory 1/2/3. These trajectories are drawn separately in Fig. 7a and b according to the direct demonstration information recording approach and key-point-based PbD recording approach.

Fig. 7
figure 7

Demonstration trajectories comparison of the holding water glass task

Full size image

Analysis of experimental results

From the above Fig. 7, it can clearly find that the trajectories obtained with the direct demonstration information recording approach cost much more time than the key-point-based PbD recording approach. The trajectories obtained by the former approach contain many fluctuations, which is clearly shown in Fig. 7a. Moreover, the trajectories obtained by the latter approach are much smoother than the former approach. All these phenomenons show that the key-point-based PbD recording approach can obtain better trajectories with less time and fluctuation. Although this paper only analyzes the trajectory shapes of the holding water glass task, it should be noted that this task is randomly selected and its experimental conclusions are universal here. So there is no need to draw the relative trajectories of the other two tasks one by one.

Additionally, to reduce the influence of the differences in the operation level of different users on the demonstration information recording approach, this paper takes the average of the experimental results of the three volunteers at the same location as the experimental data at this position and carries out the quantitative comparative analysis and discussion of the relative parameters at different positions from the evaluation of demonstration trajectories and demonstration process two aspects.

Evaluation of demonstration trajectories

The evaluation of demonstration trajectories is mainly achieved indirectly by comparing the total current information in different demonstration trajectories. The total current information comparison of the holding water glass, eating, and opening door tasks are shown in Fig. 8a–c, respectively.

Fig. 8
figure 8

Current information comparison of typical tasks

Full size image

From the above three sub-figures under the same initial experimental conditions, the total current values of the demonstration trajectories obtained by the key-point-based PbD recording approach are significantly lower than that obtained by the demonstration information directly recording approach. Especially for complex task such as eating task, the difference between the two approaches is relatively large. Taking the eating task for example, the total current values consumed at the selected three positions from left to right are 121.6A and 191.2A, 94A and 156.1A, 105.4A and 171.4A in turn. Here, the first value represents the key-point-based PbD recording approach, and the last value represents the demonstration information directly recording approach. Additionally, the last approach consumes 62.6% more total current on average than the first approach. Under the premise of completing the same demonstration task, the above phenomenon indicates that a demonstration trajectory obtained by the key-point-based PbD recording approach contains relatively few mis-operations, adjustment actions, or pauses. For these reasons, the demonstration trajectory consumes less current and the quality of the trajectory is better.

Meanwhile, this paper also carries out the t-test analysis of the two approaches at the three positions of the above three tasks one by one. The nine results of t-tests all show that the total current consumption of the key-point-based PbD recording approach is significantly lower than the value of the direct information recording approach with P < 0.05. This indicates the difference between the above two mentioned current information is mainly caused by the different demonstration information acquisition methods. In other words, the key-point-based PbD recording approach is easier to obtain a demonstration trajectory with lower total energy consumption and better quality.

In addition, the comparative analysis of current information for three repeated experiments at the same location can also filter out the volunteer’s best demonstration trajectory with the lowest energy consumption at that location, so as to lay the foundation for the subsequent tasks learning and generalization.

Evaluation and analysis of demonstration process

As to the evaluation of the demon-stration process, the comparative analysis is mainly carried out from the number of demonstration and demonstration process time. The specific experimental results and analysis are shown as follows.

(1) Number of demonstrations analysis. The number comparison results of the demonstrations for the holding water glass, eating and opening door tasks are shown in Fig. 9a–c, respectively.

Fig. 9
figure 9

Number of demonstrations comparison of typical tasks

Full size image

It can clearly draw the conclusion from the above three sub-figures that the number of demonstrations with the key-point-based PbD recording approach is less than that with the direct demonstration information recording approach. Especially for the holding water glass task and eating task, the number difference of demonstrations is very obvious.

During the demonstration processes of the above three household tasks, the number of demonstrations with the key-point-based PbD recording approach is relatively stable with the value 1 time. Only the average number of demonstrations for the holding water glass task is 1.1 times. However, the number of demonstrations with the direct demonstration information recording approach fluctuates relatively large. The demonstration times of the holding water glass task and eating task are large with the average times 1.63 times and 1.87 times, respectively. And, the demonstration times of opening door task is small with the average times 1.1 times. The reason for this phenomenon is that the direct demonstration information recording approach lacks the adjustment and pause during the demonstration process. When in face of complex and cumbersome tasks such as holding water glass and eating, the demonstrators have to frequently carry out the demonstration again due to mis-operations or excessive adjustment actions, resulting in an increase in the number of demonstrations. For example, the average numbers of demonstrations for holding water glass task and eating task are 1.76 and 1.90, respectively. However, the key-point-based PbD recording approach allows users to continuously adjust so as to select the key points and generate coherent demonstration trajectories automatically, and this approach rarely requires the user to carry out the demonstration again. For these reasons, the number of demonstrations is small and the generated trajectory is smooth.

It should be noted that in Fig. 9a, the number of demonstrations of handing water glass task with the key-point-based PbD recording approach is 1.1 times. This may be caused by the volunteer has not fully understood and familiar with this approach. In Fig. 10c, the number of demonstrations of the opening door task with the direct demonstration information recording approach is much smaller than the other two tasks with a value of 1.1 times. This is caused by the addition of auxiliary pendants connected by flexible ropes, which requires less adjustment or repeat the demonstration in the real demonstration process.

Fig. 10
figure 10

Demonstration operation time comparison of typical tasks

Full size image

(2) Demonstration operation time analysis. The demonstration operation time comparisons of holding water glass task, eating task, and opening door task are shown in Fig. 10a–c, respectively.

It can clearly draw the conclusion from the above three sub-figures that the demonstration operation time cost by holding water glass task and eating task with the direct demonstration information is significantly more than the time cost with the key-point-based PbD recording approach. As to these two tasks, the direct approach costs about 15% and 30% more time than the key-point-based approach, respectively. However, the time cost to accomplish the opening door task is just the opposite. The former saves about 20% more time than the latter.

For the holding water glass task and eating task, these kinds of household tasks are often complicated and the operation steps are cumbersome. In the actual demonstration operation process, it often requires multiple reciprocating adjustments or carries out the demonstration again to complete the designated tasks. At the same time, the key-point-based PbD recording approach can be used to easily select the key points in the task teaching process, avoid to carry out the demonstration again, and save the time of the demonstration operation. While during the demonstration process with the direct demonstration information recording approach, it often contains the time used to re-teach the task due to the operating errors, leading to a relative increase in the demonstration operation time.

As to the opening door task, the direct information recording approach takes less time because the operation of the door opening task becomes simple after the addition of auxiliary pendants and does not require excessive adjustment or re-teaching. In Fig. 10c, the number of demonstrations is about 1.1 times. The key-point-based PbD recording approach costs time for the user to pause and think are added during the teaching operation, so the actual operation time is slightly more than the direct demonstration information recording approach. This is one drawback of this approach, especially when dealing with simple tasks. However, this approach can acquire better demonstration trajectories from the analysis of the current information. The abovementioned time abnormality also reflects that the key-point-based PbD recording approach is more suitable for the acquisition of complex, cumbersome, and multi-step daily household tasks.

Conclusion

To reduce the difficulty of acquiring relative demonstration information for the WMRA built with the JACO robotic arm, this paper proposes the dedicated key-point-based PbD recording approach and corresponding evaluation approach from the demonstration trajectories and demonstration process two aspects. Additionally, it carries out three typical household tasks of “holding water glass task”, “eating task”, and “opening door task” to verify the validity of the proposed approach. From the analysis of the experimental results, it can easily conclude that compared with the direct demonstration information recording approach, the key-point-based PbD recording approach can obtain a demonstration trajectory with lower energy consumption and requires less demonstration operation time and the number of demonstrations, especially suitable for some complex, cumbersome, multi-step daily household tasks. However, when selecting the key points, the user has to frequently switch between the handle and the demonstration interface of the JACO robotic arm. It is not very friendly for one person to accomplish the control of handle and the selection of key points. To improve this drawback is also the focus of our future research.

The key-point-based PbD recording approach of WMRA proposed in this paper can not only effectively reduce the user’s operation burden and mental burden, but also offer a convenient and feasible way to acquire the demonstration information of daily living tasks. Importantly, the study lays a good foundation for the assistive robot to learn relative motion skills, especially for the demonstrated dexterous manipulation skills, and semi-autonomously accomplish complex, multi-step tasks following the user’s instruction in the daily home environment.