Robotic dexterous manipulation: from tele-operation to autonomous learning and adaptive control

1 CITEC/Bielefeld University, Bielefeld, Germany 2 CNRS, Montpellier, France 3 Bristol Robotics Laboratory, UWE, Bristol, UK 4 The Chinese University of Hong Kong, Hong Kong, China 2. Recent progresses have also been seen on humaninspired robotic manipulation. For such methods, human’s “intelligence” is recorded and analysed from neuroscience and neuro-adaptive and control aspects, then transferred to robot platforms quickly to implement complex tasks. The main drawback of these approaches is that it is difficult to evaluate/predict the system performance with existing control analysis tools and thus their design is usually empirical and case dependent/sensitive. 3. Observations also note that sensory-based manipulation is exploiting rich sensor feedback (vision, tactile, biological etc.) in manipulation theory and consequently the closed loop control systems are enhanced and adaptive controllers can be designed to improve the robustness of manipulation. 4. Finally, AR, VR and haptic rendering technologies enable humans to perceive the remote scenario in an immersive way, and this greatly facilitates the introduction of human’s intelligence into robotic manipulation and turns out to be a fairly practical approach. However, along this line, safe and reliable haptic technical system development, stable and transparent bilateral perception and control for human–robot systems still stand as big challenges to tackle for researchers in this domain.


Introduction
Manipulation is one essential capability that enables robots to interact with and change the world. It is very important to understand the principles on how dexterous manipulation can be achieved from theoretical and technical aspects. From this special issue, we hope that it can bring to readers the latest progresses (or studies) on.
• How the complex robotic systems have been developed for solving complex manipulation tasks • How the manipulation complexity/intelligence is solved by model-based, data-driven or resorted to human intelligence approaches • How dexterous manipulation is deployed in the real-world applications, e.g. in industry, medical, service and agriculture etc. domains How to endow robots with high-level manipulation intelligence is the lifelong mission of many roboticists. In recent years, "AI powered robots" have become one slogan everyone knows. Scientists are working towards the goal of achieving manipulation intelligence from four aspects. 2. Recent progresses have also been seen on humaninspired robotic manipulation. For such methods, human's "intelligence" is recorded and analysed from neuroscience and neuro-adaptive and control aspects, then transferred to robot platforms quickly to implement complex tasks. The main drawback of these approaches is that it is difficult to evaluate/predict the system performance with existing control analysis tools and thus their design is usually empirical and case dependent/sensitive. 3. Observations also note that sensory-based manipulation is exploiting rich sensor feedback (vision, tactile, biological etc.) in manipulation theory and consequently the closed loop control systems are enhanced and adaptive controllers can be designed to improve the robustness of manipulation. 4. Finally, AR, VR and haptic rendering technologies enable humans to perceive the remote scenario in an immersive way, and this greatly facilitates the introduction of human's intelligence into robotic manipulation and turns out to be a fairly practical approach. However, along this line, safe and reliable haptic technical system development, stable and transparent bilateral perception and control for human-robot systems still stand as big challenges to tackle for researchers in this domain.
We believe now it is a good time to bring these all four branches into one special issue to.
• understand and define the state of the art of dexterous manipulation from fundamental principle and applications aspects • get to know what complex manipulation have been solved by such methods • find the common scientific problems/challenges for these four approaches and merge the branches to solve more complex tasks which cannot be dealt with by a single method.
In this special issue, we solicited original papers from different domains, e.g. perception and feature extraction, manipulation planning, skill's imitation and adaptive control. After a strict peer-review process, 14 papers were accepted for publication in this special issue.
From the direction of perception and feature extraction, Rong et al. proposed a method for peduncle cutting point localization and pose estimation in automating the harvesting. YOLOv4-Tiny detector and Neural Network were used to detect and segment the close-up distance, and a geometric model was established to estimate the pose with high accuracy. Qi et al. presented a new CbCr-I component Gaussian mixture model (GMM) to detect the hand's skin region. Based the detection, multiclass support vector machine classifier was utilized to recognize the hand posture. The results showed the effectiveness of the proposed approach compared with other methods. Considering the missing of depth information via the two-dimensional images acquired by endoscopy, Li et al. presented a method to reconstruct a 3D model of soft tissues from image sequences acquired from a robotic camera holder. In this algorithm, a sparse reconstruction module based on the SIFT and SURF features was designed, and a multilevel feature matching strategy was proposed to improve the algorithm efficiency. A collaborative manipulator with a mounted camera mimic an assistant surgeon holding an endoscope to show a 3D reconstruction of soft tissue by the proposed method.
Given the environment perception, manipulation planning is required before the robot implement the task. Veeramani and Muthuswamy presented their multiple heterogeneous robots planning approach. The offline planner followed the hybrid type decentralized planning strategy and complex manipulation was formalized as several sub-problems which mathematic representation was modelled. The proposed approach coordinated all three robots (one Comau and two SwarmItFIX) in sheet metal milling process. Liu et al. investigated the safety-critical motion planning and control problem to balance robotic safety against manipulation performance. The balance mechanism was modelled as a partially observable Markov decision process and executed as belief tree planning. The proposed approach was evaluated in the door-closing scenario and the robot took the reactive plan and control to protect itself avoid the risk.
One important way that a robot acquires manipulation skills is imitation. In this case, the human behaviour will be tracked by either wearable devices or external tracking systems. Chi et al. developed a programming by demonstration robot which could help researchers to capture such data in the elderly care tasks. Employing the wearable device, Liu et al. used sEMG signals to capture and model the human lower limb intention. The main contribution of this work was that authors proposed and proved an innovative projected recurrent neural network (PRNN) model, which could relax the prerequisite condition of the convex function required in the traditional RNN model. Wang et al. employed VR setup to track human's arm movement to model the approaching intention. The trained intention model was used in the context of shared autonomy to teleoperate one robot arm approaching one goal given the multiple-goals scenario. Goal prediction was done according to the similarity metrics between user's short-term movements and the learned user model. The results showed that the approach performed well in user goal prediction and was able to respond quickly to dynamic changing of the user's goals. Instead of one arm learning, Lu et al. were working on the human's skill learn by DMP and applying it to dual-arm manipulation scenario.
To transfer the learned behaviour to robot platforms, adaptive control is needed which can online fine tuning the robot's motion to respond to the changed environment. Dong et al. proposed a scenario-oriented grasp controller. Authors used the valid grasps for a parallel-jaw gripper as vectors in a two-dimensional (2D) image and detected them with a fully convolutional network that simultaneously estimates the vectors' origins and directions. The detected vectors were then converted to 6 degree-of-freedom (6-DOF) grasps with a tailored strategy. Su et al. presented a novel control framework to incorporate model predictive control (MPC) with the fuzzy approximation. In the framework, the fuzzy approximation was mainly to manage the kinematic uncertainties existing in the MPC control. Simulations were performed and analysed to validate the proposed algorithm. The adaptive mechanism is also required in the outdoors robotic exploration. e.g. Cui et al. studied a fault-tolerant motion planning and generation method for quadruped robots while the joints were locked. The proposed fault-tolerant method was applicable to construct a quasi-static wholebody controller, and it did not require additional operations and constraints of the fault leg. The experiments were implemented on the three joint lock failure scenarios for quadruped robots. The left two contributions were related to teleoperation. Fu et al. were developing a virtual teleoperation environment. The research focus was-contact model. A new contact model was proposed to be applicable in various materials, which included both the Kelvin-Voigt model (KVM) and Hunt-Crossley model (HCM). An extra parameter was set in the model to express the capacity of continuous switching between KVM and HCM, whose rationality was proved based on the energy loss. The model's continuous switching was verified with ideal simulation, and the model parameters were continuously changed without jumpy switch error. Gao et al. focused on the bilateral teleoperation task. In this work, the authors proposed an invertible mapping approach to realize teleoperation through online motion mapping by taking into account the locations of objects or tools in manipulation skills. The proposed approach could generate trajectories in an online manner to adapt to moving objects, where impedance controllers allow the user to exploit the haptic feedback to teleoperate the robot. Teleoperation experiments of pick-and-place tasks and valve turning tasks were carried out with two 7-axis torque-controlled Panda robots.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.