Skip to main content
SpringerLink
Log in

Human-robot shared control system based on 3D point cloud and teleoperation

  • Article
  • Published:
Science China Technological Sciences Aims and scope Submit manuscript

Abstract

Owing to the constraints of unstructured environments, it is difficult to ensure safe, accurate, and smooth completion of tasks using autonomous robots. Moreover, for small-batch and customized tasks, autonomous operation requires path planning for each task, thus reducing efficiency. We propose a human-robot shared control system based on a 3D point cloud and teleoperation for a robot to assist human operators in the performance of dangerous and cumbersome tasks. The system leverages the operator’s skills and experience to deal with emergencies and perform online error correction. In this framework, a depth camera acquires the 3D point cloud of the target object to automatically adjust the end-effector orientation. The operator controls the manipulator trajectory through a teleoperation device. The force exerted by the manipulator on the object is automatically adjusted by the robot, thus reducing the workload for the operator and improving the efficiency of task execution. In addition, hybrid force/motion control is used to decouple teleoperation from force control to ensure that force and position regulation will not interfere with each other. The proposed framework was validated using the ELITE robot to perform a force control scanning task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Refefences

  1. Yang C, Luo J, Liu C, et al. Haptics electromyography perception and learning enhanced intelligence for teleoperated robot. IEEE Trans Automat Sci Eng, 2019, 16: 1512–1521

    Article  Google Scholar 

  2. Yang C, Luo J, Pan Y, et al. Personalized variable gain control with tremor attenuation for robot teleoperation. IEEE Trans Syst Man Cybern Syst, 2018, 48: 1759–1770

    Article  Google Scholar 

  3. Sheridan T B. Space teleoperation through time delay: Review and prognosis. IEEE Trans Robot Automat, 1993, 9: 592–606

    Article  Google Scholar 

  4. Li Y, Tee K P, Ge S S, et al. Building Companionship Through Human-Robot Collaboration. Cham: Springer International Publishing, 2013. 1–7

    Google Scholar 

  5. Kong K, Bae J, Tomizuka M. A compact rotary series elastic actuator for human assistive systems. IEEE ASME Trans Mechatron, 2012, 17: 288–297

    Article  Google Scholar 

  6. Daniel R. Telerobotics, automation and human supervisory control. Automatica, 1994, 30: 1072–1073

    Article  Google Scholar 

  7. Li Y, Tee K P, Chan W L, et al. Continuous role adaptation for human-robot shared control. IEEE Trans Robot, 2015, 31: 672–681

    Article  Google Scholar 

  8. Wang J, Wang W, Hou Z G. Toward improving engagement in neural rehabilitation: Attention enhancement based on brain-computer interface and audiovisual feedback. IEEE Trans Cogn Dev Syst, 2020, 12: 787–796

    Article  Google Scholar 

  9. Wang C, Peng L, Hou Z G, et al. Quantitative assessment of upper-limb motor function for post-stroke rehabilitation based on motor synergy analysis and multi-modality fusion. IEEE Trans Neural Syst Rehabil Eng, 2020, 28: 943–952

    Article  Google Scholar 

  10. Luo L, Peng L, Wang C, et al. A greedy assist-as-needed controller for upper limb rehabilitation. IEEE Trans Neural Netw Learn Syst, 2019, 30: 3433–3443

    Article  Google Scholar 

  11. Liu Y, Nejat G. Robotic urban search and rescue: A survey from the control perspective. J Intell Robot Syst, 2013, 72: 147–165

    Article  Google Scholar 

  12. Hirche S, Buss M. Human-oriented control for haptic teleoperation. Proc IEEE, 2012, 100: 623–647

    Article  Google Scholar 

  13. Yang C, Wang X, Cheng L, et al. Neural-learning-based telerobot control with guaranteed performance. IEEE Trans Cybern, 2017, 47: 3148–3159

    Article  Google Scholar 

  14. Yang C, Wang X, Li Z, et al. Teleoperation control based on combination of wave variable and neural networks. IEEE Trans Syst Man Cybern: Syst, 2017, 47: 2125–2136

    Article  Google Scholar 

  15. Liang Z, Guo Y, Feng Y, et al. Stereo matching using multi-level cost volume and multi-scale feature constancy. IEEE Trans Pattern Anal Mach Intell, 2021, 43: 300–315

    Article  Google Scholar 

  16. Guo Y, Sohel F, Bennamoun M, et al. Rotational projection statistics for 3D local surface description and object recognition. Int J Comput Vis, 2013, 105: 63–86

    Article  MathSciNet  MATH  Google Scholar 

  17. Guo Y, Bennamoun M, Sohel F, et al. 3D object recognition in cluttered scenes with local surface features: A survey. IEEE Trans Pattern Anal Machine Intell, 2014, 36: 2270–2287

    Article  Google Scholar 

  18. Li M, Zhang Y. 3D point cloud labeling tool for driving automatically. In: Proceedings of the 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). Auckland, 2020. 1666–1672

  19. Chen X, Chen C H. Model-based point cloud alignment with principle component analysis for robot welding. In: Proceedings of the 2017 International Conference on Advanced Robotics and Intelligent Systems (ARIS). Taipei, 2017. 83–87

  20. Lin H I, Cong M N. Inference of 6-DOF robot grasps using point cloud data. In: Proceedings of the 2019 19th International Conference on Control, Automation and Systems (ICCAS). Jeju, 2019. 944–948

  21. Xiao D, Lian C, Deng H, et al. Estimating reference bony shape models for orthognathic surgical planning using 3D point-cloud deep learning. IEEE J Biomed Health Inform, 2021, 25: 2958–2966

    Article  Google Scholar 

  22. Chen X, Ma H, Wan J, et al. Multi-view 3D object detection network for autonomous driving. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hawaii, 2017. 6526–6534

  23. Rusu R B, Cousins S. 3D is here: Point cloud library (PCL). In: Proceedings of the 2011 IEEE International Conference on Robotics and Automation. Shanghai, 2011. 1–4

  24. Hsieh C T. An efficient development of 3D surface registration by point cloud library (PCL). In: Proceedings of the 2012 International Symposium on Intelligent Signal Processing and Communications Systems. Tamsui, 2012. 729–734

  25. Wang N, Chen C, Nuovo A D. A framework of hybrid force/motion skills learning for robots. IEEE Trans Cogn Dev Syst, 2021, 13: 162–170

    Article  Google Scholar 

  26. Kong L, He W, Yang C, et al. Robust neurooptimal control for a robot via adaptive dynamic programming. IEEE Trans Neural Netw Learn Syst, 2021, 32: 2584–2594

    Article  MathSciNet  Google Scholar 

  27. He W, Chen Y, Yin Z. Adaptive neural network control of an uncertain robot with full-state constraints. IEEE Trans Cybern, 2016, 46: 620–629

    Article  Google Scholar 

  28. He W, David A O, Yin Z, et al. Neural network control of a robotic manipulator with input deadzone and output constraint. IEEE Trans Syst Man Cybern: Syst, 2016, 46: 759–770

    Article  Google Scholar 

  29. Yang C, Peng G, Cheng L, et al. Force sensorless admittance control for teleoperation of uncertain robot manipulator using neural networks. IEEE Trans Syst Man Cybern: Syst, 2021, 51: 3282–3292

    Article  Google Scholar 

  30. Huang H, Yang C, Chen C L P. Optimal robot-environment interaction under broad fuzzy neural adaptive control. IEEE Trans Cybern, 2021, 51: 3824–3835

    Article  Google Scholar 

  31. Peng G, Yang C, He W, et al. Force sensorless admittance control with neural learning for robots with actuator saturation. IEEE Trans Ind Electron, 2020, 67: 3138–3148

    Article  Google Scholar 

  32. Yang C, Peng G, Li Y, et al. Neural networks enhanced adaptive admittance control of optimized robot-environment interaction. IEEE Trans Cybern, 2019, 49: 2568–2579

    Article  Google Scholar 

  33. Peng G, Chen C P, Yang C. Robust admittance control of optimized robot-environment interaction using reference adaptation. IEEE Trans Neural Networks Learn Syst, 2022, doi: https://doi.org/10.1109/TNNLS.2021.3131261

  34. Peng G, Chen C L P, Yang C. Neural networks enhanced optimal admittance control of robot-environment interaction using reinforcement learning. IEEE Trans Neural Networks Learn Syst, 2022, 33: 4551–4561

    Article  MathSciNet  Google Scholar 

  35. Peng G, Chen C L P, He W, et al. Neural-learning-based force sensor-less admittance control for robots with input deadzone. IEEE Trans Ind Electron, 2021, 68: 5184–5196

    Article  Google Scholar 

  36. Lu Z, Wang N, Yang C. A novel iterative identification based on the optimised topology for common state monitoring in wireless sensor networks. Int J Syst Sci, 2022, 53: 25–39

    Article  MathSciNet  MATH  Google Scholar 

  37. Tsai R Y, Lenz R K. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans Robot Automat, 1989, 5: 345–358

    Article  Google Scholar 

  38. Conti F, Khatib O. Spanning large workspaces using small haptic devices. In: Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Tele-operator Systems. World Haptics Conference, Pisa, 2005. 183–188

  39. Conti F, Park J, Khatib O. Interface design and control strategies for a robot assisted ultrasonic examination system. Springer Tracts Adv Robot, 2014, 79: 97–113

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510641, China

    ChenGuang Yang, Ying Zhang & GuanYi Zhao

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Long Cheng

Authors
  1. ChenGuang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. GuanYi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Long Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ChenGuang Yang.

Additional information

This work was supported by the National Natural Science Foundation of China (NSFC) (Grant No. U20A20200), the Major Research (Grant No. 92148204), the Guangdong Basic and Applied Basic Research Foundation (Grant Nos. 2019B1515120076 and 2020B1515120054), and the Industrial Key Technologies R&D Program of Foshan (Grant Nos. 2020001006308 and 2020001006496).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, C., Zhang, Y., Zhao, G. et al. Human-robot shared control system based on 3D point cloud and teleoperation. Sci. China Technol. Sci. 66, 2406–2414 (2023). https://doi.org/10.1007/s11431-022-2205-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11431-022-2205-9

Keywords

Search

Navigation