Intuitive Bare-Hand Teleoperation of a Robotic Manipulator Using Virtual Reality and Leap Motion

  • Inmo JangEmail author
  • Joaquin Carrasco
  • Andrew Weightman
  • Barry Lennox
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11650)


Despite various existing works on intuitive human-robot interaction (HRI) for teleoperation of robotic manipulators, to the best of our knowledge, the following research question has not been investigated yet: Can we have a teleoperated robotic manipulator that simply copies a human operator’s bare hand posture and gesture in a real-time manner without having any hand-held devices? This paper presents a novel teleoperation system that attempts to address this question. Firstly, we detail how to set up the system practically by using a Universal Robots UR5, a Robotiq 3-finger gripper, and a Leap Motion based on Unity and ROS, and describe specifically what information is communicated between each other. Furthermore, we provide the details of the ROS nodes developed for controlling the robotic arm and gripper, given the information of a human’s bare hands sensed by the Leap Motion. Then, we demonstrate our system executing a simple pick-and-place task, and discuss possible benefits and costs of this HRI concept.


Human-robot interaction Teleoperation Virtual Reality Leap Motion 


  1. 1.
    Allspaw, J., Roche, J., Lemiesz, N., Yannuzzi, M., Yanco, H.A.: Remotely teleoperating a humanoid robot to perform fine motor tasks with virtual reality. In: Waste Management Symposium (WM 2018), Phoenix, AZ (2018)Google Scholar
  2. 2.
    Cancedda, L., Cannavò, A., Garofalo, G., Lamberti, F., Montuschi, P., Paravati, G.: Mixed reality-based user interaction feedback for a hand-controlled interface targeted to robot teleoperation. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) AVR 2017. LNCS, vol. 10325, pp. 447–463. Springer, Cham (2017). Scholar
  3. 3.
    Chae, J., Jin, Y., Sung, Y., Cho, K.: Genetic algorithm-based motion estimation method using orientations and EMGs for robot controls. Sensors 18(2), 183 (2018). Scholar
  4. 4.
    Chen, S., Ma, H., Yang, C., Fu, M.: Hand gesture based robot control system using leap motion. In: Liu, H., Kubota, N., Zhu, X., Dillmann, R., Zhou, D. (eds.) ICIRA 2015. LNCS (LNAI), vol. 9244, pp. 581–591. Springer, Cham (2015). Scholar
  5. 5.
    Crick, C., Jay, G., Osentoski, S., Pitzer, B., Jenkins, O.C.: Rosbridge: ROS for non-ROS users. Springer Tracts Adv. Robot. 100, 493–504 (2017)CrossRefGoogle Scholar
  6. 6.
    Hawkins, K.P.: Analytic inverse kinematics for the universal robots UR-5/UR-10 arms. Technical report, Georgia Institute of Technology (2013).
  7. 7.
    Hokayem, P.F., Spong, M.W.: Bilateral teleoperation: an historical survey. Automatica 42(12), 2035–2057 (2006). Scholar
  8. 8.
    Jin, H., Chen, Q., Chen, Z., Hu, Y., Zhang, J.: Multi-LeapMotion sensor based demonstration for robotic refine tabletop object manipulation task. CAAI Trans. Intell. Technol. 1(1), 104–113 (2016). Scholar
  9. 9.
    Krupke, D., Einig, L., Langbehn, E., Zhang, J., Steinicke, F.: Immersive remote grasping: realtime gripper control by a heterogenous robot control system. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 02–04-Nove, pp. 337–338 (2016).
  10. 10.
    Kruusamae, K., Pryor, M.: High-precision telerobot with human-centered variable perspective and scalable gestural interface. In: Proceedings - 2016 9th International Conference on Human System Interactions, HSI 2016, pp. 190–196 (2016).
  11. 11.
    Li, C., Yang, C., Wan, J., Annamalai, A.S.S., Cangelosi, A.: Teleoperation control of Baxter robot using Kalman filter-based sensor fusion. Syst. Sci. Control Eng. 5(1), 156–167 (2017). Scholar
  12. 12.
    Lipton, J.I., Fay, A.J., Rus, D.: Baxter’s homunculus: virtual reality spaces for teleoperation in manufacturing. IEEE Robot. Autom. Lett. 3(1), 179–186 (2018). Scholar
  13. 13.
    Makris, S., et al.: Dual arm robot in cooperation with humans for flexible assembly. CIRP Ann. 66(1), 13–16 (2017). Scholar
  14. 14.
    Nuño, E., Basañez, L., Ortega, R.: Passivity-based control for bilateral teleoperation: A tutorial. Automatica 47(3), 485–495 (2011). Scholar
  15. 15.
    Pancake, D., et al.: A novel and cost effective approach to the decommissioning and decontamination of legacy glove boxes - minimizing TRU waste and maximizing LLW waste - 13634. In: Waste Management Symposium (WM 2013), Phoenix, AZ (2013)Google Scholar
  16. 16.
    Peppoloni, L., Brizzi, F., Avizzano, C.A., Ruffaldi, E.: Immersive ROS-integrated framework for robot teleoperation. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 177–178. IEEE, March 2015.
  17. 17.
    Roldán, J.J., et al.: Multi-robot systems, virtual reality and ros: developing a new generation of operator interfaces. In: Koubaa, A. (ed.) Robot Operating System (ROS). SCI, vol. 778, pp. 29–64. Springer, Cham (2019). Scholar
  18. 18.
    Tang, G., Webb, P.: The design and evaluation of an ergonomic contactless gesture control system for industrial robots. J. Robot. (2018). Scholar
  19. 19.
    Whitney, D., Rosen, E., Phillips, E., Konidaris, G., Tellex, S.: Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality. In: International Symposium on Robotics Research, pp. 1–16 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Inmo Jang
    • 1
    Email author
  • Joaquin Carrasco
    • 1
  • Andrew Weightman
    • 1
  • Barry Lennox
    • 1
  1. 1.The University of ManchesterManchesterUK

Personalised recommendations