An Auto-Correction Teleoperation Method for a Mobile Manipulator Using Gaze Tracking and Hand Motion Detection

  • Junshen Chen
  • Ze JiEmail author
  • Hanlin Niu
  • Rossitza Setchi
  • Chenguang Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11650)


Situational awareness in remote environments is crucial for human operators to teleoperate mobile manipulators confidently and reliably. Visual feedback is the most common way for environment perception, providing rich information to human operators. This paper proposes an intuitive teleoperation method by combining gaze tracking and hand motion detection to teleoperate a mobile manipulator. A camera is fixed on the end-effector of the mobile robot’s arm to provide visual feedback, acting as the eye of the teleoperator. Rather than direct remote control of the robot, an adaptive auto-correction mechanism is introduced for helping human operators to achieve better hand-eye coordination of the teleoperation experiences. The mobile manipulator can adjust its behaviours, such as speed, while gaze and hand movements of the operator are in different states. The experiments carried out demonstrated the effectiveness of the proposed algorithm and the survey evaluation verified the practical application value of the system.


Teleoperation Hand-eye coordination Mobile manipulator 


  1. 1.
    Bischoff, R., Huggenberger, U., Prassler, E.: KUKA youBot-a mobile manipulator for research and education. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4. IEEE (2011)Google Scholar
  2. 2.
    Brooke, J., et al.: SUS-a quick and dirty usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 4–7. CRC Press, London (1996)Google Scholar
  3. 3.
    Du, G., Zhang, P., Liu, X.: Markerless human-manipulator interface using leap motion with interval kalman filter and improved particle filter. IEEE Trans. Industr. Inf. 12(2), 694–704 (2016). Scholar
  4. 4.
    Dziemian, S., Abbott, W.W., Faisal, A.A.: Gaze-based teleprosthetic enables intuitive continuous control of complex robot arm use: writing amp; drawing. In: 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 1277–1282, June 2016.
  5. 5.
    Ggo, D., Carreto, C., Figueiredo, L.: Teleoperation of a mobile robot based on eye-gaze tracking. In: 2017 12th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6, June 2017.
  6. 6.
    Li, X., et al.: Active cannula robot with misorientation auto-recovery camera: a method to improve hand-eye coordination in minimally invasive surgery. In: 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013), pp. 276–280, Oct 2013.
  7. 7.
    Peppoloni, L., Brizzi, F., Avizzano, C.A., Ruffaldi, E.: Immersive ROS-integrated framework for robot teleoperation. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 177–178, March 2015.
  8. 8.
    Qiu, R., et al.: Towards robust personal assistant robots: experience gained in the SRS project. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1651–1657, Oct 2012.
  9. 9.
    Tunstel, E., Maimone, M., Trebi-Ollennu, A., Yen, J., Petras, R., Willson, R.: Mars exploration rover mobility and robotic arm operational performance. In: 2005 IEEE International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1807–1814. IEEE (2005)Google Scholar
  10. 10.
    Xiang, Z., Su, J., Ma, Z.: Uncalibrated hand-eye coordination based HRI on humanoid robot. In: Proceedings of the 33rd Chinese Control Conference, pp. 8438–8443, July 2014.
  11. 11.
    Yu, M., Wang, X., Lin, Y., Bai, X.: Gaze tracking system for teleoperation. In: The 26th Chinese Control and Decision Conference (2014 CCDC), pp. 4617–4622, May 2014.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Junshen Chen
    • 1
  • Ze Ji
    • 1
    Email author
  • Hanlin Niu
    • 1
  • Rossitza Setchi
    • 1
  • Chenguang Yang
    • 2
  1. 1.School of EngineeringCardiff UniversityCardiffUK
  2. 2.Bristol Robotics LaboratoryUniversity of the West of EnglandBristolUK

Personalised recommendations