Abstract
Robot teleoperation is an essential tool for performing complex tasks that require dexterity beyond the capabilities of state-of-the-art algorithms. Existing teleoperation methods are often non-intuitive for human operators or require special sensors and equipment, making them cost-ineffective and impractical in many scenarios. In this paper, we propose a robot arm teleoperation framework that relies on monocular camera images. The proposed framework first uses lightweight neural networks for estimating the human operator body pose and recognizing their hand gesture. An efficient inverse kinematics algorithm then finds the desired robot arm configuration, achieving end-effector motion that imitates operator’s wrist movement. Our teleoperation framework can be executed on an average laptop using a web camera with any robotic arm that has a ROS interface. We verified its performance in real-world experiments with a Kinova Jaco robotic arm, demonstrating the ability to grasp and move an object in the environment.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Vertut, J.: Teleoperation and Robotics: Applications and Technology, vol. 3. Springer Science & Business Media (2013)
Kofman, J., Wu, X., Luu, T.J., Verma, S.: Teleoperation of a robot manipulator using a vision-based human-robot interface. IEEE Trans. Ind. Electr. 52(5), 1206–1219 (2005)
Hainsworth, D.W.: Teleoperation user interfaces for mining robotics. Auton. Rob. 11(1), 19–28 (2001)
Birk, A., Schwertfeger, S., Pathak, K.: A networking framework for teleoperation in safety, security, and rescue robotics. IEEE Wireless Commun 16(1), 6–13 (2009)
Artemiadis, P.K., Kyriakopoulos, K.J.: Emg-based control of a robot arm using low-dimensional embeddings. IEEE Trans. Robot. 26(2), 393–398 (2010)
Gupta, G.S., Mukhopadhyay, S.C., Messom, C.H., Demidenko, S.N.: Master-slave control of a teleoperated anthropomorphic robotic arm with gripping force sensing. IEEE Trans. Inst. Measur. 55(6), 2136–2145 (2006)
Szymańska, E., Petrović, L., Marković, I., Petrović, I.: Mobile robot teleoperation via android mobile device with udp communication. In: 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO), pp. 1143–1148. IEEE (2021)
Yepes, J.C., Yepes, J.J., Martinez, J.R., Pérez, V.Z.: Implementation of an android based teleoperation application for controlling a kuka-kr6 robot by using sensor fusion. In: Pan American Health Care Exchanges (PAHCE), pp. 1–5. IEEE (2013)
Syakir, M., Ningrum, E.S., Sulistijono, I.A.: Teleoperation robot arm using depth sensor. In: International Electronics Symposium (IES), pp. 394–399. IEEE (2019)
Marić, F., Jurin, I., Marković, I., Kalafatić, Z., Petrović, I.: Robot arm teleoperation via rgbd sensor palm tracking. In: 2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1093–1098. IEEE (2016)
Handa, A., Van Wyk, K., Yang, W., Liang, J., Chao, Y.W., Wan, Q., Birchfield, S., Ratliff, N., Fox, D.: Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9164–9170. IEEE (2020)
Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., Grundmann, M.: Blazepose: On-device real-time body pose tracking. arXiv preprint arXiv:2006.10204 (2020)
Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., Grundmann, M.: Blazeface: Sub-millisecond neural face detection on mobile gpus. ArXiv preprint arXiv:1907.05047 (2019)
Sung, G., Sokal, K., Uboweja, E., Bazarevsky, V., Baccash, J., Bazavan, E.G., Chang, C.L., Grundmann, M.: On-device real-time hand gesture recognition. ArXiv preprint arXiv:2111.00038 (2021)
Beeson, P., Ames, B.: Trac-ik: An open-source library for improved solving of generic inverse kinematics. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 928–935. IEEE (2015)
Acknowledgements
This research has been supported by the European Regional Development Fund under the grant PK.1.1.02.0008 (DATACROSS).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Parfaite, B., Petrović, L., Štironja, VJ., Marković, I., Petrović, I. (2024). Cost-Effective Robot Arm Teleoperation Via Human Pose Tracking with Monocular Camera. In: Pisla, D., Carbone, G., Condurache, D., Vaida, C. (eds) Advances in Service and Industrial Robotics. RAAD 2024. Mechanisms and Machine Science, vol 157. Springer, Cham. https://doi.org/10.1007/978-3-031-59257-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-59257-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-59256-0
Online ISBN: 978-3-031-59257-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)