Abstract
This paper aims to study eye-to-hand robotic tracking and grabbing based on binocular vision. Two cameras placed on different locations gave different three-dimensional coordinates of the object from a binocular vision. The robot then analyzed the dynamic vision acquired from the stereo cameras and tracked the object in three-dimensional space utilizing continuously adaptive mean shift (CAMSHIFT) algorithm. Afterward, inverse and forward kinematics were implemented to move the robotic arm to the appropriate position so that it can grab the object using the end effector. The inverse kinematics was analyzed by a geometric algorithm to reduce the computational burden. Consequently, the experimental results verified that the eye-to-hand system was able to track and grasp the target successfully.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Abbreviations
- \( f \) :
-
Focal length of the camera
- s :
-
Scale factor in image
- \( f_{x} \), \( f_{y} \) :
-
Focal length in X and Y directions of the image plane
- \( b \) :
-
Distance between two cameras
- d :
-
Disparity
- \( P \) :
-
The coordinates of the end effector
- \( P_{i} = (x_{i} ,y_{i} ) \) :
-
An image plane point
- \( c_{x} \), \( c_{y} \) :
-
Center of the image
- \(_{{i + 1}}^{i} P \) :
-
Homogeneous transformation matrix
- \( {\rm A}_{n} \) :
-
Link length
- \( u,v \) :
-
Image plane coordinate
- \( d_{i} \) :
-
Link distance
- \( \theta_{n} \) :
-
Joint angle
- \( \alpha_{i} \) :
-
Link twist
- \( (x_{c} ,y_{c} ) \) :
-
Center of mass
- Ms(x):
-
Mean shift value of the vector x
- \( M_{00} \) :
-
Zeroth-order momentum
- \( M_{10} \), \( M_{01} \) :
-
First-order momentum
- \( I(x,y) \) :
-
Color histogram value
- X′–Y′–Z′:
-
Wrist coordinate system
- X–Y–Z:
-
Base coordinate system
- X″–Y″–Z″:
-
End effector coordinate frame with respect to base frame
- \( P(X_{p} ,Y_{p} ,Z_{p} ) \) :
-
World coordinate
- \( R \) :
-
An orthonormal rotation matrix (3 by 3)
- T :
-
Translation matrix (3 by 1)
References
Aljuaid H, Mohamad D (2013) Object Tracking Simulates Babysitter Vision Robot Using GMM. In: 2013 international conference on soft computing and pattern recognition (SoCPaR), pp 60–65
Bradski GR (1998) Computer vision face tracking for use in a perceptual user interface. Intel Technol J 2:15
Cheng Y (1995) Mean shift, mode seeking, and clustering. IEEE Trans Pattern Anal Mach Intell 17:790–799
Fukunaga K, Hostetler L (1975) The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans Inf Theory 21:32–40
Hu M-C, Chang M-H, Wu J-L, Chi L (2011) Robust camera calibration and player tracking in broadcast basketball video. IEEE Trans Multimed 13:266–279
Hung SC, Li MF (2007) A SIFT method application for a six DOF manipulator grasping an object against complicated environment. In: the 33rd annual conference of the IEEE industrial electronics society, pp. 465–470
Itoh K, Kikuchi T, Mizoguchi H, Takemura H (2006) Development of a person following mobile robot in complicated background by using distance and color information. In: 32nd annual conference on IEEE industrial electronics, pp. 3839–3844
Jiang X, Huang WF (2006) Analysis and application of control theory of six-axis manipulators. J Mech Ind J 277:57–73 (in Chinese)
Kale K, Pawar S, Dhulekar P (2015) Moving object tracking using optical flow and motion vector estimation. In: 2015 4th international conference on reliability, infocom technologies and optimization (ICRITO) (trends and future directions), pp 1–6
Lee S, Jang D, Kim E, Hong S, Hong JH (2005) A real-time 3D workspace modeling with stereo camera. In: IEEE international conference on intelligent robots and systems, pp. 2140–2147
Long W, Yang Y (1990) Stationary background generation: an alternative to the difference of two images. Pattern Recogn 23:1351–1359
Mitsubishi (2010) RV-3SD/3SDJ/3SDB/3SDBJ Series standard specifications manual
Opencv dev team, OpenCv (2019) API Reference–camera calibration and 3D reconstruction, 2014, https://docs.opencv.org/3.0-beta/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html. Accessed 15 Feb 2019
Robotiq (2015) Robotiq-gripper instruction manual. http://robotiq.com/media/ROBOTIQ-INSTRUCTIONMANUAL-S-MODEL-ROBOT-GRIPPER.pdf. Accessed 15 Feb 2019
Taryudi T, Wang MS (2017) Eye to hand calibration using ANFIS for stereo vision-based object manipulation system. Microsyst Technol 23:1–13
Tsai LW (2005) Robot analysis: the mechanics of serial and parallel manipulators. Wiley-Interscience, Taiwan
Xiu C, Ba F (2016) Target tracking based on the improved Camshift method. In: 2016 Chinese control and decision conference (CCDC), pp. 3600–3604
Yu Y, Bi S, Mo Y, Qiu W (2016) Real-time gesture recognition system based on Camshift algorithm and Haar-like feature. In: 2016 IEEE international conference on cyber technology in automation, control, and intelligent systems (CYBER), pp. 337–342
Zhou J (2014) EKF based object detect and tracking for UAV by using visual-attention-model. In: 2014 IEEE international conference on progress in informatics and computing, pp 168–172
Acknowledgements
The authors would like to express their appreciation for the financial support bestowed by the following: Ministry of Science and Technology, Taiwan, contract no. MOST 107-2622-E-218-008-CC2 and Ministry of Education, Higher Education Sprout Project.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Du, YC., Taryudi, T., Tsai, CT. et al. Eye-to-hand robotic tracking and grabbing based on binocular vision. Microsyst Technol 27, 1699–1710 (2021). https://doi.org/10.1007/s00542-019-04475-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00542-019-04475-3