Journal of Intelligent & Robotic Systems

, Volume 83, Issue 3–4, pp 339–358

Robotic Hand Pose Estimation Based on Stereo Vision and GPU-enabled Internal Graphical Simulation

  • Pedro Vicente
  • Lorenzo Jamone
  • Alexandre Bernardino

DOI: 10.1007/s10846-016-0376-6

Cite this article as:
Vicente, P., Jamone, L. & Bernardino, A. J Intell Robot Syst (2016) 83: 339. doi:10.1007/s10846-016-0376-6


Humanoid robots have complex kinematic chains whose modeling is error prone. If the robot model is not well calibrated, its hand pose cannot be determined precisely from the encoder readings, and this affects reaching and grasping accuracy. In our work, we propose a novel method to simultaneously i) estimate the pose of the robot hand, and ii) calibrate the robot kinematic model. This is achieved by combining stereo vision, proprioception, and a 3D computer graphics model of the robot. Notably, the use of GPU programming allows to perform the estimation and calibration in real time during the execution of arm reaching movements. Proprioceptive information is exploited to generate hypotheses about the visual appearance of the hand in the camera images, using the 3D computer graphics model of the robot that includes both kinematic and texture information. These hypotheses are compared with the actual visual input using particle filtering, to obtain both i) the best estimate of the hand pose and ii) a set of joint offsets to calibrate the kinematics of the robot model. We evaluate two different approaches to estimate the 6D pose of the hand from vision (silhouette segmentation and edges extraction) and show experimentally that the pose estimation error is considerably reduced with respect to the nominal robot model. Moreover, the GPU implementation ensures a performance about 3 times faster than the CPU one, allowing real-time operation.


Robot hand pose estimation Robot self-calibration Humanoid robots 3D graphical simulation GPU programming Online reaching adaptation Computer vision 

Funding information

Funder NameGrant NumberFunding Note
Fundação para a Ciência e a Tecnologia
  • UID/EEA/50009/2013
Seventh Framework Programme (BE)
  • FP7-ICT-288382
  • PIEF-GA-2013-628315

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Pedro Vicente
    • 1
  • Lorenzo Jamone
    • 1
  • Alexandre Bernardino
    • 1
  1. 1.Institute for Systems and Robotics (ISR/IST), LARSySInstituto Superior Técnico, University LisboaLisboaPortugal

Personalised recommendations