Synergy-Driven Performance Enhancement of Vision-Based 3D Hand Pose Reconstruction

  • Simone CiottiEmail author
  • Edoardo Battaglia
  • Iason Oikonomidis
  • Alexandros Makris
  • Aggeliki Tsoli
  • Antonio Bicchi
  • Antonis A. Argyros
  • Matteo Bianchi
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 192)


In this work we propose, for the first time, to improve the performance of a Hand Pose Reconstruction (HPR) technique from RGBD camera data, which is affected by self-occlusions, leveraging upon postural synergy information, i.e., a priori information on how human most commonly use and shape their hands in everyday life tasks. More specifically, in our approach, we ignore joint angle values estimated with low confidence through a vision-based HPR technique and fuse synergistic information with such incomplete measures. Preliminary experiments are reported showing the effectiveness of the proposed integration.


Particle Swarm Optimization Joint Angle Hand Posture Visual Tracking Hand Part 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is supported in part by the European Research Council under the Advanced Grant SoftHands (No. ERC-291166), by the EU H2020 projects SoftPro (No. 688857) and SOMA (No. 645599), and by the EU FP7 project WEARHAP (No. 601165).


  1. 1.
    Ciotti, S., et al.: A synergy-based optimally designed sensing glove for functional grasp recognition. Sensors 16(6), 811 (2016)CrossRefGoogle Scholar
  2. 2.
    Sturman, D.J., et al.: A survey of glove-based input. IEEE Comput. Graphics Appl. 14(1), 30–39 (1994)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Dipietro, L., et al.: A survey of glove-based systems and their applications. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 38(4), 461–482 (2008)CrossRefGoogle Scholar
  4. 4.
    Bianchi, M., et al.: Synergy-based hand pose sensing: Reconstruction enhancement. Int. J. Robot. Res. 32(4), 396–406 (2013)CrossRefGoogle Scholar
  5. 5.
    Oikonomidis, I., et al.: Efficient model-based 3D tracking of hand articulations using kinect. In: British Machine Vision Conference (BMVC 2011), vol. 1, no. 2, pp. 1–11. BMVA, Dundee (2011)Google Scholar
  6. 6.
    Muth, J.T., et al.: Embedded 3D printing of strain sensors within highly stretchable elastomers. Adv. Mater. 26(36), 6307–6312 (2014)CrossRefGoogle Scholar
  7. 7.
    Hsiao, P.-C., et al.: Data glove embedded with 9-axis imu and force sensing sensors for evaluation of hand function. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4631–4634. IEEE (2015)Google Scholar
  8. 8.
    Bianchi, M., et al.: On the use of postural synergies to improve human hand pose reconstruction. In: 2012 IEEE Haptics Symposium (HAPTICS), pp. 91–98. IEEE (2012)Google Scholar
  9. 9.
    Bianchi, M., et al.: Synergy-based optimal design of hand pose sensing. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3929–3935, October 2012Google Scholar
  10. 10.
    Bianchi, M., et al.: Synergy-based hand pose sensing: optimal glove design. Int. J. Robot. Res. 32(4), 407–424 (2013)CrossRefGoogle Scholar
  11. 11.
    Bianchi, M., et al.: Exploiting hand kinematic synergies and wearable under-sensing for hand functional grasp recognition. In: 2014 EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth), pp. 168–171, November 2014Google Scholar
  12. 12.
    Bianchi, M., et al.: A multi-modal sensing glove for human manual-interaction studies. Electronics 5(3), 42 (2016)CrossRefGoogle Scholar
  13. 13.
    Santello, M., et al.: Postural hand synergies for tool use. J. Neurosci. 18(23), 10 105–10 115 (1998)Google Scholar
  14. 14.
    Santello, M., et al.: Hand synergies: integration of robotics and neuroscience for understanding the control of biological and artificial hands. Phys. Life Rev. 17, 1–23 (2016)CrossRefGoogle Scholar
  15. 15.
    Catalano, M.G., et al.: Adaptive synergies for the design and control of the Pisa/IIT softhand. Int. J. Robot. Res. 33(5), 768–782 (2014)CrossRefGoogle Scholar
  16. 16.
    Matrone, G.C., et al.: Principal components analysis based control of a multi-dof underactuated prosthetic hand. J. Neuroeng. Rehabil. 7(1), 1 (2010)CrossRefGoogle Scholar
  17. 17.
    Kennedy, J., et al.: Particle swarm optimization. In: International Conference on Neural Networks, vol. 4, no. 3, pp. 1942–1948. IEEE, January 1995Google Scholar
  18. 18.
    Sun, X., et al.: Cascaded hand pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 824–832 (2015)Google Scholar
  19. 19.
    Keskin, C., Kıraç, F., Kara, Y.E., Akarun, L.: Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 852–863. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33783-3_61 CrossRefGoogle Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2017

Authors and Affiliations

  • Simone Ciotti
    • 1
    • 2
    Email author
  • Edoardo Battaglia
    • 1
  • Iason Oikonomidis
    • 3
  • Alexandros Makris
    • 3
  • Aggeliki Tsoli
    • 3
  • Antonio Bicchi
    • 1
    • 2
  • Antonis A. Argyros
    • 3
  • Matteo Bianchi
    • 1
  1. 1.Research Center E. PiaggioUniversity of PisaPisaItaly
  2. 2.Department of Advanced Robotics (ADVR)Istituto Italiano di Tecnologia (IIT)GenovaItaly
  3. 3.Institute of Computer ScienceFoundation for Research and Technology, HellasHeraklionGreece

Personalised recommendations