Dexterous Hand Motion Classification and Recognition Based on Multimodal Sensing

  • Yaxu Xue
  • Zhaojie Ju
  • Kui Xiang
  • Chenguang Yang
  • Honghai Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10462)


Human hand motions analysis is an essential research topic in recent applications, especially for dexterous robot hand manipulation learning from human hand skills. It provides important information about gestures, moving, speed and the control force captured via multimodal sensing technologies. This paper presents a comprehensive discussion of the nature of human hand motions in terms of simple motions, such as grasps and gestures, and complex motions, e.g. in-hand manipulations and re-grasps. And then, a novel multimodal sensing based hand motion capture system is proposed to acquire the sensory information. By using an adaptive directed acyclic graph algorithm, the experimental results show the proposed system has a higher recognition rate compared with those with individual sensing technologies.


Multimodal sensing EMG Contact force Data glove Support vector machine 


  1. 1.
    Saudabayev, A., Varol, H.A.: Sensors for robotic hands: a survey of state of the art. IEEE. Access 3(9), 1765–1782 (2015)CrossRefGoogle Scholar
  2. 2.
    Liu, H., Nguyen, K.C., Perdereau, V., et al.: Finger contact sensing and the application in dexterous hand manipulation. Auton. Robots 39(1), 25–41 (2015)CrossRefGoogle Scholar
  3. 3.
    Khan, R.Z., Ibraheem, N.A.: Survey on gesture recognition for hand image postures. Comput. Inf. Sci. 5(3), 110–121 (2012)Google Scholar
  4. 4.
    Liu, H., Ju, Z., Ji, X., Chan, S., Khoury, M.: Human Motion Sensing and Recognition: A Fuzzy Qualitative Approach. Springer, Heidelberg (2017)CrossRefGoogle Scholar
  5. 5.
    Wheatland, N., Wang, Y., Song, H., et al.: State of the art in hand and finger modeling and animation. Comput. Graph. Forum 34(2), 735–760 (2015)CrossRefGoogle Scholar
  6. 6.
    Ju, Z., Liu, H.: Human hand motion analysis with multisensory information. IEEE/ASME Trans. Mechatron. 675(5), 456–466 (2014)CrossRefGoogle Scholar
  7. 7.
    Marin, G., Dominio, F., Zanuttigh, P.: Hand gesture recognition with leap motion and kinect devices. In: 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, pp. 1565–1569. IEEE (2014)Google Scholar
  8. 8.
    Ju, Z., Liu, H.: Human hand motion recognition using empirical copula. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, pp. 4625–4630. IEEE (2010)Google Scholar
  9. 9.
    Elliott, J.M., Connolly, K.J.: A classification of manipulative hand movements. Dev. Med. Child Neurol. 3(6), 283–296 (1984)Google Scholar
  10. 10.
    Exner, C.E.: The zone of proximal development in in-hand manipulation skills of nondysfunctional 3-and 4-year-old children. Am. J. Occup. Ther. 44(10), 884–891 (1990)CrossRefGoogle Scholar
  11. 11.
    Pont, K., Wallen, M., Bundy, A.: Conceptualising a modified system for classification of inhand manipulation. Aust. Occup. Ther. J. 56(2), 2–15 (2009)CrossRefGoogle Scholar
  12. 12.
    Fougner, A., Scheme, E., Chan, A.D.C., et al.: A multi-modal approach for hand motion classification using surface EMG and accelerometers. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC, Boston, USA, pp. 4247–4250. IEEE (2011)Google Scholar
  13. 13.
    Bullock, I.M., Dollar, A.M.: Classifying human manipulation behavior. In: IEEE International Conference on Rehabilitation Robotics (ICORR), Zurich, Switzerland, pp. 1–6. IEEE (2011)Google Scholar
  14. 14.
    Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appli. Rev.) 33(4), 311–324 (2007)CrossRefGoogle Scholar
  15. 15.
    Bullock, I.M., Ma, R.R., Dollar, A.M.: A hand-centric classification of human and robot dexterous manipulation. IEEE Trans. Haptics 6(9), 129–144 (2013)CrossRefGoogle Scholar
  16. 16.
    Jiang, M.W., Wang, R.C., Wang, J.Z., et al.: A method of recognizing finger motion using wavelet transform of surface EMG signal. In: 27th Annual International Conference on Engineering in Medicine and Biology, Shanghai, China, pp. 2672–2674. IEEE (2006)Google Scholar
  17. 17.
    Ju, Z., Ouyang, G., Wilamowska-Korsak, M., et al.: Surface EMG based hand manipulation identification via nonlinear feature extraction and classification. IEEE Sens. J. 13(4), 3302–3311 (2013)CrossRefGoogle Scholar
  18. 18.
    Lu, Y., Lu, G., Bu, X., Yu, Y.: Classification of hand manipulation using bp neural network and support vector machine based on surface electromyography signal. IFAC-PapersOnLine 45(12), 869–873 (2015)CrossRefGoogle Scholar
  19. 19.
    Kappassov, Z., Corrales, J.A., Perdereau, V.: Tactile sensing in dexterous robot hands - Review. Robot. Autonom. Syst. 74(7), 195–220 (2015)CrossRefGoogle Scholar
  20. 20.
    Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2(6), 121–167 (1998)CrossRefGoogle Scholar
  21. 21.
    Abe, S.: Support Vector Machines for Pattern Classification. Springer, London (2005)MATHGoogle Scholar
  22. 22.
    Ben-Hur, A., Weston, J.: A users guide to support vector machines. Data Min. Tech. Life Sci. 609(10), 223–239 (2010)Google Scholar
  23. 23.
    Kijsirikul, B., Ussivakul, N.: Multiclass support vector machines using adaptive directed acyclic graph. In: Proceedings of the 2002 International Joint Conference on Neural Networks, IJCNN 2002, Honolulu, HI, USA, pp. 980–985. IEEE (2002)Google Scholar
  24. 24.
    Platt, J.C., Cristianini, N., Shawe-Taylor, J.: Large margin DAGs for multiclass classification. In: Advances in Neural Information Processing Systems, pp. 547–553 (1999)Google Scholar
  25. 25.
    Ju, Z., Ji, X., Li, J., Liu, H.: An integrative framework of human hand gesture segmentation for human robot interaction. IEEE Syst. J. 99(9), 1–11 (2015)CrossRefGoogle Scholar
  26. 26.
    Ju, Z., Gao, D., Cao, J., Liu, H.: A novel approach to extract hand gesture feature in depth images. Multimedia Tools Appl. 75(19), 11929–11943 (2017). Springer, HeidelbergCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yaxu Xue
    • 1
    • 2
    • 3
  • Zhaojie Ju
    • 2
  • Kui Xiang
    • 1
  • Chenguang Yang
    • 4
  • Honghai Liu
    • 2
  1. 1.School of AutomationWuhan University of TechnologyWuhanChina
  2. 2.School of ComputingThe University of PortsmouthPortsmouthUK
  3. 3.School of Electrical and Mechanical EngineeringPingdingshan UniversityPingdingshanChina
  4. 4.College of EngineeringSwansea UniversitySwanseaUK

Personalised recommendations