Curve Matching from the View of Manifold for Sign Language Recognition

  • Yushun LinEmail author
  • Xiujuan Chai
  • Yu Zhou
  • Xilin Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9010)


Sign language recognition is a challenging task due to the complex action variations and the large vocabulary set. Generally, sign language conveys meaning through multichannel information like trajectory, hand posture and facial expression simultaneously. Obviously, trajectories of sign words play an important role for sign language recognition. Although the multichannel features are helpful for sign representation, this paper only focuses on the trajectory aspect. A method of curve matching based on manifold analysis is proposed to recognize isolated sign language word with 3D trajectory captured by Kinect. From the view of manifold, the main structure of the curve is found by the intrinsic linear segments, which are characterized by some geometric features. Then the matching between curves is transformed into the matching between two sets of sequential linear segments. The performance of the proposed curve matching strategy is evaluated on two different sign language datasets. Our method achieves a top-1 recognition rate of 78.3 % and 61.4 % in a 370 daily words dataset and a large dataset containing 1000 vocabularies.


Hide Markov Model Recognition Rate Sign Language Dynamic Time Warping Sign Language Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was partially supported by the Microsoft Research Asia, and Natural Science Foundation of China under contract Nos. 61303170, 61472398, and the FiDiPro program of Tekes.


  1. 1.
    Murakami, K., Taguchi, H.: Gesture recognition using recurrent neural networks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1991, pp. 237–242. ACM, New York (1991)Google Scholar
  2. 2.
    Huang, C.L., Huang, W.Y., Lien, C.C.: Sign language recognition using 3-d hopfield neural network. In: Proceedings of the International Conference on Image Processing, vol. 2, pp. 611–614 (1995)Google Scholar
  3. 3.
    Kim, J.S., Jang, W., Bien, Z.: A dynamic gesture recognition system for the korean sign language (ksl). IEEE Trans. Syst. Man Cybern. Part B: Cybern. 26, 354–359 (1996)CrossRefGoogle Scholar
  4. 4.
    Grobel, K., Assan, M.: Isolated sign language recognition using hidden markov models. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics, Computational Cybernetics and Simulation, vol. 1, pp. 162–167 (1997)Google Scholar
  5. 5.
    Starner, T., Weaver, J., Pentland, A.: Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1371–1375 (1998)CrossRefGoogle Scholar
  6. 6.
    Mokhtarian, F., Mackworth, A.: Scale-based description and recognition of planar curves and two-dimensional shapes. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–8, 34–43 (1986)CrossRefGoogle Scholar
  7. 7.
    Zuliani, M., Bhagavathy, S., Manjunath, B., Kenney, C.: Affine-invariant curve matching. In: 2004 International Conference on Image Processing, ICIP 2004, vol. 5, pp. 3041–3044 (2004)Google Scholar
  8. 8.
    Efrat, A., Fan, Q., Venkatasubramanian, S.: Curve matching, time warping, and light fields: New algorithms for computing similarity between curves. J. Math. Imaging Vis. 27, 203–216 (2007)CrossRefMathSciNetGoogle Scholar
  9. 9.
    Pajdla, T., Gool, L.V.: Matching of 3-d curves using semi-differential invariants. In: Proceedings of the Fifth International Conference on Computer Vision, pp. 390–395 (1995)Google Scholar
  10. 10.
    Kishon, E., Hastie, T., Wolfson, H.: 3-d curve matching using splines. In: Faugeras, O. (ed.) ECCV 1990. LNCS, vol. 427, pp. 589–591. Springer, Heidelberg (1990) CrossRefGoogle Scholar
  11. 11.
    Shahraray, B., Anderson, D.: Uniform resampling of digitized contours. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–7, 674–681 (1985)CrossRefGoogle Scholar
  12. 12.
    Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: A \({\$}\)1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST 2007, pp. 159–168. ACM, New York (2007)Google Scholar
  13. 13.
    Wang, R., Shan, S., Chen, X., Chen, J., Gao, W.: Maximal linear embedding for dimensionality reduction. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1776–1792 (2011)CrossRefGoogle Scholar
  14. 14.
    Bahlmann, C., Burkhardt, H.: The writer independent online handwriting recognition system frog on hand and cluster generative statistical dynamic time warping. IEEE Trans. Pattern Anal. Mach. Intell. 26, 299–310 (2004)CrossRefGoogle Scholar
  15. 15.
    Martens, R., Claesen, L.: On-line signature verification by dynamic time-warping. In: Proceedings of the 13th International Conference on Pattern Recognition, vol. 3, pp. 38–42 (1996)Google Scholar
  16. 16.
    Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 26, 43–49 (1978)CrossRefzbMATHGoogle Scholar
  17. 17.
    Ren, Z., Meng, J., Yuan, J., Zhang, Z.: Robust hand gesture recognition with kinect sensor. In: Proceedings of the 19th ACM International Conference on Multimedia, MM 2011, pp. 759–760. ACM, New York (2011)Google Scholar
  18. 18.
    Tong, J., Zhou, J., Liu, L., Pan, Z., Yan, H.: Scanning 3d full human bodies using kinects. IEEE Trans. Visual Comput. Graphics 18, 643–650 (2012)CrossRefGoogle Scholar
  19. 19.
    Zafrulla, Z., Brashear, H., Starner, T., Hamilton, H., Presti, P.: American sign language recognition with the kinect. In: Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI 2011, pp. 279–286, ACM, New York (2011)Google Scholar
  20. 20.
    Sun, C., Zhang, T., Bao, B.K., Xu, C., Mei, T.: Discriminative exemplar coding for sign language recognition with kinect. IEEE Trans. Cybern. 43, 1418–1428 (2013)CrossRefGoogle Scholar
  21. 21.
    Al-Hajj Mohamad, R., Likforman-Sulem, L., Mokbel, C.: Combining slanted-frame classifiers for improved hmm-based arabic handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31, 1165–1177 (2009)CrossRefGoogle Scholar
  22. 22.
    Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X., Zhou, M.: Sign language recognition and translation with kinect. In: IEEE Conference on AFGR (2013)Google Scholar
  23. 23.
    Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1290–1297 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Yushun Lin
    • 1
    • 2
    Email author
  • Xiujuan Chai
    • 1
  • Yu Zhou
    • 3
  • Xilin Chen
    • 1
  1. 1.Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS)Institute of Computing Technology, CASBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina
  3. 3.Institute of Information EngineeringCASBeijingChina

Personalised recommendations