Modelling and Recognition of Signed Expressions Using Subunits Obtained by Data–Driven Approach

  • Mariusz Oszust
  • Marian Wysocki
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7557)


The paper considers automatic vision based modelling and recognition of sign language expressions using smaller units than words. Modelling gestures with subunits is similar to modelling speech by means of phonemes. To define the subunits a data–driven procedure is proposed. The procedure consists in partitioning time series of feature vectors obtained from video material into subsequences which form homogeneous clusters. The cut points are determined by an optimisation procedure based on quality assessment of the resulting clusters. Then subunits are selected in two ways: as clusters’ representatives or as hidden Markov models of clusters. These two approaches result in differences in classifier design. Details of the solution and results of experiments on a database of 101 Polish words and 35 sentences used at the doctor’s and in the post office are given. Our subunit–based classifiers outperform their whole–word–based counterpart, which is particularly evident when new expressions are recognised on the basis of a small number of examples.


Sign language recognition Time series segmentation Time series clustering Evolutionary optimisation Hidden Markov models Computer vision Data mining 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    De Castro, L.N., Von Zuben, F.J.: Learning and optimization using the clonal selection principle. IEEE Trans. on Evolutionary Computation 6, 239–251 (2002)CrossRefGoogle Scholar
  2. 2.
    Han, J., Awad, G., Sutherland, A.: Modelling and segmenting subunits for sign language recognition based on hand motion analysis. Pattern Recognition Letters 30, 623–633 (2009)CrossRefGoogle Scholar
  3. 3.
    Hendzel, J.K.: Dictionary of Polish Sign Language. OFFER, Olsztyn (1985) (in Polish)Google Scholar
  4. 4.
    Kong, W.W., Ranganath, S.: Automatic Hand Trajectory Segmentation and Phoneme Transcription for Sign Language. In: 8th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2008, pp. 1–6 (2008)Google Scholar
  5. 5.
    Kraiss, K.F.: Advanced man–machine interaction. Springer, Berlin (2006)CrossRefGoogle Scholar
  6. 6.
    Liao, T.W.: Clustering of time series data – a survey. Pattern Recognition 38, 1857–1874 (2005)zbMATHCrossRefGoogle Scholar
  7. 7.
    Ong, S.C.W., Ranganath, S.: Automatic sign language analysis: a survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27(6), 873–891 (2005)CrossRefGoogle Scholar
  8. 8.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proceedings 77, 257–286 (1989)CrossRefGoogle Scholar
  9. 9.
    Ratanamahatana, C.A., Keogh, E.: Three myths about dynamic time warping data mining. In: SIAM Int. Conf. on Data Mining, pp. 506–510 (2005)Google Scholar
  10. 10.
    Segouat, J., Braffort, A.: Toward Modeling Sign Language Coarticulation. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 325–336. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Theodorakis, S., Pitsikalis, V., Maragos, P.: Model–level data-driven sub–units for signs in videos of continuous sign language. In: IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 2262–2265 (2010)Google Scholar
  12. 12.
    Theodoridis, A., Kontroumbas, K.: Pattern Recognition. Acad. Press, London (1999)Google Scholar
  13. 13.
    Trojanowski, K., Wierzchon, S.: Immune-based algorithms for dynamic optimization. Information Sciences 179, 1495–1515 (2009)CrossRefGoogle Scholar
  14. 14.
    Tseng, V.S., Chen, C.H., Huang, P.C., Hong, T.P.: Cluster-based genetic segmentation of time series with DWT. Pattern Recognition Letters 30(13), 1190–1197 (2009)CrossRefGoogle Scholar
  15. 15.
    Vogler, C., Metaxas, D.A.: Framework for recognizing the simultaneous aspects of American sign language. Computer Vision and Image Understanding 81(3), 358–384 (2001)zbMATHCrossRefGoogle Scholar
  16. 16.
    Xu, R., Wunsch, D.C.: Clustering. J. Wiley and Sons, Inc., Hoboken (2009)Google Scholar
  17. 17.
    Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P.: The HTK Book. Cambridge University (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Mariusz Oszust
    • 1
  • Marian Wysocki
    • 1
  1. 1.Department of Computer and Control EngineeringRzeszow University of TechnologyRzeszowPoland

Personalised recommendations