Recognition of Signed Expressions Using Symbolic Aggregate Approximation

  • Mariusz Oszust
  • Marian Wysocki
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8467)


Complexity of sign language recognition system grows with growing word vocabulary. Therefore it is advisable to use units smaller than words. Such elements, called subunits, resemble phonemes in spoken language. They are concatenated to form word models. We propose a data–driven procedure for finding subunits in time series representing signed expressions. The procedure consists in: (i) transformation of video material to time series describing hand movements, (ii) using Piecewise Aggregate Approximation (PAA) coefficients to represent subunits, and (iii) applying Symbolic Aggregate Approximation (SAX), which is based on PAA, to obtain appropriate symbolic description. Signed words represented by strings of SAX symbols are classified using nearest neighbour method with Dynamic Time Warping (DTW) technique. We compare the approach with whole–word recognition by presenting ten–fold cross–validation tests on a Polish sign language (PSL) corpus of 30 words. Recognition of new words using small number of examples is also considered. The experiments show superiority of the SAX based approach.


Sign Language Recognition Piecewise Aggregate Approximation Symbolic Aggregate Approximation Dynamic Time Warping 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Han, J.W., Awad, G., Sutherland, A.: Modelling and Segmenting Subunits for Sign Language Recognition based on Hand Motion Analysis. Pattern Recognition Letters 30(6), 623–633 (2009)CrossRefGoogle Scholar
  2. 2.
    Oszust, M., Wysocki, M.: Modelling and Recognition of Signed Expressions Using Subunits Obtained by Data–Driven Approach. In: Ramsay, A., Agre, G. (eds.) AIMSA 2012. LNCS, vol. 7557, pp. 315–324. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  3. 3.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time Human Pose Recognition in Parts from Single Depth Images. In: IEEE Computer Vision and Pattern Recognition, pp. 1297–1304 (2011)Google Scholar
  4. 4.
  5. 5.
    Capilla, D.: Sign Language Translator using Microsoft Kinect XBOX 360. MSc Thesis Department of Electrical Engineering and Computer Science Computer Vision Lab University of Tennessee, Knoxville (2012)Google Scholar
  6. 6.
    Lang, S., Block, M., Rojas, R.: Sign Language Recognition Using Kinect. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 394–402. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  7. 7.
    Zafrulla, Z., Brashear, H., Starner, T., Hamilton, H., Presti, P.: American Sign Language Recognition with the Kinect. In: ICMI, pp. 279–286 (2011)Google Scholar
  8. 8.
    Suarez, J., Murphy, R.R.: Hand Gesture Recognition with Depth Images: a Review, pp. 411–417. IEEE Press (2012)Google Scholar
  9. 9.
    Uebersax, D., Gall, J., den Bergh, M.V., Gool, L.J.V.: Real-time Sign Language Letter and Word Recognition from Depth Data. In: ICCV Workshops, pp. 383–390. IEEE Press (2011)Google Scholar
  10. 10.
    Kraiss, K.F.: Advanced Man-machine Interaction. Springer, Berlin (2006)CrossRefGoogle Scholar
  11. 11.
    Zahedi, M., Manashty, A.R.: Robust Sign Language Recognition System using ToF Depth Cameras. Computing Research Repository - CORR, abs/1105.0 (2011)Google Scholar
  12. 12.
    Cooper, H.: Sign Language Recognition: Generalising to More Complex Corpora. Centre for Vision Speech and Signal Processing, PhD thesis. University of Surrey (2010)Google Scholar
  13. 13.
    Agris, U., Zieren, J., Canzler, U., Bauer, B., Kraiss, K.F.: Recent Developments in Visual Sign Language Recognition. Universal Access in the Information Society 6(4), 323–362 (2008)CrossRefGoogle Scholar
  14. 14.
    Ong, E.J., Cooper, H., Pugeault, N., Bowden, R.: Sign Language Recognition using Sequential Pattern Trees. In: IEEE Computer Vision and Pattern Recognition, pp. 2200–2207 (2012)Google Scholar
  15. 15.
    Keogh, E., Lin, J.: Hot Sax: Efficiently Finding the most Unusual Time Series Subsequence. In: 5th IEEE International Conference on Data Mining, pp. 226–233 (2005)Google Scholar
  16. 16.
    Lin, J., Keogh, E., Wei, L., Lonardi, S.: Experiencing SAX: a Novel Symbolic Representation of Time Series. Data Mining and Knowledge Discovery 15(2), 107–144 (2007)CrossRefMathSciNetGoogle Scholar
  17. 17.
    Keogh, E., Chakrabarti, K., Pazzani, M., Mehrotra, S.: Dimensionality Reduction for Fast Similarity Search in Large Time Series Databases. Knowledge and Information Systems 3(3), 263–286 (2001)CrossRefzbMATHGoogle Scholar
  18. 18.
    Junejo, I.N., Al Aghbari, Z.: Using SAX Representation for Human Action Recognition. Journal of Visual Communication and Image Representation 23(6), 853–861 (2012)CrossRefGoogle Scholar
  19. 19.
    Kohlsdorf, D.K.H., Starner, T.E.: MAGIC Summoning: Towards Automatic Suggesting and Testing of Gestures with Low Probability of False Positives During Use. Journal of Machine Learning Research 14, 209–242 (2013)zbMATHMathSciNetGoogle Scholar
  20. 20.
    Polish Association of the Deaf,
  21. 21.
    Tomaszewski, P.: Visual phonology of Polish Sign Language. Publishing House Matrix (2010) (in Polish)Google Scholar
  22. 22.
    Hendzel, J.: Polish Sign Language Dictionary. Publishing House Pojezierze (1986) (in Polish)Google Scholar
  23. 23.
    Lakaemper, R.: KinectTCP. Temple University,
  24. 24.
    Obdrzalek, S., Kurillo, G., Ofli, F., Bajcsy, R., Seto, E., Jimison, H., Pavel, M.: Accuracy and Robustness of Kinect Pose Estimation in the Context of Coaching of Elderly Population. In: IEEE Annual International Conference on Engineering in Medicine and Biology Society (EMBC), pp. 1188–1193 (2012)Google Scholar
  25. 25.
    Vezhnevets, V., Sazonov, V., Andreeva, A.: A survey on Pixel-Based Skin Color Detection Techniques. In: GraphiCon, pp. 85–92 (2003)Google Scholar
  26. 26.
    Stokoe, W.C.: Sign Language Structure: an Outline of the Visual Communication Systems of the American Deaf. Journal of Deaf Studies and Deaf Education (1960, 2005) Google Scholar
  27. 27.
    Xu, R., Wunsch, D.: Clustering. Wiley-IEEE Press (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Mariusz Oszust
    • 1
  • Marian Wysocki
    • 1
  1. 1.Department of Computer and Control EngineeringRzeszow University of TechnologyRzeszowPoland

Personalised recommendations