GIST Descriptors for Sign Language Recognition: An Approach Based on Symbolic Representation

  • H.S. Nagendraswamy
  • B.M. Chethana Kumara
  • R. Lekha Chinmayi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9468)


This paper presents an approach for recognizing signs made by hearing impaired people at sentence level. The signs are captured in the form of video and each frame is processed to efficiently extract sign information to model the sign and recognize instances of new test signs. Low-dimensional global “gist” descriptors are used to capture sign information from every frame of a sign video. K-means clustering is used to choose fixed number of frames, which are discriminative enough to distinguish between signs. Also, selection of fixed number of frames helps us to deal with unequal number of frames among the instances of same sign due to different signers and reduce the complexity of subsequent processing. Further, we exploit the concept of symbolic data analysis to effectively represent a sign. A fuzzy trapezoidal membership function is used to establish the similarity between test and a reference sign and a nearest neighbour classification technique is used to recognize the given test sign. A considerably large database of signs (UoM-ISL) is created and an extensive experimentation is conducted on this database to study the efficacy of the proposed methodology. The experimental results are found to be encouraging.


Gist descriptor Sign language Symbolic representation Video sequence 



We would like to thank the students and the teaching staff of Sai Ranga Residential Boy’s School for Hearing Impaired, Mysore, and N K Ganpaiah Rotary School for physically challenged, Sakaleshpura, Hassan, Karnataka, INDIA, their immense support in the process of UoM-ISL Sign language dataset creation.


  1. 1.
    Al-Ahdal, M., Tahir, N.: Review in sign language recognition systems. In: IEEE Symposium on Computers and Informatics (ISCI), pp. 52–57 (2012)Google Scholar
  2. 2.
    Ghotkar, A.S., Kharate, G.K.: Study of vision based hand gesture recognition using Indian sign language. IJSS Intell. Syst. 7(1), 96–115 (2014)Google Scholar
  3. 3.
    Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. In: Martinez-Conde, S., Macknik, S.L., Martinez, L.M., Alonso, J.-M., Tse, P.U. (eds.) Progress in Brain Research, vol. 155 (2006). ISSN 0079-6123Google Scholar
  4. 4.
    Bock, H.H., Diday, E.: Analysis of Symbolic Data. Springer, Berlin (2000)CrossRefGoogle Scholar
  5. 5.
    Dahmani, D., Larabi, S.: User-independent system for sign language finger spelling recognition. J. Vis. Commun. Image Represent. 25(5), 1240–1250 (2014)CrossRefGoogle Scholar
  6. 6.
    Gourley, C.: Neural network utilizing posture input for sign language recognition. Technical report Computer Vision and Robotics Research Laboratory, University of Tenessee Knoxville, November 1994Google Scholar
  7. 7.
    Gowda, K.C., Diday, E.: Symbolic clustering using a new dissimilarity measure. Pattern Recogn. 24(6), 567–578 (1991)CrossRefGoogle Scholar
  8. 8.
    Gowda, K.C., Ravi, T.V.: Divisive clustering of symbolic objects using the concepts of both similarity and dissimilarity. Pattern Recogn. 28(8), 1277–1282 (1995)CrossRefGoogle Scholar
  9. 9.
    Guru, D.S., Prakash, H.N.: Online signature verification and recognition: an approach based on symbolic representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(6), 1059–1073 (2009)CrossRefGoogle Scholar
  10. 10.
    Guru, D.S., Nagendraswamy, H.S.: Clustering of interval-valued symbolic patterns based on mutual similarity value and the concept of k-mutual nearest neighborhood. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 234–243. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Guru, D.S., Kiranagi, B.B., Nagabhushan, P.: Multivalued type proximity measure and concept of mutual similarity value useful for clustering symbolic patterns. Pattern Recogn. Lett. 25(10), 1203–1213 (2004)CrossRefGoogle Scholar
  12. 12.
    Handouyahia, M., Zion, D., Wang, S.: Sign language recognition using moment based size functions. In: Vision Interface 1999, Trois-Rivieres, Canada, 19–21 May 1999Google Scholar
  13. 13.
    Handouyahia, M.: Sign Language Recognition using moment based size functions. MSc en Informatique demathematique et d informations, universite de sherbrooke, sherbrooke (1998)Google Scholar
  14. 14.
    Hu, M.: Visual pattern recognition by moment invariants. IRE Trans Inf. Theory 8, 179 (1962)zbMATHGoogle Scholar
  15. 15.
    Kang, S., Nam, M., Rhee, P.: Colour based hand and finger detection technology for user interaction. In: International Conference on Convergence and Hybrid Information Technology, pp. 229–236 (2008)Google Scholar
  16. 16.
    Karthick, P., Prathiba, N., Rekha, V.B., Thanalaxmi, S.: Transforming Indian sign language into text using leap motion. Int. J. Innovative Res. Sci. Eng. Technol. (An ISO 3297: 2007 Certified Organization) 3(4), 10906–10910 (2014)Google Scholar
  17. 17.
    Kong, W.W., Ranganath, S.: Towards subject independent continuous sign language recognition: a segment and merge approach. Pattern Recogn. 47, 1294–1308 (2014)CrossRefGoogle Scholar
  18. 18.
    Liwicki, S., Everingham, M.: Automatic recognition of finger spelled words in British sign language. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2009. IEEE (2009)Google Scholar
  19. 19.
    MohanKumar, H.P., Nagendraswamy, H.S.: Change energy image for gait recognition: an approach based on symbolic representation. Int. J. Image Graphics Signal Proc. (IJIGSP) 6(4), 1–8 (2014)CrossRefGoogle Scholar
  20. 20.
    Murakami, K., Taguchi, H.: Gesture recognition using recurrent neural network. In: Actes de CHI 1991 Workshop on User Interface by Hand Gesture, pp 237–242. ACM (1991)Google Scholar
  21. 21.
    Nagendraswamy, H.S., Naresh, Y.G.: Representation and classification of medicinal plants: a symbolic approach based on fuzzy inference technique. In: Proceedings of the Second International Conference on Soft Computing for Problem Solving (SocProS 2012), 28–30 December 2012Google Scholar
  22. 22.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  23. 23.
    Oliva, A., Schyns, P.: Coarse blobs or fine edges? Evidence that information diagnosticity changes the perception of complex visual stimuli. Cogn. Psychol. 34, 72–107 (1997)CrossRefGoogle Scholar
  24. 24.
    Holden, E.-J., Lee, G., Owens, R.: Australian sign language recognition. Mach. Vis. Appl. 16(5), 312–320 (2005)CrossRefGoogle Scholar
  25. 25.
    Harling, P.A.: Gesture input using neural networks. Technical report, University of York, UK (1993)Google Scholar
  26. 26.
    Potter, M.C.: Meaning in visual search. Science 187(4180), 965–966 (1975)CrossRefGoogle Scholar
  27. 27.
    Rensink, R.A.: The dynamic representation of scenes. Vis. Cogn. 7, 17–42 (2000)CrossRefGoogle Scholar
  28. 28.
    Haralick, R.M., Shaipro, L.G.: Local invariant feature detectors: a survey. In: Computer and Robot Vision, vol. 2. Addison-Wesley Publishing Company, Boston (1993)Google Scholar
  29. 29.
    Ross, T.J.: Fuzzy Logic with Engineering Applications. Wiley, New York (2009)Google Scholar
  30. 30.
    Siagian, C., Itti, L.: Comparison of gist models in rapid scene categorization tasks. In: Proceedings of Vision Science Society Annual Meeting (VSS 2008), May 2008Google Scholar
  31. 31.
    Starner, T., Weaver, J., Pentland, A.: Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1371–1375 (1998)CrossRefGoogle Scholar
  32. 32.
    Suraj, M.G., Guru, D.S.: Secondary diagonal FLD for fingerspelling Recognition. In: International Conference on Computing: Theory and Applications, International Conference on Computing: Theory and Applications, ICCTA 2007, pp. 693–697, (2007). doi: 10.1109/ICCTA.2007
  33. 33.
    Takahashi, T., Kishino, F.: Hand gesture coding based on experiments using a hand gesture interface device. SIGCHI Bull. 23, 67–74 (1991)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • H.S. Nagendraswamy
    • 1
  • B.M. Chethana Kumara
    • 1
  • R. Lekha Chinmayi
    • 1
  1. 1.Department of Studies in Computer ScienceUniversity of MysoreMysoreIndia

Personalised recommendations