Real-Time Sign Language Gesture (Word) Recognition from Video Sequences Using CNN and RNN

  • Sarfaraz Masood
  • Adhyan Srivastava
  • Harish Chandra Thuwal
  • Musheer Ahmad
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 695)


There is a need of a method or an application that can recognize sign language gestures so that the communication is possible even if someone does not understand sign language. With this work, we intend to take a basic step in bridging this communication gap using Sign Language Recognition. Video sequences contain both the temporal and the spatial features. To train the model on spatial features, we have used inception model which is a deep convolutional neural network (CNN) and we have used recurrent neural network (RNN) to train the model on temporal features. Our dataset consists of Argentinean Sign Language (LSA) gestures, belonging to 46 gesture categories. The proposed model was able to achieve a high accuracy of 95.2% over a large set of images.


  1. 1.
    Ronchetti, F., Quiroga, F., Estrebou, C.A., Lanzarini, L.C.: Handshape recognition for Argentinian sign language using probsom. J. Comput. Sci. Technol. 16 (2016)Google Scholar
  2. 2.
    Singha, J., Das, K.: Automatic Indian Sign Language recognition for continuous video sequence. ADBU J. Eng. Technol. 2(1) (2015)Google Scholar
  3. 3.
    Tripathi, K., Nandi, N.B.G.C.: Continuous Indian Sign Language gesture recognition and sentence formation. Procedia Comput. Sci. 54, 523–531 (2015)CrossRefGoogle Scholar
  4. 4.
    Nandy, A., Prasad, J.S., Mondal, S., Chakraborty, P., Nandi, G.C.: Recognition of isolated Indian Sign Language gesture in real time. Inf. Process. Manag., 102–107 (2010)Google Scholar
  5. 5.
    Pigou, L., Dieleman, S., Kindermans, P.-J., Schrauwen, B.: Sign language recognition using convolutional neural networks. In: Workshop at the European Conference on Computer Vision 2014, pp. 572–578. Springer International Publishing (2014)CrossRefGoogle Scholar
  6. 6.
    Sharma, R., Bhateja, V., Satapathy, S.C., Gupta, S.: Communication device for differently abled people: a prototype model. In: Proceedings of the International Conference on Data Engineering and Communication Technology, pp. 565–575. Springer, Singapore (2017)Google Scholar
  7. 7.
    Masood, S., Thuwal, H.C., Srivastava, A.: American sign language character recognition using convolution neural network. In: Proceedings of Smart Computing and Informatics, pp. 403–412. Springer, Singapore (2018)Google Scholar
  8. 8.
    Vicars, W.: Sign language resources at Accessed 23 Sept 2017
  9. 9.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)Google Scholar
  10. 10.
    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems (2016). arXiv preprint arXiv:1603.04467
  11. 11.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)CrossRefGoogle Scholar
  12. 12.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  13. 13.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization (2014). arXiv preprint arXiv:1412.6980
  14. 14.
    Ronchetti, F., Quiroga, F., Estrebou, C.A., Lanzarini, L.C., Rosete, A.: LSA64: an Argentinian sign language dataset. In: XXII Congreso Argentino de Ciencias de la Computación (CACIC 2016) (2016)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  • Sarfaraz Masood
    • 1
  • Adhyan Srivastava
    • 1
  • Harish Chandra Thuwal
    • 1
  • Musheer Ahmad
    • 1
  1. 1.Department of Computer EngineeringJamia Millia IslamiaNew DelhiIndia

Personalised recommendations