Gesture Recognition of Indian Sign Language

Abstract

In today’s world, many people are not as fortunate as us. They suffer from speaking and hearing disabilities that make daily tasks very difficult to perform. This paper mainly deals with people suffering from the loss of hearing and cannot speak. In this study, we have introduced a recognition and classification technique which can detect hand gestures used by the deaf and dumb community, which is then translated to an equivalent English word. The vision-based system is built using two machine learning algorithms, namely convolution neural network (CNN) and recurrent neural network (RNN). The video sequences are not directly fed into the model instead those videos are first converted into frames and then fed into the model. Further, each frame is directed into CNN, the exact features are extracted, trained on those features and stored in a model file, these model files are fed into RNN for further feature extraction, and the model is trained on those features. The model can convert the sign language to text with an accuracy of 96%. To make the system as user-friendly as possible, we have provided a feature where the general public can record the video of the person doing the hand gesture and use it to make the equivalent English word prediction.