Abstract
There are nearly 70 million deaf people in the world. A significant portion of them and their families use sign language as a medium for communicating with each other. As automation is being gradually introduced to many parts of everyday life, the ability for machines to understand the act on sign language will be critical to creating an inclusive society. This paper presents multiple convolutional neural network based approaches, suitable for fast classification of hand sign characters. We propose two custom convolutional neural network (CNN) based architectures which are able to generalize 24 static American Sign Language (ASL) signs using only convolutional and fully connected layers. We compare these networks with transfer learning based approaches, where multiple pre-trained models were utilized. Our models have remarkably outperformed all the preceding models by accomplishing \(86.52\%\) and \(85.88\%\) accuracy on RGB images of the ASL Finger Spelling dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, R., Wiryana, F., Chandra, M., Putra, G.: Sign language recognition application systems for deaf-mute people: a review based on input-process-output. Procedia Comput. Sci. 116, 441–448 (2017)
Arge, F.O.R.L., Mage in CI: Vdcnl-s i r, pp. 1–14 (2015)
2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies, pp. 1412–1415 (2014)
Núñez Fernández, D., Kwolek, B.: Hand posture recognition using convolutional neural network. In: Mendoza, M., VelastÃn, S. (eds.) CIARP 2017. LNCS, vol. 10657, pp. 441–449. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75193-1_53
Ghotkar, A., Kharate, G.K.: Study of vision based hand gesture recognition using Indian sign language (2017)
Chollet, F.: Xception: deep learning with depthwise separable convolutions (2014)
Hoque, T., Kabir, F.: Automated Bangla sign language translation system: prospects, limitations and applications, pp. 856–862 (2016)
Hosoe, H., Sako, S.: Recognition of JSL finger spelling using convolutional neural networks, pp. 85–88 (2017)
Huang, G., Weinberger, K.Q.: Densely connected convolutional networks (2016)
Karabasi, M., Bhatti, Z., Shah, A.: A model for Real-time recognition and textual representation of Malaysian sign language through image processing. In: 2013 International Conference on Advanced Computer Science Applications and Technologies (2013)
Karmokar, B.C., Alam, K.R., Siddiquee, K.: Bangladeshi sign language recognition employing neural network ensemble (2012)
Kishore, P.V.V., Kumar, P.R.: Segment, track, extract, recognize and convert sign language videos to voice/text. IJACSA 3, 35–47 (2012)
Koller, O., Forster, J., Ney, H.: Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers. Comput. Vis. Image Underst. 141, 108–125 (2015)
Kumar, P.K., Prahlad, P., Loh, A.P.: Attention based detection and recognition of hand postures against complex backgrounds (2012)
Masood, S., Srivastava, A., Thuwal, H.C., Ahmad, M.: Real-time sign language gesture (word) recognition from video sequences using CNN and RNN. In: Bhateja, V., Coello Coello, C.A., Satapathy, S.C., Pattnaik, P.K. (eds.) Intelligent Engineering Informatics. AISC, vol. 695, pp. 623–632. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-7566-7_63
Mekala, P., Gao, Y., Fan, J., Davari, A.: Real-time sign language recognition based on neural network architecture, pp. 195–199 (2011)
Prajapati, R., Pandey, V., Jamindar, N., Yadav, N., Phadnis, P.N.: Hand gesture recognition and voice conversion for deaf and dumb. IRJET 5, 1373–1376 (2018)
Pugeault, N., Bowden, R.: Spelling it out: real-time ASL fingerspelling recognition (2011)
Rahaman, M.A., Jasim, M., Ali, H.: Real-time computer vision-based Bengali sign language recognition, pp. 192–197 (2014)
Rajam, P.S., Balakrishnan, G.: Real time Indian sign language recognition system to aid deaf-dumb people, pp. 1–6 (2011)
Rao, G.A., Kishore, P.V.: Selfie video based continuous Indian sign language recognition system. Ain Shams Eng. J. 9, 1929 (2017)
Sandler, M., Zhu, M., Zhmoginov, A., Howard, A., Chen, L.-C.: MobileNetV2: inverted residuals and linear bottlenecks (2018)
Savur, C.: Real-time American sign language recognition system by using surface EMG signal, pp. 497–502 (2015)
Sarawate, N., Leu, M.C., ÖZ, C.: A real-time American sign language word recognition system based on neural networks and a probabilistic model. Turk. J. Electr. Eng. Comput. Sci. 23, 2107–2123 (2015)
Seth, D., Ghosh, A., Dasgupta, A., Nath, A.: Real time sign language processing system. In: Unal, A., Nayak, M., Mishra, D.K., Singh, D., Joshi, A. (eds.) SmartCom 2016. CCIS, vol. 628, pp. 11–18. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-3433-6_2
Singha, J., Das, K.: Recognition of Indian sign language in live video. Int. J. Comput. Appl. 70, 17–22 (2013)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net, pp. 1–14 (2015)
Szegedy, C., Vanhoucke, V., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision (2014)
Tripathi, K., Baranwal, N., Nandi, G.C.: Continuous Indian sign language gesture recognition and sentence formation. Procedia Comput. Sci. 54, 523–531 (2015)
Uddin, S.J.: Bangla sign language interpretation using image processing (2017)
Wazalwar, S., Shrawankar, U.: Interpretation of sign language into English using NLP techniques. J. Inf. Optim. Sci. 38, 895 (2017)
Zoph, B., Shlens, J.: Learning transferable architectures for scalable image recognition (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Paul, P., Bhuiya, M.AUA., Ullah, M.A., Saqib, M.N., Mohammed, N., Momen, S. (2019). A Modern Approach for Sign Language Interpretation Using Convolutional Neural Network. In: Nayak, A., Sharma, A. (eds) PRICAI 2019: Trends in Artificial Intelligence. PRICAI 2019. Lecture Notes in Computer Science(), vol 11672. Springer, Cham. https://doi.org/10.1007/978-3-030-29894-4_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-29894-4_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29893-7
Online ISBN: 978-3-030-29894-4
eBook Packages: Computer ScienceComputer Science (R0)