Abstract
In this work, deep transfer learning is proposed for recognition of sign sequence in sentences continuously signed in the Indian sign language using sufficient labelled data of isolated signs and limited amount of labelled sentence data. The data is collected using multiple six degree-of-freedom inertial measurement units (IMUs) on both hands of the signer. The proposed deep learning model consists of convolutional neural network (CNN), two bidirectional long short-term memory (Bi-LSTM) layers and connectionist temporal classification (CTC) to enable end-to-end sentence recognition without requiring the knowledge of sign boundaries. Initially, the network is trained on isolated signs data. Based on the hypothesis that generic features learned from isolated signs will enhance the classification of continuous sentence sign data, a novel transfer learning framework is proposed, wherein last few layers of the pre-trained network are retrained using limited amount of labelled sentence data. Model is assessed under various transferring schemes, different vocabulary sizes and different amounts of labelled sentence data. When the number of observations of each sentence available for training the model is reduced from 10 to just 3, the degradation observed in average classification accuracies without using transfer learning is 54.0%. However, this degradation is reduced up to 11.5% when the proposed deep transfer learning approach is employed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bhagat NK, Vishnusai Y, Rathna GN (2019) Indian sign language gesture recognition using image processing and deep learning. In: 2019 Digital image computing: techniques and applications (DICTA), pp 1–8. IEEE. https://doi.org/10.1109/DICTA47822.2019.8945850
Bird JJ, Ekárt A, Faria DR (2020) British sign language recognition via late fusion of computer vision and leap motion with transfer learning to American sign language. Sensors 20(18):5151. https://doi.org/10.3390/s20185151
Bu Q, Yang G, Ming X, Zhang T, Feng J, Zhang J (2020) Deep transfer learning for gesture recognition with WiFi signals. Pers Ubiquitous Comput. https://doi.org/10.1007/s00779-019-01360-8
Elakkiya R, Selvamani K (2019) Subunit sign modeling framework for continuous sign language recognition. Comput Electr Eng 74:379–390. https://doi.org/10.1016/j.compeleceng.2019.02.012
Fang B, Co J, Zhang M (2017) DeepASL: enabling ubiquitous and non-intrusive word and sentence-level sign language translation. In: Proceedings of the 15th ACM conference on embedded network sensor systems, pp 1–13. https://doi.org/10.1145/3131672.3131693
Graves A, Fernández S, Gomez F, Schmidhuber J (2006) Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on machine learning, pp 369–376. https://doi.org/10.1145/1143844.1143891
Gupta R, Rajan S (2020) Comparative analysis of convolution neural network models for continuous indian sign language classification. Proc Comput Sci 171:1542–1550. https://doi.org/10.1016/j.procs.2020.04.165
Halvardsson G, Peterson J, Soto-Valero C, Baudry B (2020) Interpretation of Swedish sign language using convolutional neural networks and transfer learning. arXiv preprint arXiv:2010.07827
Ibrahim NB, Zayed HH, Selim MM (2020) Advances, challenges and opportunities in continuous sign language recognition. J Eng Appl Sci 15(5):1205–1227. https://doi.org/10.36478/jeasci.2020.1205.1227
Indian Sign Language Dictionary (2015) Ramakrishna mission Vivekananda University, Coimbatore. http://indiansignlanguage.org/dictionary/. Accessed 07 Dec 2020
Jaramillo JC, Murillo-Fuentes JJ, Olmos PM (2018) Boosting handwriting text recognition in small databases with transfer learning. In: 16th International conference on frontiers in handwriting recognition (ICFHR), Niagara Falls, NY, 2018, pp 429–434. https://doi.org/10.1109/ICFHR-2018.2018.00081
Jiang X, Hu B, Chandra Satapathy S, Wang SH, Zhang YD (2020) Fingerspelling identification for Chinese sign language via AlexNet-based transfer learning and Adam optimizer. Sci Program. https://doi.org/10.1155/2020/3291426
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
Mittal A, Kumar P, Roy PP, Balasubramanian R, Chaudhuri BB (2019) A modified LSTM model for continuous sign language recognition using leap motion. IEEE Sens J 19(16):7056–7063. https://doi.org/10.1109/JSEN.2019.2909837
Papastratis I, Dimitropoulos K, Daras P (2021) Continuous sign language recognition through a context-aware generative adversarial network. Sensors 21(7):2437. https://doi.org/10.3390/s21072437
Qi W, Su H, Aliverti A (2020) A smartphone-based adaptive recognition and real-time monitoring system for human activities. IEEE Trans Hum Mach Syst 50(5):414–423. https://doi.org/10.1109/THMS.2020.2984181
Qin CX, Qu D, Zhang LH (2018) Towards end-to-end speech recognition with transfer learning. EURASIP J Audio Speech Music Process. https://doi.org/10.1186/s13636-018-0141-9
Rastgoo R, Kiani K, Escalera S (2020) Hand sign language recognition using multi-view hand skeleton. Expert Syst Appl 150:113336. https://doi.org/10.1016/j.eswa.2020.113336
Saggio G, Cavallo P, Ricci M, Errico V, Zea J, Benalcázar ME (2020) Sign language recognition using wearable electronics: implementing k-nearest neighbors with dynamic time warping and convolutional neural network algorithms. Sensors 20(14):3879. https://doi.org/10.3390/s20143879
Sharma S, Gupta R (2018) On the use of temporal and spectral central moments of forearm surface EMG for finger gesture classification. In: 2018 2nd International conference on micro-electronics and telecommunication engineering (ICMETE). https://doi.org/10.1109/icmete.2018.00059
Shu Y, Zhang D, Chen P, Li Y (2021) Mini neural network based on knowledge distillation for dynamic gesture recognition in real scenes. In: 2021 IEEE international conference on consumer electronics and computer engineering (ICCECE), pp 630–634. IEEE. https://doi.org/10.1109/ICCECE51280.2021.9342127
Sridhar A, Ganesan RG, Kumar P, Khapra M (2020) INCLUDE: a large scale dataset for indian sign language recognition. In: Proceedings of the 28th ACM international conference on multimedia, pp 1366–1375. https://doi.org/10.1145/3394171.3413528
Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A survey on deep transfer learning. In: International conference on artificial neural networks, pp 270–279. Springer, Cham. https://doi.org/10.1007/978-3-030-01424-7_27
Wadhawan A, Kumar P (2019) Sign language recognition systems: a decade systematic literature review. Arch Comput Methods Eng. https://doi.org/10.1007/s11831-019-09384-2
Wadhawan A, Kumar P (2020) Deep learning-based sign language recognition system for static signs. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04691-y
Wang Z, Zhao T, Ma J, Chen H, Liu K, Shao H, Wang Q, Ren J (2020) Hear sign language: a real-time end-to-end sign language recognition system. IEEE Trans Mob Comput. https://doi.org/10.1109/TMC.2020.3038303
Yang W, Tao J, Ye Z (2016) Continuous sign language recognition using level building based on fast hidden Markov model. Pattern Recognit Lett 78:28–35. https://doi.org/10.1016/j.patrec.2016.03.030
Zhang Q, Wang D, Zhao R, Yu Y (2019) MyoSign: enabling end-to-end sign language recognition with wearables. In: Proceedings of the 24th international conference on intelligent user interfaces, pp 650–660. https://doi.org/10.1145/3301275.3302296
Zhang S, Do CT, Doddipatla R, Renals S (2020) Learning noise invariant features through transfer learning for robust end-to-end speech recognition. In: ICASSP 2020–2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 7024–7028. IEEE. https://doi.org/10.1109/ICASSP40776.2020.9053169
Acknowledgements
The author gratefully acknowledges the funding support provided by Science and Engineering Research Board (SERB), from the Department of Science and Technology (DST), (ECR/2016/000637) a statutory body of Government of India. The author also express thanks to all the volunteers for their patience and support while recording the data.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sharma, S., Gupta, R. & Kumar, A. Continuous sign language recognition using isolated signs data and deep transfer learning. J Ambient Intell Human Comput 14, 1531–1542 (2023). https://doi.org/10.1007/s12652-021-03418-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-021-03418-z