Skip to main content
SpringerLink
Log in

An Adaptive Thresholding-Based Movement Epenthesis Detection Technique Using Hybrid Feature Set for Continuous Fingerspelling Recognition

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Sign language recognition systems are gaining importance in recent times as these have established themselves as important elements of human–computer interaction. Also these provide an opportunity for the deaf and hearing impaired to communicate with the common people without the need of an interpreter. Yet there are plenty of challenges in this field which are worth exploring and solutions formulated. In this paper, we have addressed the design of a continuous fingerspelling recognition system which segments a fingerspelling sequence into meaningful extracts and non-sign patterns and thereby recognizes the meaningful signs. Sign segmentation is carried out by means of a unique set of features which comprises of shape matching, velocity change and displacement of centroid between successive frames. Specialized techniques like adaptive thresholding and finite state machine model are also incorporated into our system for efficient classification of sign and movement epenthesis frames. We have validated the performance of our proposed system taking into account the continuous fingerspelling alphabets of American Sign Language considering both native and non-native signers and have obtained an accuracy of almost 91.29%. Our proposed system also has the potential to tackle complex backgrounds involving multiple objects, backgrounds with multiple signers and different brightness conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Bhuyan M, Ghosh D, Bora P. Co-articulation detection in hand gestures. TENCON 2005-2005 IEEE region 10 conference. 2005; pp. 1–4. IEEE.

  2. Bhuyan MK, Kumar DA, MacDorman KF, Iwahori Y. A novel set of features for continuous hand gesture recognition. J. Multimod User Interfaces. 2014;8(4):333–43.

    Article  Google Scholar 

  3. Bradski G, Kaehler A. Learning OpenCV: computer vision with the OpenCV library. Newton: O’Reilly Media, Inc.; 2008.

    Google Scholar 

  4. Chen Q, Georganas ND, Petriu EM. Hand gesture recognition using Haar-like features and a stochastic context-free grammar. IEEE Trans Instrum Meas. 2008;57(8):1562–71.

    Article  Google Scholar 

  5. Chetverikov D. A simple and efficient algorithm for detection of high curvature points in planar curves. International conference on computer analysis of images and patterns. Berlin: Springer; 2003. p. 746–53.

  6. Choudhury A, Talukdar AK, Bhuyan MK, Sarma KK. Movement epenthesis detection for continuous sign language recognition. J Intell Syst. 2017;26(3):471–81.

    Article  Google Scholar 

  7. Choudhury A, Talukdar AK, Sarma KK. A conditional random field based Indian sign language recognition system under complex background. 2014 fourth international conference on communication systems and network technologies.2014; p. 900–4. IEEE.

  8. Choudhury A, Talukdar AK, Sarma KK. A review on vision-based hand gesture recognition and applications. Intelligent applications for heterogeneous system modeling and design. IGI Global. 2015; p. 256–81.

  9. Dalitz C, Brandt C, Goebbels S, Kolanus D. Fourier descriptors for broken shapes. EURASIP J Adv Signal Process. 2013;2013(1):161.

    Article  Google Scholar 

  10. Gonzalez R, Woods R. Processing digital image. Chennai: Pearson Education India; 2009.

    Google Scholar 

  11. Hayes MH. Statistical digital signal processing and modeling. New York: Wiley; 2009.

    Google Scholar 

  12. Hse H, Newton AR. Sketched symbol recognition using zernike moments. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004, vol. 1, 2004; pp. 367–370. IEEE.

  13. Hsu CW, Lin CJ. A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw. 2002;13(2):415–25.

    Article  Google Scholar 

  14. Jerde TE, Soechting JF, Flanders M. Coarticulation in fluent fingerspelling. J Neurosci. 2003;23(6):2383–93.

    Article  Google Scholar 

  15. Karray F, Alemzadeh M, Saleh JA, Arab MN. Human–computer interaction: overview on state of the art. Citeseer; 2008.

  16. Kelly D, McDonald J, Markham C. Recognizing spatiotemporal gestures and movement epenthesis in sign language. In: 2009 13th international machine vision and image processing conference, 2009; pp. 145–150. IEEE.

  17. Khotanzad A, Hong YH. Invariant image recognition by Zernike moments. IEEE Trans Pattern Anal Mach Intell. 1990;12(5):489–97.

    Article  Google Scholar 

  18. Kim T, Shakhnarovich G, Livescu K. Fingerspelling recognition with semi-Markov conditional random fields. In: Proceedings of the IEEE international conference on computer vision, 2013; pp. 1521–1528.

  19. Kumar AA. Fundamentals of digital circuits. New Delhi: PHI Learning Pvt. Ltd; 2016.

    Google Scholar 

  20. Kumar P, Saini R, Behera SK, Dogra DP, Roy PP. Real-time recognition of sign language gestures and air-writing using leap motion. In: 2017 fifteenth IAPR international conference on machine vision applications (MVA), 2017; pp. 157–160. IEEE.

  21. Lee HK, Kim JH. An HMM-based threshold model approach for gesture recognition. IEEE Trans Pattern Anal Mach Intell. 1999;21(10):961–73.

    Article  Google Scholar 

  22. Lucas BD, Kanade T, et al. An iterative image registration technique with an application to stereo vision. British Columbia,Vancouver; 1981.

  23. Neustaedter C. An evaluation of optical flow using Lucas and Kanade’s algorithm. Calgary: University of Calgary Department of Computer Science; 2002.

    Google Scholar 

  24. Ormel E, Crasborn O, van der Kooij E. Coarticulation of hand height in sign language of the Netherlands is affected by contact type. J Phonet. 2013;41(3–4):156–71.

    Article  Google Scholar 

  25. Oz C, Leu MC. Linguistic properties based on American sign language isolated word recognition with artificial neural networks using a sensory glove and motion tracker. Neurocomputing. 2007;70(16–18):2891–901.

    Article  Google Scholar 

  26. Papoulis A, Pillai SU. Probability, random variables, and stochastic processes. New York: Tata McGraw-Hill Education; 2002.

    Google Scholar 

  27. Phung SL, Bouzerdoum A, Chai D. Skin segmentation using color pixel classification: analysis and comparison. IEEE Trans Pattern Anal Mach Intell. 2005;27(1):148–54.

    Article  Google Scholar 

  28. Ricco S, Tomasi C. Fingerspelling recognition through classification of letter-to-letter transitions. In: Asian conference on computer vision, 2009; pp. 214–225. Springer.

  29. Segouat J, Braffort A. Toward modeling sign language coarticulation. In: International gesture workshop. Springer; 2009; p. 325–36.

  30. Teague MR. Image analysis via the general theory of moments. JOSA. 1980;70(8):920–30.

    Article  MathSciNet  Google Scholar 

  31. Wilson JN, Ritter GX. Handbook of computer vision algorithms in image algebra. New York: CRC Press; 2000.

    Book  Google Scholar 

  32. Yang HD, Lee SW. Robust sign language recognition with hierarchical conditional random fields. In: 2010 20th international conference on pattern recognition, 2010; pp. 2202–2205. IEEE.

  33. Yang HD, Sclaroff S, Lee SW. Sign language spotting with a threshold model based on conditional random fields. IEEE Trans Pattern Anal Mach Intell. 2008;31(7):1264–77.

    Article  Google Scholar 

  34. Yang HD, Sclaroff S, Lee SW. Sign language spotting with a threshold model based on conditional random fields. IEEE Trans Pattern Anal Mach Intell. 2009;31(7):1264–77.

    Article  Google Scholar 

  35. Yang R, Sarkar S. Detecting coarticulation in sign language using conditional random fields. In: 18th international conference on pattern recognition (ICPR’06), 2006; vol. 2, pp. 108–112. IEEE.

  36. Yang R, Sarkar S, Loeding B. Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. IEEE Trans Pattern Anal Mach Intell. 2010;32(3):462–77.

    Article  Google Scholar 

  37. Yang W, Tao J, Ye Z. Continuous sign language recognition using level building based on fast hidden Markov model. Pattern Recogn Lett. 2016;78:28–35.

    Article  Google Scholar 

  38. Zhang D, Lu G. Review of shape representation and description techniques. Pattern Recogn. 2004;37(1):1–19.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Gauhati University, Guwahati, Assam, India

    Ananya Choudhury & Kandarpa Kumar Sarma

  2. Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Guwahati, Assam, India

    Anjan Kumar Talukdar & M. K. Bhuyan

Authors
  1. Ananya Choudhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anjan Kumar Talukdar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kandarpa Kumar Sarma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. K. Bhuyan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anjan Kumar Talukdar.

Ethics declarations

Conflict of interest

On behalf of all authors, the co****rresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choudhury, A., Talukdar, A.K., Sarma, K.K. et al. An Adaptive Thresholding-Based Movement Epenthesis Detection Technique Using Hybrid Feature Set for Continuous Fingerspelling Recognition. SN COMPUT. SCI. 2, 128 (2021). https://doi.org/10.1007/s42979-021-00544-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00544-5

Keywords

Search

Navigation