Skip to main content
SpringerLink
Log in

Sign Language Recognition Based on Notations and Neural Networks

  • Conference paper
  • First Online:
Digital Transformation and Global Society (DTGS 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1242))

Included in the following conference series:

Abstract

Automated translation from sign languages used by the hearing-impaired people worldwide is an important but so far unresolved task ensuring universal communication in the society. In the paper, we propose an original approach to recognizing gestures of the Russian Sign Language ​​based on the combined use of the linguistic Hamburg System of Notations (HamNoSys) and OpenPose library for tracking human movements. Our software based on the specially constructed and trained artificial neural network (ANN) model performs recognition of the two main components commonly identified in gestures: handshape and location (while the hand orientation, the movement and the non-manual component are so far not considered). The recognition accuracy obtained in the experimental validation with the standard Leap Motion SDK hand tracking algorithm was 100% for adult signers and about 76% for the children. Details of the software architecture and the image recognition process with skeletal data are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://сурдофон.рф.

  2. 2.

    https://www.sign-lang.uni-hamburg.de/dgs-korpus/index.php/hamnosys-97.html.

  3. 3.

    http://www.surdoserver.ru/.

  4. 4.

    http://handtalk.me.

  5. 5.

    https://www.wps.de/en/research/delegs/.

  6. 6.

    http://www.spreadthesign.com.

  7. 7.

    https://www.handspeak.com/word/.

  8. 8.

    https://www.leapmotion.com/.

  9. 9.

    https://github.com/CMU-Perceptual-Computing-Lab/openpose.

References

  1. Kendon, A.: Current issues in the study of gesture. the biological foundation of gestures: motor and semiotic aspects. Lawrence Erlbaum Associate, pp. 23–47 (1986)

    Google Scholar 

  2. Keane, J., Sehyr, Z.S., Emmorey, K., Brentari, D.: A theory-driven model of handshape similarity. Phonology 34(2), 221–241 (2017)

    Article  Google Scholar 

  3. Battison, R.: Phonological deletion in American sign language. Sign Lang. Stud. 5(1), 1–19 (1974)

    Article  Google Scholar 

  4. Al-khazraji, S., et al.: Modeling the speed and timing of American sign language to generate realistic animations. In: Proceedings 20th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 259–270 (2018)

    Google Scholar 

  5. Hanke, T.: HamNoSys – representing sign language data in language resources and language processing contexts. In: Proceedings LREC Workshop, Paris: ELRA, pp. 1–6 (2004)

    Google Scholar 

  6. Grif, M.G., Prikhodko, A.L.: Approach to the Sign language gesture recognition framework based on HamNoSys analysis. In: 2018 XIV International Conference on Actual Problems of Electronics Instrument Engineering (APEIE), pp. 426–429 (2018)

    Google Scholar 

  7. Koller, O., Bowden, R., Ney, H.: Automatic alignment of HamNoSys subunits for continuous sign language recognition. In: LREC 2016 Proceedings, pp. 121–128 (2016)

    Google Scholar 

  8. Bragg, D., et al: Sign language recognition, generation, and translation: an interdisciplinary perspective. In: 21st ACM SIGACCESS Conf on Comp and Accessibility, pp. 16–31 (2019)

    Google Scholar 

  9. Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2012). https://doi.org/10.1007/s10462-012-9356-9

    Article  Google Scholar 

  10. Wei, S., Chen, X., Yang, X., Cao, S., Zhang, X.: A component-based vocabulary-extensible sign language gesture recognition framework. Sensors 16(4), 556 (2016)

    Article  Google Scholar 

  11. Konstantinidis, D., et al.: Sign language recognition based on hand and body skeletal data. In Proceedings 2018-3DTV-Conference, pp. 1–4. IEEE (2018)

    Google Scholar 

  12. Ronchetti, F., et al.: LSA64: a dataset of Argentinian sign language. In: XXII Congreso Argentino de Ciencias de la Computación (CACIC) (2016)

    Google Scholar 

  13. Wei, S.E., et al.: Convolutional pose machines. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 4724–4732 (2016)

    Google Scholar 

  14. Bragg, D., et al.: Sign language recognition, generation, and translation: an interdisciplinary perspective. In: Proceedings 21st International ACM SIGACCESS Conference, pp. 16–31 (2019)

    Google Scholar 

  15. Du, Y. et al.: Hand gesture recognition with leap motion. arXiv:1711.04293 (2017)

  16. Kumar, P., et al.: A multimodal framework for sensor based sign language recognition. Neurocomputing 259, 21–38 (2017)

    Article  Google Scholar 

Download references

Acknowledgment

The reported study was funded by RFBR and DST according to the research project No. 19-57-45006.

Author information

Authors and Affiliations

  1. Novosibirsk State Technical University, Novosibirsk, Russia

    Alexey Prikhodko, Mikhail Grif & Maxim Bakaev

Authors
  1. Alexey Prikhodko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikhail Grif
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maxim Bakaev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maxim Bakaev .

Editor information

Editors and Affiliations

  1. National Research University Higher School of Economics, St. Petersburg, Russia

    Daniel A. Alexandrov

  2. ITMO University, St. Petersburg, Russia

    Alexander V. Boukhanovsky

  3. ITMO University, St. Petersburg, Russia

    Andrei V. Chugunov

  4. National Research University Higher School of Economics, St. Petersburg, Russia

    Yury Kabanov

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. National Research University Higher School of Economics, St. Petersburg, Russia

    Ilya Musabirov

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Prikhodko, A., Grif, M., Bakaev, M. (2020). Sign Language Recognition Based on Notations and Neural Networks. In: Alexandrov, D.A., Boukhanovsky, A.V., Chugunov, A.V., Kabanov, Y., Koltsova, O., Musabirov, I. (eds) Digital Transformation and Global Society. DTGS 2020. Communications in Computer and Information Science, vol 1242. Springer, Cham. https://doi.org/10.1007/978-3-030-65218-0_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-65218-0_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-65217-3

  • Online ISBN: 978-3-030-65218-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation