Skip to main content
SpringerLink
Log in

Signer independent isolated Italian sign recognition based on hidden Markov models

  • Short Paper
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Sign languages represent the most natural way to communicate for deaf and hard of hearing. However, there are often barriers between people using this kind of languages and hearing people, typically oriented to express themselves by means of oral languages. To facilitate the social inclusiveness in everyday life for deaf minorities, technology can play an important role. Indeed many attempts have been recently made by the scientific community to develop automatic translation tools. Unfortunately, not many solutions are actually available for the Italian Sign Language (Lingua Italiana dei Segni—LIS) case study, specially for what concerns the recognition task. In this paper, the authors want to face such a lack, in particular addressing the signer-independent case study, i.e., when the signers in the testing set are to included in the training set. From this perspective, the proposed algorithm represents the first real attempt in the LIS case. The automatic recognizer is based on Hidden Markov Models (HMMs) and video features have been extracted using the OpenCV open source library. The effectiveness of the HMM system is validated by a comparative evaluation with Support Vector Machine approach. The video material used to train the recognizer and testing its performance consists in a database that the authors have deliberately created by involving 10 signers and 147 isolated-sign videos for each signer. The database is publicly available. Computer simulations have shown the effectiveness of the adopted methodology, with recognition accuracies comparable to those obtained by the automatic tools developed for other sign languages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. http://www.bu.edu/asllrp/ncslgr.html.

  2. http://www.atlas.polito.it/index.php/en/home.

  3. http://www.a3lab.dii.univpm.it/projects/a3lis.

  4. http://www.ensancona.it.

References

  1. von Agris, U, Kraiss, KF (2007) Towards a video corpus for signer-independent continuous sign language recognition. In: Proceedings of the International Workshop on Gesture in Human–Computer Interaction and Simulation, Lisbon

  2. von Agris U, Zieren J, Canzler U, Bauer B, Kraiss KF (2008) Recent developments in visual sign language recognition. Univers Access Inf Soc 6(4):323–362

    Article  Google Scholar 

  3. Al-Ahdal M, Tahir N (2012) Review in sign language recognition systems. In: Proceedings of IEEE Symposium on Computers Informatics, pp 52–57

  4. Bertoldi N, Tiotto G, Prinetto P, Piccolo E, Nunnari F, Lombardo V, Mazzei A, Damiano R, Lesmo L, Del Principe A (2010) On the creation and the annotation of a large-scale Italian-LIS parallel corpus. In: Proceedings of the International Conference on Language Resources and Evaluation, pp 19–22

  5. Bishop C (2006) Pattern recognition and machine learning. Springer Science+Business Media, LLC, New York

    MATH  Google Scholar 

  6. Bradski G (2000) The OpenCV library. Dr Dobb’s J Softw Tools 25(11):120 (122–125)

    Google Scholar 

  7. Bradski G, Kaehler A (2008) Learning OpenCV: computer vision with the OpenCV library. O’Reilly Media Inc., Sebastopol

    Google Scholar 

  8. Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698

    Article  Google Scholar 

  9. Chang CC, Lin CJ (2011) LIBSVM: A library for support vector machines. ACM Trans Intell Syst Technol 2:1–27

    Article  Google Scholar 

  10. Cortes C, Vapnik V (1995) Support-vector networks. In: Machine Learning, pp 273–297

  11. Dreuw P, Forster J, Deselaers T, Ney H (2008) Efficient approximations to model-based joint tracking and recognition of continuous sign language. In: Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition, Los Alamitos, pp 1–6

  12. Dreuw P, Neidle C, Sclaroff VAS, Ney H (2008) Proc Int Conf Lang Resour Eval., Benchmark databases for video-based automatic sign language recognitionEuropean Language Resources Association, Marrakech

    Google Scholar 

  13. Dreuw P, Ney H (2008) Visual modeling and feature adaptation in sign language recognition. In: ITG Conference on Voice Communication (SprachKommunikation), pp 1–4

  14. Dreuw P, Ney H, Martinez G, Crasborn O, Piater J, Moya JM, Wheatley M (2010) The signspeak project—bridging the gap between signers and speakers. In: Proceedings of the International Conference on Language Resources and Evaluation. Valletta, Malta

  15. Dreuw P, Rybach D, Deselaers T, Zahedi M, Ney H (2007) Speech recognition techniques for a sign language recognition system. In: Proceedings of Interspeech, pp 2513–2516

  16. Fagiani M, Principi E, Squartini S, Piazza F (2012) A new Italian sign language database. In: Zhang H, Hussain A, Liu D, Wang Z (eds) Advances in brain inspired cognitive systems, Lecture Notes in Computer Science, vol 7366. Springer, pp 164–173

  17. Fagiani M, Principi E, Squartini S, Piazza F (2013) A new system for automatic recognition of italian sign language. In: Apolloni B, Bassis S, Esposito A, Morabito FC (eds) Neural nets and surroundings, smart innovation, systems and technologies, vol 19. Springer, Berlin, pp 69–79

  18. Fang G, Gao W, Ma J (2001) Signer-independent sign language recognition based on SOFM/HMM. In: Proceedings of the IEEE ICCV workshop on recognition, analysis, and tracking of faces and gestures in real-time systems, pp 90–95

  19. Gonzalez RC, Woods RE (2007) Digital image processing, 3rd edn. Prentice Hall, USA

    Google Scholar 

  20. Gweth Y, Plahl C, Ney H (2012) Enhanced continuous sign language recognition using PCA and neural network features. In: Proceedings of the computer vision and pattern recognition workshops, pp 55–60

  21. Haasdonk B, Bahlmann C (2004) Pattern recognition. In: Rasmussen C, Bülthoff HH, Schölkopf B, Giese MA (eds) Learning with distance substitution kernels., Lecture notes in computer scienceSpringer, Berlin, pp 220–227

    Google Scholar 

  22. Hanke T (2004) HamNoSys-representing sign language data in language resources and language processing contexts. In: Proceedings of LREC, pp 1–6

  23. Hu MK (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8(2):179–187

    Article  MATH  Google Scholar 

  24. Infantino I, Rizzo R, Gaglio S (2007) A framework for sign language sentence recognition by commonsense context. IEEE Trans Syst Man Cybern C Appl Rev 37(5):1034–1039

    Article  Google Scholar 

  25. Kelly D, McDonald J, Markham C (2010) A person independent system for recognition of hand postures used in sign language. Pattern Recognit Lett 31(11):1359–1368

    Article  Google Scholar 

  26. Kukharev G, Nowosielski A (2004) Visitor identification—elaborating real time face recognition system. In: Proceedings of the international conference on computer graphics, visualization and computer vision. UNION Agency, Plzen, pp 157–164

  27. Maebatake M, Suzuki I, Nishida M, Horiuchi Y, Kuroiwa S (2008) Sign language recognition based on position and movement using multi-stream HMM. In: Proceedings of the international symposium on universal communication. Los Alamitos, pp 478–481

  28. Quan Y (2010) Chinese sign language recognition based on video sequence appearance modeling. In: Proceedings of the IEEE Conference on Industrial Electronics and Applications, pp 1537–1542

  29. Rabiner L, Juang BH (1993) Fundamentals of speech recognition. Prentice-Hall Inc., Upper Saddle River

    Google Scholar 

  30. Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(62):257–286

    Article  Google Scholar 

  31. Ripley BD (1996) Pattern recognition and neural networks. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  32. Sandler W, Lillo-Martin D (2006) Sign language and linguistic universals. Cambridge University Press, Cambridge

    Book  Google Scholar 

  33. Saon G, Chien JT (2012) Large-vocabulary continuous speech recognition systems: a look at some recent advances. IEEE Signal Process Mag 29(6):18–33

    Article  Google Scholar 

  34. Serra J (1983) Image analysis and mathematical morphology. Academic Press Inc., Orlando

    Google Scholar 

  35. Starner T, Pentland A (1995) Real-time American sign language recognition from video using hidden Markov models. In: Proceedings of the international symposium on computer vision, pp 265–270

  36. Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375

    Article  Google Scholar 

  37. Theodorakis S, Katsamanis A, Maragos P (2009) Product-HMMs for automatic sign language recognition. In: Proceedings of the international conference on acoustics, speech and signal processing. Taipei, Taiwan, pp. 1601–1604

  38. Theodoridis S, Koutroumbas K (2008) Pattern recognition, 4th edn. Academic Press, Burlington

    Google Scholar 

  39. Vezhnevets V, Sazonov V, Andreeva A (2003) A survey on pixel-based skin color detection techniques. In: Proceedings of GraphiCon, pp 85–92

  40. Young SJ, Evermann G, Gales MJF, Hain T, Kershaw D, Moore G, Odell J, Ollason D, Povey D, Valtchev V, Woodland PC (2006) The HTK Book, version 3.4. Cambridge University Press, Cambridge

    Google Scholar 

  41. Zahedi M, Keysers D, Deselaers T, Ney H (2005) Pattern recognition. In: Kropatsch W, Sablatnig R, Hanbury A (eds) Combination of tangent distance and an image distortion model for appearance-based sign language recognition., Lecture Notes in Computer ScienceSpringer, Berlin, pp 401–408

    Google Scholar 

  42. Zaki MM, Shaheen SI (2011) Sign language recognition using a combination of new vision based features. Pattern Recognit. Lett. 32(4):572–577

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Engineering, Università Politecnica delle Marche, Via Brecce Bianche 1, 60131, Ancona, Italy

    Marco Fagiani, Emanuele Principi, Stefano Squartini & Francesco Piazza

Authors
  1. Marco Fagiani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emanuele Principi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefano Squartini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francesco Piazza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emanuele Principi.

Appendix: HamNoSys Sign Transcription

Appendix: HamNoSys Sign Transcription

figure a
figure b

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fagiani, M., Principi, E., Squartini, S. et al. Signer independent isolated Italian sign recognition based on hidden Markov models. Pattern Anal Applic 18, 385–402 (2015). https://doi.org/10.1007/s10044-014-0400-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-014-0400-z

Keywords

Search

Navigation