Pattern Analysis and Applications

, Volume 18, Issue 2, pp 385–402 | Cite as

Signer independent isolated Italian sign recognition based on hidden Markov models

  • Marco Fagiani
  • Emanuele Principi
  • Stefano Squartini
  • Francesco Piazza
Short Paper
First Online:
Received:
Accepted:

Abstract

Sign languages represent the most natural way to communicate for deaf and hard of hearing. However, there are often barriers between people using this kind of languages and hearing people, typically oriented to express themselves by means of oral languages. To facilitate the social inclusiveness in everyday life for deaf minorities, technology can play an important role. Indeed many attempts have been recently made by the scientific community to develop automatic translation tools. Unfortunately, not many solutions are actually available for the Italian Sign Language (Lingua Italiana dei Segni—LIS) case study, specially for what concerns the recognition task. In this paper, the authors want to face such a lack, in particular addressing the signer-independent case study, i.e., when the signers in the testing set are to included in the training set. From this perspective, the proposed algorithm represents the first real attempt in the LIS case. The automatic recognizer is based on Hidden Markov Models (HMMs) and video features have been extracted using the OpenCV open source library. The effectiveness of the HMM system is validated by a comparative evaluation with Support Vector Machine approach. The video material used to train the recognizer and testing its performance consists in a database that the authors have deliberately created by involving 10 signers and 147 isolated-sign videos for each signer. The database is publicly available. Computer simulations have shown the effectiveness of the adopted methodology, with recognition accuracies comparable to those obtained by the automatic tools developed for other sign languages.

Keywords

Italian sign language (LIS) Automatic sign recognition Video feature extraction Hidden Markov models 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    von Agris, U, Kraiss, KF (2007) Towards a video corpus for signer-independent continuous sign language recognition. In: Proceedings of the International Workshop on Gesture in Human–Computer Interaction and Simulation, LisbonGoogle Scholar
  2. 2.
    von Agris U, Zieren J, Canzler U, Bauer B, Kraiss KF (2008) Recent developments in visual sign language recognition. Univers Access Inf Soc 6(4):323–362CrossRefGoogle Scholar
  3. 3.
    Al-Ahdal M, Tahir N (2012) Review in sign language recognition systems. In: Proceedings of IEEE Symposium on Computers Informatics, pp 52–57Google Scholar
  4. 4.
    Bertoldi N, Tiotto G, Prinetto P, Piccolo E, Nunnari F, Lombardo V, Mazzei A, Damiano R, Lesmo L, Del Principe A (2010) On the creation and the annotation of a large-scale Italian-LIS parallel corpus. In: Proceedings of the International Conference on Language Resources and Evaluation, pp 19–22Google Scholar
  5. 5.
    Bishop C (2006) Pattern recognition and machine learning. Springer Science+Business Media, LLC, New YorkMATHGoogle Scholar
  6. 6.
    Bradski G (2000) The OpenCV library. Dr Dobb’s J Softw Tools 25(11):120 (122–125)Google Scholar
  7. 7.
    Bradski G, Kaehler A (2008) Learning OpenCV: computer vision with the OpenCV library. O’Reilly Media Inc., SebastopolGoogle Scholar
  8. 8.
    Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698CrossRefGoogle Scholar
  9. 9.
    Chang CC, Lin CJ (2011) LIBSVM: A library for support vector machines. ACM Trans Intell Syst Technol 2:1–27CrossRefGoogle Scholar
  10. 10.
    Cortes C, Vapnik V (1995) Support-vector networks. In: Machine Learning, pp 273–297Google Scholar
  11. 11.
    Dreuw P, Forster J, Deselaers T, Ney H (2008) Efficient approximations to model-based joint tracking and recognition of continuous sign language. In: Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition, Los Alamitos, pp 1–6Google Scholar
  12. 12.
    Dreuw P, Neidle C, Sclaroff VAS, Ney H (2008) Proc Int Conf Lang Resour Eval., Benchmark databases for video-based automatic sign language recognitionEuropean Language Resources Association, MarrakechGoogle Scholar
  13. 13.
    Dreuw P, Ney H (2008) Visual modeling and feature adaptation in sign language recognition. In: ITG Conference on Voice Communication (SprachKommunikation), pp 1–4Google Scholar
  14. 14.
    Dreuw P, Ney H, Martinez G, Crasborn O, Piater J, Moya JM, Wheatley M (2010) The signspeak project—bridging the gap between signers and speakers. In: Proceedings of the International Conference on Language Resources and Evaluation. Valletta, MaltaGoogle Scholar
  15. 15.
    Dreuw P, Rybach D, Deselaers T, Zahedi M, Ney H (2007) Speech recognition techniques for a sign language recognition system. In: Proceedings of Interspeech, pp 2513–2516Google Scholar
  16. 16.
    Fagiani M, Principi E, Squartini S, Piazza F (2012) A new Italian sign language database. In: Zhang H, Hussain A, Liu D, Wang Z (eds) Advances in brain inspired cognitive systems, Lecture Notes in Computer Science, vol 7366. Springer, pp 164–173Google Scholar
  17. 17.
    Fagiani M, Principi E, Squartini S, Piazza F (2013) A new system for automatic recognition of italian sign language. In: Apolloni B, Bassis S, Esposito A, Morabito FC (eds) Neural nets and surroundings, smart innovation, systems and technologies, vol 19. Springer, Berlin, pp 69–79Google Scholar
  18. 18.
    Fang G, Gao W, Ma J (2001) Signer-independent sign language recognition based on SOFM/HMM. In: Proceedings of the IEEE ICCV workshop on recognition, analysis, and tracking of faces and gestures in real-time systems, pp 90–95Google Scholar
  19. 19.
    Gonzalez RC, Woods RE (2007) Digital image processing, 3rd edn. Prentice Hall, USAGoogle Scholar
  20. 20.
    Gweth Y, Plahl C, Ney H (2012) Enhanced continuous sign language recognition using PCA and neural network features. In: Proceedings of the computer vision and pattern recognition workshops, pp 55–60Google Scholar
  21. 21.
    Haasdonk B, Bahlmann C (2004) Pattern recognition. In: Rasmussen C, Bülthoff HH, Schölkopf B, Giese MA (eds) Learning with distance substitution kernels., Lecture notes in computer scienceSpringer, Berlin, pp 220–227Google Scholar
  22. 22.
    Hanke T (2004) HamNoSys-representing sign language data in language resources and language processing contexts. In: Proceedings of LREC, pp 1–6Google Scholar
  23. 23.
    Hu MK (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8(2):179–187CrossRefMATHGoogle Scholar
  24. 24.
    Infantino I, Rizzo R, Gaglio S (2007) A framework for sign language sentence recognition by commonsense context. IEEE Trans Syst Man Cybern C Appl Rev 37(5):1034–1039CrossRefGoogle Scholar
  25. 25.
    Kelly D, McDonald J, Markham C (2010) A person independent system for recognition of hand postures used in sign language. Pattern Recognit Lett 31(11):1359–1368CrossRefGoogle Scholar
  26. 26.
    Kukharev G, Nowosielski A (2004) Visitor identification—elaborating real time face recognition system. In: Proceedings of the international conference on computer graphics, visualization and computer vision. UNION Agency, Plzen, pp 157–164Google Scholar
  27. 27.
    Maebatake M, Suzuki I, Nishida M, Horiuchi Y, Kuroiwa S (2008) Sign language recognition based on position and movement using multi-stream HMM. In: Proceedings of the international symposium on universal communication. Los Alamitos, pp 478–481Google Scholar
  28. 28.
    Quan Y (2010) Chinese sign language recognition based on video sequence appearance modeling. In: Proceedings of the IEEE Conference on Industrial Electronics and Applications, pp 1537–1542Google Scholar
  29. 29.
    Rabiner L, Juang BH (1993) Fundamentals of speech recognition. Prentice-Hall Inc., Upper Saddle RiverGoogle Scholar
  30. 30.
    Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(62):257–286CrossRefGoogle Scholar
  31. 31.
    Ripley BD (1996) Pattern recognition and neural networks. Cambridge University Press, CambridgeCrossRefMATHGoogle Scholar
  32. 32.
    Sandler W, Lillo-Martin D (2006) Sign language and linguistic universals. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  33. 33.
    Saon G, Chien JT (2012) Large-vocabulary continuous speech recognition systems: a look at some recent advances. IEEE Signal Process Mag 29(6):18–33CrossRefGoogle Scholar
  34. 34.
    Serra J (1983) Image analysis and mathematical morphology. Academic Press Inc., OrlandoGoogle Scholar
  35. 35.
    Starner T, Pentland A (1995) Real-time American sign language recognition from video using hidden Markov models. In: Proceedings of the international symposium on computer vision, pp 265–270Google Scholar
  36. 36.
    Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375CrossRefGoogle Scholar
  37. 37.
    Theodorakis S, Katsamanis A, Maragos P (2009) Product-HMMs for automatic sign language recognition. In: Proceedings of the international conference on acoustics, speech and signal processing. Taipei, Taiwan, pp. 1601–1604Google Scholar
  38. 38.
    Theodoridis S, Koutroumbas K (2008) Pattern recognition, 4th edn. Academic Press, BurlingtonGoogle Scholar
  39. 39.
    Vezhnevets V, Sazonov V, Andreeva A (2003) A survey on pixel-based skin color detection techniques. In: Proceedings of GraphiCon, pp 85–92Google Scholar
  40. 40.
    Young SJ, Evermann G, Gales MJF, Hain T, Kershaw D, Moore G, Odell J, Ollason D, Povey D, Valtchev V, Woodland PC (2006) The HTK Book, version 3.4. Cambridge University Press, CambridgeGoogle Scholar
  41. 41.
    Zahedi M, Keysers D, Deselaers T, Ney H (2005) Pattern recognition. In: Kropatsch W, Sablatnig R, Hanbury A (eds) Combination of tangent distance and an image distortion model for appearance-based sign language recognition., Lecture Notes in Computer ScienceSpringer, Berlin, pp 401–408Google Scholar
  42. 42.
    Zaki MM, Shaheen SI (2011) Sign language recognition using a combination of new vision based features. Pattern Recognit. Lett. 32(4):572–577CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Marco Fagiani
    • 1
  • Emanuele Principi
    • 1
  • Stefano Squartini
    • 1
  • Francesco Piazza
    • 1
  1. 1.Department of Information EngineeringUniversità Politecnica delle MarcheAnconaItaly

Personalised recommendations