A Serious Game for Learning Portuguese Sign Language - “iLearnPSL”

  • Marcus Torres
  • Vítor CarvalhoEmail author
  • Filomena Soares
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 22)


Several thousands of people in Portugal use Portuguese sign language (PSL). Children belonging to this community have difficulties with communication and learning processes. There are some applications that focuses on teaching/learning PSL but with a limited interaction with the user, such as presentation of images of a hand gesture picture or avatar where the user must mimic the respective PSL hand gesture. Some other applications are more advanced using sensors to detect movements and gestures performed by the user. However, these applications do not implement an automatic interaction with the user of PSL. Following this trend, the main idea of this project is to develop a solution where deaf child can learn PSL in an interactive, dynamic and funny way. Therefore, this paper describes the first insights in the development of an interactive and didactic virtual game tool to learn the Portuguese Sign Language, in particular the numbers from 0 to 9. The alphabet and colors will be also included in the future. The target group considers deaf children in the first cycle of education level (primary Portuguese school). The developed game promotes the automatic interaction with the user through the Leap Motion controller. The system captures the hands and fingers gestures performed by the user and the graphical interface returns the adequate feedback. The preliminary results obtained show a good level of user experience with therapists.


Leap motion Portuguese sign language Deaf people Sign recognition Serious game 



This work has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT – “Fundação para a Ciência e Tecnologia” within the Project Scope: UID/CEC/00319/2013.


  1. 1.
    Carvalho, C.A.F.: A narrativa em crianças surdas: papel da Língua Gestual Portuguesa (2013). (in Portuguese)Google Scholar
  2. 2.
    Informação - Comunidade. Accessed 7 July 2015. (in Portuguese)
  3. 3.
    Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the leap motion controller. Sensors (Basel) 13(5), 6380–6393 (2013)CrossRefGoogle Scholar
  4. 4.
    Potter, L.E., Araullo, J., Carter, L.: The leap motion controller. In: Proceedings of the 25th Australian Computer-Human Interaction Conference on Augmentation, Application, Innovation, Collaboration, OzCHI 2013, pp. 175–178 (2013)Google Scholar
  5. 5.
    Guna, J., Jakus, G., Pogačnik, M., Tomažič, S., Sodnik, J.: An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors 14(2), 3702–3720 (2014)CrossRefGoogle Scholar
  6. 6.
    API Overview—Leap Motion C# SDK v2.3 documentation. Accessed 4 Apr 2016
  7. 7.
    Sun, C., Zhang, T., Xu, C.: Latent support vector machine modeling for sign language recognition with kinect. ACM Trans. Intell. Syst. Technol. 6(2), 1–20 (2015)CrossRefGoogle Scholar
  8. 8.
    Zafrulla, Z., Brashear, H., Starner, T., Hamilton, H., Presti, P.: American sign language recognition with the kinect. In: Proceedings of the 13th International Conference on Multimodal Interfaces - ICMI 2011, p. 279 (2011)Google Scholar
  9. 9.
    Funasaka, M., Ishikawa, Y., Takata, M., Joe, K.: Sign language recognition using leap motion controller. In: International Conference on Parallel and Distributed Processing Techniques and Applications, PDPTA 2015, pp. 263–269 (2015)Google Scholar
  10. 10.
    Khelil, B., Amiri, H.: Hand gesture recognition using leap motion controller for recognition of arabic sign language. In: 3rd International Conference on Automation, Control, Engineering and Computer Science, ACECS 2016, Proceedings of Engineering and Technology (PET), pp. 233–238 (2016)Google Scholar
  11. 11.
    Quesada, L., López, G., Guerrero, L.A.: Sign Language Recognition Using Leap Motion, pp. 277–288. Springer, Heidelberg (2015)Google Scholar
  12. 12.
    Chuan, C.-H., Regina, E., Guardino, C.: American Sign Language Recognition Using Leap Motion Sensor. In: 13th International Conference on Machine Learning Application, pp. 541–544 (2014)Google Scholar
  13. 13.
    Simos, M., Nikolaidis, N.: Greek sign language alphabet recognition using the leap motion device. In: Proceedings of the 9th Hellenic Conference on Artificial Intelligence - SETN 2016, pp. 1–4 (2016)Google Scholar
  14. 14.
    Mohandes, M., Aliyu, S., Deriche, M.: Prototype Arabic sign language recognition using multi-sensor data fusion of two leap motion controllers. In: 2015 IEEE 12th International Multi-conference on Systems, Signals and Devices, SSD15, pp. 1–6 (2015)Google Scholar
  15. 15.
    LIVPSIC - Livraria de Psicologia e Ciências da Educação. Accessed 8 July 2015. (in Portuguese)
  16. 16.
    ASP - Associação de Surdos do Porto. Accessed 8 July 2015. (in Portuguese)
  17. 17.
  18. 18.
    Sign 4 Me for iPad - A Signed English Translator para iPhone, iPod touch e iPad na App Store no iTunes. Accessed 8 July 2015
  19. 19.
    ASL American Sign Language – Aplicações Android no Google Play. Accessed 8 July 2015
  20. 20. culturas_surdos created by leticia_nadia_lgp based on kindergarten.!page-8. Accessed 16 Apr 2016
  21. 21.
    RogerVoice, An Android App That Helps The Deaf Have A Conversation On The Phone - Forbes. Accessed 8 July 2015
  22. 22.
    Tech Tackles Sign Language – MotionSavvy - Forbes. Accessed 8 July 2015
  23. 23.
    Leap Motion App Store|Leap Motion Apps for Motion Control. Accessed 16 Apr 2016
  24. 24.
    Fingerspeller|Leap Motion Developers. Accessed 16 Apr 2016
  25. 25.
    Leap Motion App Store|ASL Digits. Accessed 16 Apr 2016
  26. 26.
    Escola Virtual de Língua Gestual Portuguesa - Página Inicial. Accessed 2 Oct 2015. (in Portuguese)
  27. 27.
    Soares, F., Sena Esteves, J., Carvalho, V., Lopes, G., Barbosa, F., Ribeiro, P.: Development of a serious game for Portuguese Sign Language. In: 7th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Brno, Czech Republic, 6–8 October 2015Google Scholar
  28. 28.
    O’Leary, R.: LeapTrainer (2013)Google Scholar
  29. 29.
    Soares, F., Sena Esteves, J., Carvalho, V., Moreira, C., Lourenço, P.: Sign language learning using the hangman videogame. In: 7th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Brno, Czech Republic, 6–8 October 2015Google Scholar
  30. 30.
    ISEP. Accessed 6 Oct 2015
  31. 31.
    Unity - Game engine, tools and multiplatform. Accessed 4 Apr 2016
  32. 32.
    SDK Libraries—Leap Motion Unity SDK v3.1 documentation. Accessed 4 Apr 2016
  33. 33. culturas_surdos created by leticia_nadia_lgp based on kindergarten.!alfabeto-e-números. Accessed 16 Apr 2016
  34. 34.
    Torres, M., Carvalho, V., Soares, F.: iLearnPSL – development of an interactive application for learning Portuguese sign language: first insight. In: CISPEE16, Vila Real, Portugal, 19–21 October 2016Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Marcus Torres
    • 2
  • Vítor Carvalho
    • 1
    • 2
    Email author
  • Filomena Soares
    • 1
  1. 1.R&D ALGORITMI CentreUniversity of MinhoGuimaraesPortugal
  2. 2.IPCA – Polytechnic Institute of Cávado and AveBarcelosPortugal

Personalised recommendations