Skip to main content

A Serious Mobile Game with Visual Feedback for Training Sibilant Consonants

Part of the Lecture Notes in Computer Science book series (LNISA,volume 10714)

Abstract

The distortion of sibilant sounds is a common type of speech sound disorder (SSD) in Portuguese speaking children. Speech and language pathologists (SLP) frequently use the isolated sibilants exercise to assess and treat this type of speech errors.

While technological solutions like serious games can help SLPs to motivate the children on doing the exercises repeatedly, there is a lack of such games for this specific exercise. Another important aspect is that given the usual small number of therapy sessions per week, children are not improving at their maximum rate, which is only achieved by more intensive therapy.

We propose a serious game for mobile platforms that allows children to practice their isolated sibilants exercises at home to correct sibilant distortions. This will allow children to practice their exercises more frequently, which can lead to faster improvements. The game, which uses an automatic speech recognition (ASR) system to classify the child sibilant productions, is controlled by the child’s voice in real time and gives immediate visual feedback to the child about her sibilant productions.

In order to keep the computation on the mobile platform as simple as possible, the game has a client-server architecture, in which the external server runs the ASR system. We trained it using raw Mel frequency cepstral coefficients, and we achieved very good results with an accuracy test score of above \(91\%\) using support vector machines.

Keywords

  • Speech And Language Pathologist (SLPs)
  • Sibilant Sounds
  • Mel-frequency Cepstral Coefficients (MFCCs)
  • Phonological Disorders
  • Examinee's True Score

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-76270-8_30
  • Chapter length: 21 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   179.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-76270-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   229.00
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.

References

  1. American Speech-Language-Hearing Association (ASHA) - Speech Sound Disorders: Articulation and Phonological Processes. http://www.asha.org/public/speech/disorders/SpeechSoundDisorders/. Accessed 28 July 2017

  2. Articulation Station. http://littlebeespeech.com/articulation_station.php. Accessed 5 Jan 2016

  3. ARTUR - the ARticulation TUtoR. http://www.speech.kth.se/multimodal/ARTUR/. Accessed 16 Jan 2016

  4. Barratt, J., Littlejohns, P., Thompson, J.: Trial of intensive compared with weekly speech therapy in preschool children. Arch. Dis. Child. 67(1), 106–108 (1992)

    CrossRef  Google Scholar 

  5. Sanjit, K., Bhogal, R.T., Speechley, M.: Intensity of aphasia therapy, impact on recovery. Stroke 34(4), 987–993 (2003)

    CrossRef  Google Scholar 

  6. Carvalho, M.I.P., et al.: Interactive game for the training of Portuguese vowels (2012)

    Google Scholar 

  7. Denes, G., et al.: Intensive versus regular speech therapy in global aphasia: a controlled study. Aphasiology 10(4), 385–394 (1996)

    CrossRef  Google Scholar 

  8. Falar a Brincar. https://falarabrincar.wordpress.com/. Accessed 16 Jan 2016

  9. Freepik. http://www.freepik.com/. Accessed 28 July 2017

  10. Ganapathy, S., Thomas, S., Hermansky, H.: Comparison of modulation features for phoneme recognition. In: 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 5038–5041. IEEE (2010)

    Google Scholar 

  11. Guimarães, I.: Ciência e Arte da Voz Humana. Escola Superior de Saúde de Alcoitão (2007)

    Google Scholar 

  12. Hall, P.K., Jordan, L.S., Robin, D.A.: Developmental apraxia of speech: theory and clinical practice, p. 200. Pro Ed (1993)

    Google Scholar 

  13. Image Quiz: Vocal Tract. http://www.imagequiz.co.uk/img?img_id=ag5zfmltYWdlbGVhcm5lcnIQCxIHUXVpenplcxifotErDA. Accessed 30 July 2017

  14. Kreimer, S.: Intensive speech and language therapy found to benefit patients with chronic aphasia after stroke. Neurol. Today 17(12), 12–13 (2017)

    CrossRef  Google Scholar 

  15. Lan, T., et al.: Flappy voice: an interactive game for childhood apraxia of speech therapy. In: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play, pp. 429–430. ACM (2014)

    Google Scholar 

  16. Lopes, M., Magalhães, J., Cavaco, S.: A voice-controlled serious game for the sustained vowel exercise. In: Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology, p. 32. ACM (2016)

    Google Scholar 

  17. Matejka, P., Schwarz, P., et al.: Analysis of feature extraction and channel compensation in a GMM speaker recognition system. IEEE Trans. Audio Speech Lang. Process. 15(7), 1979–1986 (2007)

    CrossRef  Google Scholar 

  18. Mobile Operating System Market Share in Portugal, 2016 to 2017. http://gs.statcounter.com/os-market-share/mobile/portugal/#yearly-2016-2017-bar. Accessed 25 July 2017

  19. Nefian, A.V., et al.: A coupled HMM for audio-visual speech recognition. In: 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), vol. 2, pp. II–2013. IEEE (2002)

    Google Scholar 

  20. Nwe, T.L., Foo, S.W., De Silva, L.C.: Speech emotion recognition using hidden Markov models. Speech Commun. 41(4), 603–623 (2003)

    CrossRef  Google Scholar 

  21. Parnandi, A., et al.: Development of a remote therapy tool for childhood apraxia of speech. ACM Trans. Access. Comput. (TACCESS) 7(3), 10 (2015)

    Google Scholar 

  22. Preston, J., Edwards, M.L.: Phonological awareness and types of sound errors in preschoolers with speech sound disorders. J. Speech Lang. Hear. Res. 53(1), 44–60 (2010)

    CrossRef  Google Scholar 

  23. Rubin, Z., Kurniawan, S.: Speech adventure: using speech recognition for cleft speech therapy. In: Proceedings of the 6th International Conference on Pervasive Technologies Related to Assistive Environments, p. 35. ACM (2013)

    Google Scholar 

  24. Sharma, S., et al.: Feature extraction using non-linear transformation for robust speech recognition on the Aurora database. In: Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2000, vol. 2, pp. II1117–II1120. IEEE (2000)

    Google Scholar 

  25. Shriberg, L.D., Paul, R., Flipsen, P.: Childhood speech sound disorders: from postbehaviorism to the postgenomic era. In: Speech Sound Disorders in Children, pp. 1–33 (2009)

    Google Scholar 

  26. Talker. http://speech-trainer.com/children-speech-therapy. Accessed 15 Jan 2016

  27. Tan, C.T., et al.: sPeAK-MAN: towards popular gameplay for speech therapy. In: Proceedings of the 9th Australasian Conference on Interactive Entertainment: Matters of Life and Death, p. 28. ACM (2013)

    Google Scholar 

  28. The Statistics Portal - Forecast of tablet user numbers in Portugal from 2014 to 2021 (in million users). https://www.statista.com/statistics/566416/predicted-number-of-tablet-users-portugal/. Accessed 02 Feb 2016

  29. The Statistics Portal - Forecast of the tablet user penetration rate in Portugal from 2014 to 2021. https://www.statista.com/statistics/568594/predicted-tablet-user-penetration-rate-in-portugal/. Accessed 02 Feb 2016

  30. VITHEA - Virtual Therapist for Aphasia treatment. https://vithea.l2f.inesc-id.pt/wiki/index.php/Main_Page. Accessed 16 Jan 2016

Download references

Acknowledgments

This work was supported by the Portuguese Foundation for Science and Technology under projects BioVisualSpeech (CMUP-ERI/TIC/0033/2014) and NOVA-LINCS (PEest/UID/CEC/04516/2013).

We thank the SLPs Diana Lança and Catarina Duarte for their availability and feedback. We also thank all the 3rd and 4th year SLP students from Escola Superior de Saúde do Alcoitão who collaborated in the data collection task. Many thanks also to Inês Jorge for the graphic design of the game scenarios. Finnally, we would like to thank the schools from Agrupamento de Escolas de Almeida Garrett, and all the children who participated in the recordings.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ivo Anjos .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Anjos, I., Grilo, M., Ascensão, M., Guimarães, I., Magalhães, J., Cavaco, S. (2018). A Serious Mobile Game with Visual Feedback for Training Sibilant Consonants. In: Cheok, A., Inami, M., Romão, T. (eds) Advances in Computer Entertainment Technology. ACE 2017. Lecture Notes in Computer Science(), vol 10714. Springer, Cham. https://doi.org/10.1007/978-3-319-76270-8_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-76270-8_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-76269-2

  • Online ISBN: 978-3-319-76270-8

  • eBook Packages: Computer ScienceComputer Science (R0)