Towards a Visual Speech Learning System for the Deaf by Matching Dynamic Lip Shapes

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7382)


In this paper we propose a visual-based speech learning framework to assist deaf persons by comparing the lip movements between a student and an E-tutor in an intelligent tutoring system. The framework utilizes lip reading technologies to determine if a student learns the correct pronunciation. Different from conventional speech recognition systems, which usually recognize a speaker’s utterance, our speech learning framework focuses on recognizing whether a student pronounces are correct according to an instructor’s utterance by using visual information. We propose a method by extracting dynamic shape difference features (DSDF) based on lip shapes to recognize the pronunciation difference. The preliminary experimental results demonstrate the robustness and effectiveness of our approach on a database we collected, which contains multiple persons speaking a small number of selected words.


Lip Reading Speech Learning Dynamic Shape Difference Features Deaf people 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Awad, S.: The Application of Digital Speech Processing to Stuttering Therapy. IEEE Instrumentation and Measurement (1997)Google Scholar
  2. 2.
    Chen, S., Tian, Y., Liu, Q., Metaxas, D.: Segment and Recognize Expression Phase by Fusion of Motion Area and Neutral Divergence Features. In: IEEE Int’l Conf. on Automatic Face and Gesture Recognition, AFGR (2011)Google Scholar
  3. 3.
    Chen, S., Tian, Y., Liu, Q., Metaxas, D.: Recognizing Expressions from Face and Body Gesture by Temporal Normalized Motion and Appearance Features. In: IEEE Int’l Conf. Computer Vision and Pattern Recognition Workshop for Human Communicative Behavior Analysis, CVPR4HB (2011)Google Scholar
  4. 4.
    Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active Shape Models – Their Training and Application. Computer Vision and Image Understanding (1995)Google Scholar
  5. 5.
    Hailpern, J., Karahalios, K., DeThorne, L., Halle, J.: Encouraging Speech and Vocalization in Children with Autistic Spectrum Disorder. In: Workshop on Technology in Mental Health, CHI 2008 (2008)Google Scholar
  6. 6.
    Lavagetto, F.: Converting speech into lip movements: a multimedia telephone for hard of hearing people. IEEE Transactions on Rehabilitation Engineering 3(1), 90–102 (1995)CrossRefGoogle Scholar
  7. 7.
    Marschark, M., Sapere, P., Convertino, C., Mayer, C., Wauters, L., Sarchet, T.: Are deaf students’ reading challenges really about reading? American Annals of the Deaf 154(4), 357-176 (2009)Google Scholar
  8. 8.
    Matthews, I., Cootes, T., Bangham, J., Cox, S., Harvey, R.: Extraction of visual features for lipreading. TPAMI 24(2), 198–213 (2002)CrossRefGoogle Scholar
  9. 9.
    Potamianos, G., Neti, C., Gravier, G., Garg, A., Senior, A.: Recent Advances in the Automatic Recognition of Audio-Visual Speech. Proceedings of the IEEE 91(9), 1306–1326 (2003)CrossRefGoogle Scholar
  10. 10.
    Rahman, M., Ferdous, S., Ahmed, S.: Increasing Intelligibility in the Speech of the Autistic Children by an Interactive Computer Game. In: IEEE International Symposium on Multimedia (2010)Google Scholar
  11. 11.
    Riella, R., Linarth, A., Lippmann, L., Nohama, P.: Computerized System to Aid Deaf Children in Speech Learning. In: IEEE EMBS International Conference (2001)Google Scholar
  12. 12.
    Schipor, O., Pentiuc, S., Schipor, M.: Towards a Multimodal Emotion Recognition Framework to Be Integrated in a Computer Based Speech Therapy System. In: IEEE Conference on Speech Technology and Human Computer Dialogue, SpeD (2011)Google Scholar
  13. 13.
    Wei, Y.: Research on Facial Expression Recognition and Synthesis, Master Thesis (2009) Software available at
  14. 14.
    Zhao, G., Barnard, M., Pietikainen, M.: Lipreading with local spatialtemporal descriptors. TMM 11(7), 1254–1265 (2009)Google Scholar
  15. 15.
    Zhou, Z., Zhao, G., Pietikainen, M.: Toward a Practical Lipreading System. In: CVPR (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Department of Electrical EngineeringThe City College, City University of New YorkNew YorkUSA

Personalised recommendations