Abstract
We describe a hybrid speechreading system that is based on a Manifold Learning technique, on Neural Networks, and on Hidden Markov Models. Manifold Learning is a technique for representing and learning smooth nonlinear lowdimensional surfaces embedded in highdimensional abstract feature spaces. The technique is capable of determining the structure of the surface and of finding the closest manifold point to a given query point. We use this technique to learn the “space of lips”. The learned manifold is used for tracking and extracting the lips, for interpolating between frames in an image sequence and for providing features for recognition. The hybrid speechreading system based on this learned lip manifold, connectionist phone classifier, and Hidden Markov Models significantly improves the performance of acoustic speech recognizers in degraded environments. We also present preliminary results on a purely visual lip reader and work in progress on a new face finding technique.
Chapter PDF
Similar content being viewed by others
Keywords
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1996 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Bregler, C., Omohundro, S.M., Shi, J., Konig, Y. (1996). Towards a Robust Speechreading Dialog System. In: Stork, D.G., Hennecke, M.E. (eds) Speechreading by Humans and Machines. NATO ASI Series, vol 150. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-13015-5_31
Download citation
DOI: https://doi.org/10.1007/978-3-662-13015-5_31
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-08252-8
Online ISBN: 978-3-662-13015-5
eBook Packages: Springer Book Archive