Abstract
This paper addresses an aspect of sign language (SL) recognition that has largely been overlooked in previous work and yet is integral to signed communication. It is the most comprehensive work to-date on recognizing complex variations in sign appearances due to grammatical processes (inflections) which systematically modulate the temporal and spatial dimensions of a root sign word to convey information in addition to lexical meaning. We propose a novel dynamic Bayesian network – the Multichannel Hierarchical Hidden Markov Model (MH-HMM)– as a modelling and recognition framework for continuously signed sentences that include modulated signs. This models the hierarchical, sequential and parallel organization in signing while requiring synchronization between parallel data streams at sign boundaries. Experimental results using particle filtering for decoding demonstrate the feasibility of using the MH-HMM for recognizing inflected signs in continuous sentences.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Polhemus 3Space User’s Manual. Polhemus, Colchester, VT (1991)
CyberGlove User’s Manual. Virtual Technologies Inc. (1995)
Bourlard, H., Dupont, S.: A new ASR approach based on independent processing and recombination of partial frequency bands. In: Proc. Int’l. Conf. on Spoken Language Processing, vol. 1, pp. 426–429 (1996)
Braffort, A.: Argo: An architecture for sign language recognition and interpretation. In: Proc. Gesture Workshop, pp. 17–30 (1996)
Brand, M., Oliver, N., Pentland, A.: Coupled hidden Markov models for complex action recognition. In: Proc. IEEE Int’l. Conf. Comp. Vision and Pattern Recogn., pp. 994–999. IEEE Computer Society Press, Los Alamitos (1997)
Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden Markov model: Analysis and applications. Machine Learning 32, 41–62 (1998)
Ghahramani, Z., Jordan, M.: Factorial hidden Markov models. In: Touretzky, D., Mozer, M., Hasselmo, M. (eds.) Proc. Conf. Advances in Neural Information Processing Systems, vol. 8, pp. 472–478. MIT Press, Cambridge, Mass (1995)
Gowdy, J., Subramanya, A., Bartels, C., Bilmes, J.: DBN based multi-stream models for audio-visual speech recognition. In: IEEE Intl. Conf. Acoustics, Speech and Signal Processing, IEEE Computer Society Press, Los Alamitos (May 2004)
Gravier, G., Potamianos, G., Neti, C.: Asynchrony modeling for audio-visual speech recognition. In: Proc. Human Language Technology Conf. (March 2002)
Klima, E., Bellugi, U.: The Signs of Language. Harvard Univ. Press, Cambridge, Mass. (1979)
Liddell, S.: Grammar, Gesture, and Meaning in American Sign Language. Cambridge Univ. Press, Cambridge (2003)
Murphy, K.: Dynamic Bayesian Networks: Representation, Inference and Learning. PhD thesis, UC Berkeley, Computer Science Division (2002)
Oliver, N., Horvitz, E., Garg, A.: Layered representations for human activity recognition. In: ICMI 2002. Proc. of the Fourth IEEE Int’l. Conf. on Multimodal Interfaces, IEEE Computer Society Press, Los Alamitos (2002)
Ong, S., Ranganath, S., Venkatesh, Y.: Understanding gestures with systematic variations in movement dynamics. Patt. Recog. 39, 1633–1648 (2006)
Poizner, H., Klima, E., Bellugi, U., Livingston, R.: Motion analysis of grammatical processes in a visual-gestural language. In: Proc. ACM SIGGRAPH/SIGART Interdisciplinary Workshop, pp. 271–292. ACM Press, New York (1983)
Sagawa, H., Takeuchi, M.: A method for analyzing spatial relationships between words in sign language recognition. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 197–210. Springer, Heidelberg (2000)
Valli, C., Lucas, C.: Linguistics of American sign language: a resource text for ASL users. Gallaudet Univ. Press, Washington, D.C. (1992)
Vogler, C.: American Sign Language Recognition: Reducing the Complexity of the Task with Phoneme-based Modeling and Parallel Hidden Markov Models. PhD thesis, Univ. of Pennsylvania (2003)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ong, S.C.W., Ranganath, S. (2007). A New Probabilistic Model for Recognizing Signs with Systematic Modulations. In: Zhou, S.K., Zhao, W., Tang, X., Gong, S. (eds) Analysis and Modeling of Faces and Gestures. AMFG 2007. Lecture Notes in Computer Science, vol 4778. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75690-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-75690-3_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-75689-7
Online ISBN: 978-3-540-75690-3
eBook Packages: Computer ScienceComputer Science (R0)