Structural Analysis of Co-verbal Deictic Gesture in Multimodal Dialogue Systems
From the time it is decided to develop communication between man and machine which essentially relies on spoken (natural) language, it seems reasonable to take gesture into account as a possible complement to voice. As a matter of fact, many gestures have a semiotic function which facilitate the communication act, especially in its referential component. Among the roles played by semiotic gesture, the deictic function is undoubtedly a major one when considering graphical units of human computer dialogues. The study we present can be situated in the framework of the interpretation of deictic gesture as a complement of a spoken utterance. We have identified three parts in the shape of an elementary designation gesture: the first part corresponds to hand or device positioning (preliminary phase), the second one corresponds to the designation act (effective phase) and the last one to the hand backward movement (final phase). Not surprisingly, when several designation acts appear in one single gesture, co-articulation phenomena can be observed. This consists in a combination of final phases with preliminary phases of the consecutive elementary designations. Among these three parts, only the second one is informative in the sense that it gives some information about the spatial location of the referent. The difficulty is to isolate this meaningful part. In order to resolve this problem, we have developed a method to analyse gestural trajectories provided by a 2D or 3D device, which allows us to identify the effective phase. This analysis is based on the notion of singularity which consists in studying different parameters of the trajectory (speed, curvature etc.) so as to establish the presence of relevant segmentation clues.
Unable to display preview. Download preview PDF.
- B. Gaiffe and L. Romary. Ce matin, il pleuvait, ou la référence à l’épreuve du temps. Langages, Temps, Référence et Inférence par Jacques Moeschler, 112, December 1993.Google Scholar
- D. McNeill. Hand and Mind: What gestures reveal about thought. University of Chicago Press, 1992.Google Scholar
- C. Mignot, C. Valot, and N. Carbonell. An experimental study of future “natural” multimodal human-computer interaction. In INTERCHI ‘93, April 1993.Google Scholar
- B. Rimé and L. Schiaratura. Gesture and speech. In R. S. Feldman and B. Rimé, editors, Fundamentals of nonverbal behavior. Cambridge University Press, 1991.Google Scholar
- W. Wahlster. User and discourse models for multimodal communication. In J. Sullivan and S. Tymler, editors, Intelligent user interface. Addison-Wesley, 1991.Google Scholar