Movements and Holds in Fluent Sentence Production of American Sign Language: The Action-Based Approach

Abstract

The importance of bodily movements in the production and perception of communicative actions has been shown for the spoken language modality and accounted for by a theory of communicative actions (Cogn. Process. 2010;11:187–205). In this study, the theory of communicative actions was adapted to the sign language modality; we tested the hypothesis that in the fluent production of short sign language sentences, strong-hand manual sign actions are continuously ongoing without holds, while co-manual oral expression actions (i.e. sign-related actions of the lips, jaw, and tip of the tongue) and co-manual facial expression actions (i.e. actions of the eyebrows, eyelids, etc.), as well as weak-hand actions, show considerable holds. An American Sign Language (ASL) corpus of 100 sentences was analyzed by visually inspecting each frame-to-frame difference (30 frames/s) for separating movement and hold phases for each manual, oral, and facial action. Excluding fingerspelling and signs in sentence-final position, no manual holds were found for the strong hand (0%; the weak hand is not considered), while oral holds occurred in 22% of all oral expression actions and facial holds occurred for all facial expression actions analyzed (100%). These results support the idea that in each language modality, the dominant articulatory system (vocal tract or manual system) determines the timing of actions. In signed languages, in which manual actions are dominant, holds occur mainly in co-manual oral and co-manual facial actions. Conversely, in spoken language, vocal tract actions (i.e. actions of the lips, tongue, jaw, velum, and vocal folds) are dominant; holds occur primarily in co-verbal manual and co-verbal facial actions.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. 1.

    Ambadar Z, Schooler J, Cohn JF. Deciphering the enigmatic face: the importance of facial dynamics to interpreting subtle facial expressions. Psychol Sci. 2005;16:403–10.

    PubMed  Article  Google Scholar 

  2. 2.

    Bauer D, Kannampuzha J, Kröger BJ. Articulatory speech re-synthesis: profiting from natural acoustic speech data. In: Esposito A, Vich R, editors. Cross-modal analysis of speech, gestures, gaze and facial expressions, LNAI 5641. Berlin: Springer; 2009. p. 344–55.

    Google Scholar 

  3. 3.

    Boston-200-Sentences-ASL-Corpus of the National Center for Sign Languages and Gesture Resources at Boston University. 2000. http://www.bu.edu/asllrp/cslgr/.

  4. 4.

    Browman C, Goldstein L. Articulatory gestures as phonological units. Phonology. 1989;6:201–51.

    Article  Google Scholar 

  5. 5.

    Browman C, Goldstein L. Articulatory phonology: an overview. Phonetica. 1992;49:155–80.

    PubMed  Article  CAS  Google Scholar 

  6. 6.

    Cohn JF. Foundations of human computing: facial expression and emotion. In: Huang TS, Nijholt A, Pantic M, Pentland A, editors. Artifical intelligence for human computing (LNAI 4451). Berlin: Springer; 2007. p. 1–16.

    Google Scholar 

  7. 7.

    Cohn JF, Ambadar Z, Ekman P. Observer-based measurement of facial expression with the facial action coding system. In: Coan JA, Allen JJB, editors. Handbook of emotion elicitation and assessment. New York: Oxford University Press; 2007. p. 203–21.

    Google Scholar 

  8. 8.

    Dreuw P, Rybach D, Deselaers T, Zahedi M, Ney H. Speech Recognition Techniques for a Sign Language Recognition System. 2007. Proceedings of Interspeech 2007 (Antwerp, Belgium). pp. 2513–2516.

  9. 9.

    Ekman P, Friesen WV. Measuring facial movement. Environ Psychol Nonverbal Behavior. 1976;1:56–75.

    Article  Google Scholar 

  10. 10.

    Ekman P, Friesen WV. Facial action coding system. Palo Alto, CA: Consulting Psychologists Press; 1978.

    Google Scholar 

  11. 11.

    Emmorey K. Language, cognition, and the brain: insights from sign language research. Lawrence Erlbaum Associates; 2002.

  12. 12.

    Fontana S. Mouth actions as gesture in sign language. Gesture. 2008;8:104–23.

    Article  Google Scholar 

  13. 13.

    Goldin-Meadow S. Hearing gesture. Cambridge, London: Belknap & Harvard University Press; 2003.

    Google Scholar 

  14. 14.

    Goldstein L, Byrd D, Saltzman E. The role of vocal tract action units in understanding the evolution of phonology. In: Arbib MA, editor. Action to language via the mirror neuron system. Cambridge: Cambridge University Press; 2006. p. 215–49.

    Google Scholar 

  15. 15.

    Goldstein L, Pouplier M, Chen L, Saltzman L, Byrd D. Dynamic action units slip in speech production errors. Cognition. 2007;103:386–412.

    PubMed  Article  Google Scholar 

  16. 16.

    Kendon A. Language and gesture: unity or duality? In: McNeill D, editor. Language and gesture. Cambridge: Cambridge University Press; 2000. p. 47–63.

    Google Scholar 

  17. 17.

    Kendon A. Gesture: visible action as utterance. New York: Cambridge University Press; 2004.

    Google Scholar 

  18. 18.

    Klima E, Bellugi U. The signs of language. Cambridge, MA: Harvard University Press; 1979.

    Google Scholar 

  19. 19.

    Kopp S, Wachsmuth I. Synthesizing multimodal utterances for conversational agents. J Comput Animat Virtual Worlds. 2004;15:39–51.

    Article  Google Scholar 

  20. 20.

    Kröger BJ, Birkholz P. A gesture-based concept for speech movement control in articulatory speech synthesis. In: Esposito A, Faundez-Zanuy M, Keller E, Marinaro M, editors. Verbal and nonverbal communication behaviours, LNAI 4775. Berlin: Springer; 2007. p. 174–89.

    Google Scholar 

  21. 21.

    Kröger BJ, Birkholz P. Articulatory Synthesis of Speech and Singing: State of the Art and Suggestions for Future Research. In: Esposito A, Hussain A, Marinaro M, editors. Multimodal signals: cognitive and algorithmic issues. LNAI 5398. Berlin: Springer; 2009. p. 306–19.

    Google Scholar 

  22. 22.

    Kröger BJ, Kannampuzha J, Neuschaefer-Rube C. Towards a neurocomputational model of speech production and perception. Speech Commun. 2009;51:793–809.

    Article  Google Scholar 

  23. 23.

    Kröger BJ, Kopp S, Lowit A. A model for production, perception, and acquisition of actions in face-to-face communication. Cogn Process. 2010;11:187–205.

    Google Scholar 

  24. 24.

    Lausberg H, Sloetjes H. Coding gestural behavior with the NEUROGES-ELAN system. Behav Res Meth. 2009;41(3):841–9.

    Article  Google Scholar 

  25. 25.

    Liberman AM, Mattingly IG. The motor theory of speech perception revised. Cognition. 1985;21:1–36.

    PubMed  Article  CAS  Google Scholar 

  26. 26.

    Liddell SK, Johnson RE. American sign language: the phonological base. Sign Lang Stud. 1989;64:195–277.

    Google Scholar 

  27. 27.

    Liddell SK, Metzger M. Gesture in sign language discourse. J Pragmat. 1998;30:657–97.

    Article  Google Scholar 

  28. 28.

    Liddell SK. Grammar, gesture and meaning in American sign language. New York: Cambridge University Press; 2003.

    Google Scholar 

  29. 29.

    McNeill D. Hand and mind: what gestures reveal about thought. Chicago: University of Chicago Press; 1992.

    Google Scholar 

  30. 30.

    McNeill D. Gesture and thought. Chicago: University of Chicago Press; 2005.

    Google Scholar 

  31. 31.

    McNeill D, Quek F, McCullough K-E, Duncan SD, Furuyama N, Bryll R, Ansari R. Catchments, prosody and discourse. Gesture. 2001;1(1):9–33.

    Article  Google Scholar 

  32. 32.

    Perlmutter DM. Sonority and syllable structure in American sign language. Linguist Inquiry. 1992;23:407–42.

    Google Scholar 

  33. 33.

    Saltzman E, Byrd D. Task-dynamics of gestural timing: Phase windows and multifrequency rhythms. Hum Mov Sci. 2000;19:499–526.

    Article  Google Scholar 

  34. 34.

    Sandler W. Symbolic symbolization by hand and mouth in sign language. Semiotica. 2009;174:241–75.

    Article  Google Scholar 

  35. 35.

    Schmidt KL, Ambadar Z, Cohn JF, Reed LI. Movement differences between deliberate and spontaneous facial expressions: zygomaticus major action in smiling. J Nonverbal Behav. 2006;30:37–52.

    PubMed  Article  Google Scholar 

  36. 36.

    Schmidt KL, Bhattacharya S, Denlinger R. Comparison of deliberate and spontaneous facial movement in smiles and eyebrow raises. J Nonverbal Behav. 2009;33:35–45.

    PubMed  Article  Google Scholar 

  37. 37.

    Schmidt KL, Cohn JF, Tian Y. Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 2003;65:49–66.

    PubMed  Article  Google Scholar 

  38. 38.

    Stokoe WC (1960) Sign language structure: An outline of the visual communication systems of the American Deaf, Studies in Linguistics Occasional Paper 8, University of Buffalo.

  39. 39.

    Tian YL, Kanade T, Cohn JF. Facial expression analysis. In: Li SZ, Jain AK, editors. Handbook of face recognition. New York: Springer; 2005. p. 247–75.

    Google Scholar 

  40. 40.

    Valli C, Lucas C. Linguistics of American sign language. An Introduction. Washington: Gallaudet University Press; 2000.

    Google Scholar 

  41. 41.

    Vanger P, Hoenlinger R, Haken H. Computer aided generation of prototypical facial expressions of emotion. Methods of Psychological Research Online. 1998. Vol. 3, No. 1. http://www.dgps.de/fachgruppen/methoden/mpr-online.

  42. 42.

    Wilcox S, Morford JP. Empirical methods in signed language research. In: Gonzalez-Marquez M, Mittelberg I, Coulson S, Spivey MJ, editors. Methods in cognitive linguistics. Amsterdam/Philadelphia: John Benjamins; 2007. p. 171–200.

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the German Research Council DFG grant Kr 1439/13-1 and grant Kr 1439/15-1. We thank the two anonymous reviewers for their helpful comments on earlier versions of this paper, and we thank Cigdem Capaat for labeling the ASL corpus.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Bernd J. Kröger.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kröger, B.J., Birkholz, P., Kannampuzha, J. et al. Movements and Holds in Fluent Sentence Production of American Sign Language: The Action-Based Approach. Cogn Comput 3, 449–465 (2011). https://doi.org/10.1007/s12559-010-9071-2

Download citation

Keywords

  • Sign language
  • Spoken language
  • Action theory
  • Manual actions
  • Oral actions
  • Facial actions
  • Co-verbal actions
  • Co-manual actions
  • Movements
  • Holds