Skip to main content

Multimodal Expression in Communicative Functions, Gestures, Vocalizations, and the Contribution of Early Musicality

  • Chapter
  • First Online:
Moving and Interacting in Infancy and Early Childhood
  • 266 Accesses

Abstract

The communicative functions with which infants and pregrammatical speaking children use signs to interact with others have been the subject of different classifications, usually concentrated in one modality (gesture, word) or in bimodal combinations of gesture and word, or oriented by a particular criterion (for example, some were conceived thinking of types of signs, others of the illocutionary acts of utterances, etc.). With an integrative aim, we propose a classification of the communicative functions of oral, gestural, and combined modalities, also suitable to include signs of other types and especially designed to deal with pregrammatical expressive resources. The system of categories is forged in attention both to the structural scheme of the communicative act, bearing in mind its actors, roles, and component elements, and to the motives or intentions that lead pregrammatical infants and children to interact with others through behaviors performed, purposively, for the other to decode. This double consideration, together with the longitudinal recording of a single case during the period of first word combinations, allowed both to include categories usually omitted in classifications for competent speakers (adults, older children) and to exclude others that still require the acquisition of prior enabling cognitive skills. Three new categories are also posited, two of them liminal to the communication phenomenon, linked to musicality and multimodality, which allow conceiving the development of the child’s semiosis as a continuity between precommunicative and later communicative dyadic behavioral skills.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    As in Chap. 7, some of the observations have been previously published in different contexts (Rodríguez, 2018, 2021) and are now offered in connection with others to delineate an overall assessment of multimodality in the beginnings of communicative semiosis.

  2. 2.

    In terms of completeness, the system of categories for communicative functions developed in F. G. Rodríguez (2021) and F. G. Rodríguez and Español (2019), which is the basis of Sect. 8.2.3, fails to consider all the important aspects of the act of communication. This system is limited, as has been pointed out in the previous chapter, to the oral and gestural modalities.

  3. 3.

    Note that the designation ‘representational’ here encompasses more than in Chap. 7, where it only included iconic gestures, leaving out deictic and arbitrary gestures.

  4. 4.

    From a similar point of view can also be consulted, focusing on ostensive and indexical gestures, Moreno-Núnez et al. (2020).

  5. 5.

    An applied work of Ninio and Wheeler’s category system, extended beyond verbal communications, can be found in Rivero (2010) and Rivero García (2012).

  6. 6.

    As in other chapters, observations indicate age using Piagetian nomenclature: 1; 2 (3), meaning one year, two months, three days. Double quotation marks appear for oral utterances and single quotation marks are used to indicate the mention of a term or phrase. The meaning of gestural, oral or bimodal utterances is entered within pairs of oblique lines /xxx/, with inner parentheses/xxx (yyy) xxx/recovering missing elements in the child’s utterance, required for its proper interpretation. The absence of one or more phonemes is indicated by an underscore: x_ xx (e.g., _anks/thanks/). The absence of a complete syllable is indicated by a hyphen: x-xx, -xxx, xxx- (e.g.: -nana/banana/).

  7. 7.

    It is to be understood that to speak of a multimodal metaphor, or of “metaphors we live by” (Lakoff and Johnson 1980/2003), suggesting that there is a metaphorizing cognitive lever that orders our elementary experience of the world, responds to an extra-linguistic and extra-semiotic concept of the term ‘metaphor’. From this approach, the metaphorical crosses all other expressive-communicative functions, because it is grounded at the level of the most fundamental experience. That there is a level of conversion between different cognitive domains (or modalities) and that this entails a certain platform for the a posteriori creation of linguistic metaphors, is a whole topic that cannot be discussed here. It is only asserted, according to the traditional notion of metaphor, that young children, with little lexicon and who speak by means of word juxtapositions, do not possess the capacity to deliberately create metaphors or to aspire to say things in an original way. A metaphor in the strict sense implies that something with a name can be meant in a different way. Consequently, metaphor requires, as a condition of possibility, the alternative of choosing between at least two expressive varieties. When the child designates some object in an inadequate but understandable way, the amount of appropriateness that this name may have, capable of producing surprise and even approval in the adult receiver, does not imply on the part of the sender that there is any will or savoir faire in the substitution carried out. The whole phenomenon can be explained, in the vast majority of examples, by means of an assimilation between the misnamed object and other similar ones, to a varying degree, of which the name is known. Therefore, it is not a case that the child proceeds through a sign selection, but, on the contrary, that he/she is obliged to proceed from his high vocabulary restrictions.

  8. 8.

    Although the older brother, Marco, did not participate in the research (outside of occasional intervention), we have records of some of his cognitive-communicative progress that serve to exemplify what we are discussing. At 2 years and 11 months, Marco used to proclaim “Now is the day” or “Now is the night”. On a certain family walk, during a dark night, he suddenly saw a brightly lit shop window, pointed there and said with certainty: “There the day”. It could be attributed to this expression a creative purpose, which, however, resents the possibility, more clearly, that in his concepts of day and night it is not yet a question of an hourly cycle, of the sun and the moon, but of diaphanousness and darkness. The day exists where it can be seen, even if it happens in the space of a room with lamps on. Are we before a metaphor, before an event of pure semiotic creativity, or before concepts that the child still has to polish?

  9. 9.

    Kendon (2004) separates between referential content gestures and pragmatic gestures. The latter have, on their part, different functions. They are modal, when they give a glimpse of the sender’s position; they concern the partitioning of stretches in the speech act (parsing function); they are interactive or interpersonal (regulating the turns in a conversation or identifying the addressee of a message); or they are performative. But performativity is not defined here in our own terms. According to Kendon, “In [some] cases a gesture may have a performative function, as when it is used to indicate the kind of speech act or interactional move a person is engaging in. For example, the palm-up-open hand may be used as a way of indicating that what the speaker is saying is ‘offered’ to the interlocutor, perhaps as an example or as a proposal” (Kendon, 2004, p. 159). But, as has been argued, the performative in this gesture can function as part of a subjective emotional manifestation, as a declarative, informative or conative utterance. The fact that performative gestures were by the same author called, in a previous text, ‘illocutionary marker gestures’ (Kendon, 1995), gives us confirmation that the idea of performativity is taken in a broad sense, different from the one adopted here.

  10. 10.

    The quantitative results of the research can be found in F. G. Rodríguez and Español (2019).

  11. 11.

    Support in the environment corresponds to a basic premise for the effective expressions of this stage. At the age of 18 months and with few words, Bruno’s brother, Marco, saw reappearing at the front door, welcomed by his mother, the person who the week before had started to work on the domestic chores. She greeted him, smiling, but Marco only managed to make a noise (a harsh, dull, intense throat sound) that none of the bystanders (father, mother, the maid) could associate with anything. Then he headed for his room, stopping every three or four steps to make sure he was being followed. He stopped before the closet doors, waited for the others, and opened them wide. Then he repeated that mysterious noise and pointed to the vacuum cleaner. Now it was clearer: seeing the maid, Marco had suddenly remembered that in previous days she had cleaned with that appliance. When he first made the noise, the vagueness of the utterance and the lack of context made it impossible to understand what later, through a sign combination, was a reference to the cleaning tool. With the gesture of pointing, the noise ̶ now associated with a particular object ̶ became onomatopoeia without any problem. The person of the maid had evoked in Marco a certain activity, and the two expressions (first the sound and then the gesture-sound combination) were a predication with the structure /she + that artifact/, which without any forcing can be equated/ to /she (used) that artifact/. /She/, because she was present, did not need to be signified. The redundancy between gesture and vocalization, both concerned with the same object by means of different ways of signifying it, was therefore laterally referred to the human figure that triggered the whole process.

  12. 12.

    The verbalization of this episode is no more than emphatic, but by marking, through gesture, what it was about, the tacit element could be understood with a pointing to the place where the well was being made. By the way, this is a combination that does not respond to the standard categories (as has been stressed in the previous chapter), because the signs are not redundant, nor do they imply, as in the supplementary combination, an attributive compound. In this case, the gesture highlights what is at issue and the word is a sign of exhortation (“Come on!” = /help me/).

  13. 13.

    It is appropriate to recall here the example from the previous chapter in which Bruno transfers from the oral to the gestural modality an element of the other’s intervention. The mother had told him: “It’s not a cookie, it is a noodle” and immediately clarified “You don’t eat it”. Bruno copied the message by taking the negation to the gestural modality and preserving at the speech level, as a question, only one word: “Eat it?”. This had been interpreted by us as a case in which the total input, perhaps a very demanding verbal production, had been decompressed and split into two modalities for the relief of the sign processing system. The example can here be inserted between the echo function of single final elements (the last word) and the longer reproductions that keep the modality of the input unaltered.

  14. 14.

    Tangentially it can be added, in order to make the relevance of these two categories visible in our results, that the other-function and the doubtful function had a high frequency and ranked third and fifth among the total categories. The sum of their two frequencies brings them to second place, only behind the informative function. This was a singular and somewhat uncomfortable fact, as it hinted that some central consideration could have been omitted in the coding. It was the same deficiency that Belinchón Carmona had pointed out in Dale’s categories (see above, Sect. 8.2). However, the fundamental categories of all the other codes consulted had been contemplated and others of our own had even been added. Modulation restrictions, coupled with the ambiguity of certain situations, must be counted as partially causal factors in these results. In any case, this problem began to mend when we took a closer look at the interventions within the other-function (see Sect. 8.3).

  15. 15.

    Bruno’s second Spanish word was “Mo-to” (motorbyke), here replaced to maintain the bisyllabicity.

  16. 16.

    This episode was observed by the father, who had also gone to pick up Bruno’s brother from kindergarten and was able to see it all first hand.

References

  • Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for linguists: A handy primer. Language and Linguistics Compass, 9(11), 437–451. https://doi.org/10.1111/lnc3.12168

    Article  PubMed  PubMed Central  Google Scholar 

  • Austin, J. L. (1962). How to do things with words. In J. O. Urmson & M. Sbisà (Eds.), The William James Lectures delivered at Harvard University in 1955. Clarendon Press.

    Google Scholar 

  • Ayala, M. A. M., & López, M. J. R. (2011). Functions of the pointing gesture in mothers and their 12 to 36-month-old children during everyday activities. The Spanish Journal of Psychology, 14(2), 619–629. https://doi.org/10.5209/rev_SJOP.2011.v14.n2.10

    Article  Google Scholar 

  • Bates, E., Camaioni, I., & Volterra, V. (1975). The acquisition of performatives prior to speech. Merrill-Palmer Quarterly, 21, 205–226.

    Google Scholar 

  • Belinchón Carmona, M. (1985). Adquisición y evaluación de las funciones pragmáticas del lenguaje: un estudio evolutivo [Acquisition and assesement of the pragmatic functions of language: An evolutive study]. Estudios de Psicología, 19−20, 35–49.

    Google Scholar 

  • Bühler, K. (1934/1979). Sprachtheorie [Teoría del lenguaje (J. Marías, trad.)]. Alianza.

    Google Scholar 

  • Cochet, H., & Vauclair, J. (2010). Pointing gestures produced by toddlers from 15 to 30 months: Different functions, hand shapes and laterality patterns. Infant Behavior and Development, 33, 431–441. https://doi.org/10.1016/j.infbeh.2010.04.009

    Article  PubMed  Google Scholar 

  • Crowder, E. M. (1996). Gestures at work in sense-making science talk. The Journal of the Learning Sciences, 5(3), 173–208.

    Article  Google Scholar 

  • Dale, P. (1980). Is early pragmatic development measurable? Journal of Child Language, 7, 1–12.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98.

    Article  Google Scholar 

  • Español, S. (2006). Possible precursors of private speech: Deictic, symbolic and aesthetic self-directed gestures. In: I. Montero (Ed.), Current research trends in private speech. Proceedings of the first international symposium on self-regulatory functions of language (pp. 121−124). Ediciones de la Universidad Autónoma de Madrid.

    Google Scholar 

  • Guevara, I., Moreno-Llanos, I., & Rodríguez, C. (2020). The emergence of gestures in the first year of life in the Infant School classroom. European Journal of Psychology of Education, 35, 265–287. https://doi.org/10.1007/s10212-019-00444-6

    Article  Google Scholar 

  • Jakobson, R. (1960). Linguistics and poetics. In T. A. Sebeok (Ed.), Style in language (pp. 350–377). MIT Press.

    Google Scholar 

  • Kendon, A. (1995). Gestures as illocutionary and discourse structure markers in southern Italian conversation. Journal of Pragmatics, 23, 247–279.

    Article  Google Scholar 

  • Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.

    Book  Google Scholar 

  • Kong, A. P. H., Law, S. P., Kwan, C. C. Y., Lai, C., & Lam, V. (2015). A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a database of speech and gesture (DoSaGE). Journal of Nonverbal Behavior, 39(1), 93–111. https://doi.org/10.1007/s10919-014-0200-6

    Article  PubMed  PubMed Central  Google Scholar 

  • Lakoff, G., & Johnson, M. (1980/2003). Metaphors we live by. University of Chicago Press.

    Google Scholar 

  • Liszkowski, U., Carpenter, M., Striano, T., & Tomasello, M. (2006). 12-and-18-months-olds point to provide information for others. Journal Child Language, 7(2), 173–187.

    Google Scholar 

  • Liszkowski, U., Carpenter, M., & Tomasello, M. (2007). Pointing out new news, old news and absent referents at 12 months of age. Developmental Science, 10(2), F1–F7. https://doi.org/10.1111/j.1467-7687.2006.00552.x

    Article  PubMed  Google Scholar 

  • Lucca, K., & Wilbourn, M. P. (2018). Communicating to learn: Infants’ pointing gestures result in optimal learning. Child Development, 89(3), 941–960. https://doi.org/10.1111/cdev.12707

    Article  PubMed  Google Scholar 

  • Mandler, J. M. (2004). The foundations of mind: Origins of conceptual thought. Oxford University Press.

    Google Scholar 

  • Mandler, J. M. (2008). On the birth and growth of concepts. Philosophical Psychology, 21, 207–230. https://doi.org/10.1080/09515080801980179

    Article  Google Scholar 

  • McNeill, D. (1992). Hand and mind. What gestures reveal about thought. The University of Chicago Press.

    Google Scholar 

  • Moreno-Núñez, A., Rodríguez, C., & Miranda-Zapata, E. (2020). Getting away from the point: The emergence of ostensive gestures and their functions. Journal of Child Language, 47(3), 556–578. https://doi.org/10.1017/S0305000919000606

    Article  PubMed  Google Scholar 

  • Murillo, E., & Belinchón, M. (2013). Patrones comunicativos multimodales en la transición a las primeras palabras: Cambios en la coordinación de gestos y vocalizaciones [Multimodal communicative patterns in the transition to first words: Changes in the coordination of gestures and vocalizations]. Infancia y aprendizaje, 36(4), 473–487.

    Article  Google Scholar 

  • Ninio, A., & Wheeler, P. (1984). Functions of speech in mother-infant interaction. Final Science Report to the United States-Israel Binational Science Foundation (BSF).

    Google Scholar 

  • Ninio, A., & Wheeler, P. (1988). A revised manual for classifying verbal communicative acts in mother-infant interaction. Coding manual distributed by the Department of Psychology, Hebrew University, Jerusalem, Israel.

    Google Scholar 

  • Ninio, A., Snow, C. E., Pan, B. A., & Rollins, P. R. (1994). Classifying communicative acts in children’s interactions. Journal of Communication Disorders, 27(2), 157–187.

    Article  Google Scholar 

  • Rivero, M. (2010). Maternal expression of communicative intentions and pragmatic fine tuning in early infancy. Infant Behavior and Development, 33(4), 373–386. https://doi.org/10.1016/j.infbeh.2010.04.002

    Article  PubMed  Google Scholar 

  • Rivero García, M. M. (2012). Form-function rules in the first words stage: A longitudinal study of two children. Anuario de Psicología, 42(1), 65–84.

    Google Scholar 

  • Rodríguez, C. (2009). The “circumstances” of gestures: Proto-interrogatives and private gestures. New Ideas in Psychology, 27(2), 288–303. https://doi.org/10.1016/j.newideapsych.2008.04.002

    Article  Google Scholar 

  • Rodríguez, F. G. (2018). La pragmática de la comunicación antes del lenguaje [The pragmatics of communication prior to language]. Revista IRICE, 35, 95–128.

    Google Scholar 

  • Rodríguez, F. G. (2021). Semántica y pragmática en la comunicación del niño pregramatical. Gesto y palabra antes de la articulación morfosintáctica [Semantics and pragmatics in the communication of the pregrammatical child. Gesture and speech before morphosyntactic articulation]. Teseo.

    Google Scholar 

  • Rodríguez, F. G., & Español, S. (2019). Communicative functions during the pre-grammatical stage: A case study/Las funciones comunicativas durante la etapa pregramatical: un estudio de caso. Infancia y Aprendizaje, 42(2), 413–458. https://doi.org/10.1080/02103702.2019.1585654

    Article  Google Scholar 

  • Sarriá, E., & Rivière, Á. (1991). Desarrollo cognitivo y comunicación intencional preverbal: un estudio longitudinal multivariado. Estudios de psicología, 12(46), 35–52.

    Article  Google Scholar 

  • Sarriá Sánchez, E. (1991). Observación de la comunicación intencional preverbal: un sistema de codificación basado en el concepto de categoría natural. Psicothema, 3(2), 359–380.

    Google Scholar 

  • Searle, J. R. (1969). Speech acts. Cambridge University Press.

    Book  Google Scholar 

  • Searle, J. R. (1976). The classification of illocutionary acts. Language in Society, 5, 1–25.

    Article  Google Scholar 

  • Serra, M., & Solé, R. (1995). Childes database. Romance language corpora. http://childes.psy.cmv.edu.

  • Serra, M., Serrat, E., Solé, R., Bel, A., & Aparici, M. (2000). La adquisición del Lenguaje [Language acquisition]. Ariel.

    Google Scholar 

  • Southgate, V., Van Maanen, C., & Csibra, G. (2007). Infant pointing: Communication to cooperate or communication to learn? Child Development, 78(3), 735–740. https://doi.org/10.1111/j.1467-8624.2007.01028.x

    Article  PubMed  Google Scholar 

  • Tomasello, M., Carpenter, M., & Liszkowski, U. (2007). A new look at infant pointing. Child Development, 78(3), 705–722. https://doi.org/10.1111/j.1467-8624.2007.01025.x

    Article  PubMed  Google Scholar 

  • Wagner, P., Malisz, Z., & Kopp, S. (2014). Gesture and speech in interaction: An overview. Speech Communication, 57, 209–232. https://doi.org/10.1016/j.specom.2013.09.008

    Article  Google Scholar 

  • Wittgenstein, L. (1953). Philosophical investigations. Basil Blackwell Ltd.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Rodríguez, F.G. (2022). Multimodal Expression in Communicative Functions, Gestures, Vocalizations, and the Contribution of Early Musicality. In: Español, S., Martínez, M., Rodríguez, F.G. (eds) Moving and Interacting in Infancy and Early Childhood. Springer, Cham. https://doi.org/10.1007/978-3-031-08923-7_8

Download citation

Publish with us

Policies and ethics