Abstract
Language scientists have broadly addressed the problem of explaining how language users recognize the kind of speech act performed by a speaker uttering a sentence in a particular context. They have done so by investigating the role played by the illocutionary force indicating devices (IFIDs), i.e., all linguistic elements that indicate the illocutionary force of an utterance. The present work takes a first step in the direction of an experimental investigation of non-verbal IFIDs because it investigates the role played by facial expressions and, in particular, of upper-face action units (AUs) in the comprehension of three basic types of illocutionary force: assertions, questions, and orders. The results from a pilot experiment on production and two comprehension experiments showed that (1) certain upper-face AUs seem to constitute non-verbal signals that contribute to the understanding of the illocutionary force of questions and orders; (2) assertions are not expected to be marked by any upper-face AU; (3) some upper-face AUs can be associated, with different degrees of compatibility, with both questions and orders.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Much work has been done also in the field of robotics and of human/robot interaction; see, for instance Granström, House & Beskow (2002).
On the idea that humans express and recognize emotion on the basis of multiple context-sensitive modalities where speech and facial expressions constitute two relevant clues that often interact in this process see also Emerich et al. (2009).
Sadock (1974) has provided a general scheme aimed at explaining the structure of illocutionary acts. According to Sadock, human language is characterized by three crucial functions: representing the world, altering society, expressing emotions. On the basis of this assumption, he argues that identifiable in every language are three distinctive sentence types that are associated with these constitutive functions: the declarative type, the interrogative type, and the imperative type. Speech acts can therefore be classified into three main categories: assertions, questions, and requests.
To be noted is that in its spoken form, English can sometimes be as flexible as Italian in the interpretation of the modality of an utterance; e.g., "Marco’s studying!" vs. "Marco’s studying?".
That in Italian the three types of speech act can be uttered by the very same phonological sequence is also due to a phonological ambiguity (homophony) of the verb studia, which means both a third person indicative (Marco studies) and a second person imperative (Marco, study!).
The fact that Italian interrogative and imperative clauses can exhibit a linear word order that is prima facie identical to that of declarative clauses is a consequence of three well-known syntactic properties of Italian: (1) overt movement of the inflected lexical verb to C(P), (2) subject pro-drop, (3) a rich clausal “Left Periphery” (Rizzi, 1997). Since (1) V rises to a position in the CP domain in overt syntax and (2) the subject position (Spec, TP) can be occupied by a silent pro, as soon as (3) an optional DP is merged higher up in a topic position, the linear order will be “(DP) – V – pro,” with pronunciation of only the DP and the V at spell-out.
It might be argued that a way to develop an expectation about the compatibility of different AUs with different illocutionary forces would be to look at the emotions that the seminal work of Ekman & Friesen (1978), is typically associated with the AUs and those that are considered compatible with the different illocutionary forces under examination. The reason why it is hard to elaborate such an expectation, however, is that the fact that a certain illocutionary force entails a particular emotion (e.g., an order might express anger) does not directly imply that an AU conveying the same emotion can be considered compatible with this illocutionary force. As shown by Shah et al. (2013), in fact, emotions recognition that jointly uses facial expressions and speech represents a controversial issue: if, on the one hand, talking provides clues about the emotional state of the speaker, on the other, it distorts her expression of emotion on the face. Shah et al.’s experimental results show in particular that emotion prediction in the presence of speech and facial AUs is less accurate when the person is talking than it is in a silent condition. In other words, the emotion conveyed by the joint production of a certain illocutionary force with a particular action unit is expected to be the result of an interaction between the two modalities instead of a mere sum of the two components.
According to Searle (1975), they possess different preparatory conditions.
Heritage (2012), however, has argued that the questioner is not on a par with, but on a lower (epistemic) level than, the question addressee.
We are aware that, although the content conveyed by three different illocutionary forces was realized with the very same linguistic form (e.g., “Marco studia”), there might have been an effect of the semantic content of the different target sentences in the production of the speech act. A sentence conveying a more impositive content like, for instance “You are on punishment,” although its Italian translation can convey assertions, questions, or orders with the same linguistic form (i.e., Ita. Tr. “Sei in castigo”) is usually expected to be interpreted as an order (even if clearly other possible interpretations are compatible). However, we decided to avoid controlling the variable of the content of the sentences used in our experiment for two reasons: (1) in a first exploratory experimental study we preferred to leave the item as ecological as possible; (2) we intentionally admitted a potential interference of the semantic content in order to verify whether there was an effect of the illocutionary force type on the AU production even in cases where the propositional content expressed by the sentence prompted the association of a certain illocutionary force. However, this effect was controlled by including the content of the sentence as a random effect in statistical models.
During the development of the experimental material, the FACS coder who helped the actors in production of the pictures representing the different AUs noted a substantial difficulty in the production of AU1, which seems to be an AU type requiring a specific training in order to be produced voluntarily.
AU2 was produced unilaterally for the very same reason.
As filler illocutionary types we distributed two tokens across the items for each one of the following types of illocutionary forces: Advice, Vow, Bet, Felicitation, Wish, Recommend, Agree, Exhortation, Promise, Announcement, Compliment, Concession, Supplication, Plea, Appreciate, Deduction.
To be noted is that in spoken language many aspects like the tone of voice and prosody contribute to the understanding of the illocutionary force of a speech act. In our research, we decided to use linguistic stimuli in a written form in order to investigate whether specific facial expressions are associated with specific illocutionary forces independently of prosody or other verbal or non-verbal IFIDs. Further research should be conducted in order to determine whether different prosodic realizations associated with a certain facial expression may affect the interpretation of the illocutionary force of an utterance. We thank an anonymous reviewer for this suggestion. It should also be noted that our study concerns the relation between facial expressions and illocutionary forces and not between facial expressions and sentence modality. In other words, we deal with the interpretation of “utterances” not of “sentences.” For this reason, although the items were presented in a written form, the various scenarios were presented as fictional ordinary verbal interactions, and participants were required to evaluate the different sentences as if they were uttered in a real verbal interaction.
This is an interpretation of the results about AU4 + 7 that requires further investigation. Research on the relation between emotions and illocutionary forces is in fact still on the way to be conducted. Moreover, there is disagreement about what kinds of emotions are conveyed by different AUs.
According to Brown and Levinson (1987), a face-threatening act is an act that threatens either the speaker’s face or the hearer’s and they may regard either the positive or the negative face.
References
Aikhenvald AY (2010) Imperatives and commands. Oxford University Press, Oxford
Allwood J, Cerrato L, Jokinen K, Navarretta C, Paggio P (2005) The MUMIN annotation scheme for feedback, turn management and sequencing. In: Gothenburg papers in theoretical linguistics 92: Proceedings from the second Nordic conference on multimodal communication. Göteborg University, Göteborg, Sweden, pp 91–109
Alston WP (2000) Illocutionary acts and sentence meaning. Cornell University Press, Ithaca
Argyle M, Cook M (1976) Gaze and mutual gaze. Cambridge University Press, Cambridge
Austin J (1962) How to do things with words. Oxford University Press, Oxford
Bach K (2006) Speech acts and pragmatics. In: Devitt M, Hanley R (eds) Blackwell guide to the philosophy of language. Blackwell, Oxford, pp 147–167
Bartels C (1999) The intonation of English statements and questions. A compositional interpretation. Garland Publishing Inc., New York
Braun D (2011) Implicating questions. Mind Lang 26(5):574–595. doi:10.1111/j.1468-0017.2011.01431.x
Brown P, Levinson S (1987) Politeness. Some universals in language usage. Cambridge University Press, Cambridge
Condoravdi C, Lauer S (2011) Performative verbs and performative acts. In: Reich I, Horch E, Pauly D (eds) Proceedings of Sinn & Bedeutung, vol. 15 (1–15). Universaar – Saarland University Press, Saarbrücken
Davies E (1986) The English imperative. Croom Helm, London
Dresner E, Herring SC (2010) Functions of the nonverbal in CMC: emoticons and illocutionary force. Commun Theory 20(3):249–268. doi:10.1111/j.1468-2885.2010.01362.x
Ekman P (1979) About brows: emotional and conversational signals. In: von Cranach M, Foppa K, Lepenies W, Ploog D (eds) Human ethology. Cambridge University Press, Cambridge, pp 169–248
Ekman P, Friesen WV (1978) Manual for the facial action coding system. Consulting Psychologists Press, Palo Alto
Emerich S, Lupu E, Apatean A (2009) Emotions recognition by speechand facial expressions analysis. In: Signal processing conference, 2009 17th European. IEEE Computer Society, Washington, pp 1617–1621
Esposito A, Esposito AM, Martone R, Müller V, Scarpetta G (eds) (2010) Towards autonomous, adaptive, and context-aware multimodal interfaces: theoretical and practical issues. Springer, NewYork
Goffman E (1967) Interaction ritual: essays in face to face behavior. Aldine Transaction, New Brunswick
Granström B, House D, Beskow J (2002) Speech and gestures for talking faces in conversational dialogue systems. In: Granström B, House D, Karlsson I (eds) Multimodality in language and speech systems. Kluwer Academic Publishers, Dordrecht, pp 209–241
Green MS (2000) Illocutionary force and semantic content. Linguist Philos 23(5):435–473. doi:10.1023/A:1005642421177
Green M (2013) Assertions. In: Sbisà M, Turner K (eds) Handbook of pragmatics, vol II., Pragmatics of speech actionsDe Gruyter-Mouton, Berlin, pp 387–410
Hanks PW (2007) The content–force distinction. Philos Stud 134(2):141–164. doi:10.1007/s11098-007-9080-5
Heritage J (2012) Epistemics in action: action formation and territories of knowledge. Res Lang Soc Interact 45:1–29
Jary M (2007) Are explicit performatives assertions? Linguist Philos 30(2):207–234. doi:10.1007/s10988-007-9015-9
Jokinen K, Nishida M, Yamamoto S (2010) On eye-gaze and turn-taking. In: Proceedings of the 2010 workshop on Eye gaze in intelligent human machine interaction. ACM, New York, pp 118–123
Keltner D (1995) Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame. J Pers Soc Psychol 68(3):441–454. doi:10.1037/0022-3514.68.3.441
Kendon A (1995) Gestures as illocutionary and discourse structure markers in Southern Italian conversation. J Pragmat 23(3):247–279. doi:10.1016/0378-2166(94)00037-F
Kendon A (2004) Gesture: visible action as utterance. Cambridge University Press, Cambridge
Kissine M (2013) From utterances to speech acts. Cambridge University Press, Cambridge
Kohler CG, Turner T, Stolar NM, Bilker WB, Brensinger CM, Gur RE, Gur RC (2004) Differences in facial expressions of four universal emotions. Psychiatry Res 128(3):235–244. doi:10.1016/j.psychres.2004.07.003
Konig E, Siemund P (2007) Speech act distinctions in grammar. In: Shopen T (ed) Language typology and syntactic description. Cambridge University Press, Cambridge, pp 276–324
Levinson S (1983) Pragmatics. Cambridge University Press, Cambridge
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 5588(264):746–748. doi:10.1038/264746a0
Oertel C, Wlodarczak M, Edlund J, Wagner P, Gustafson J (2012) Gaze patterns in turn-taking. In: Proceedings of Interspeech 2012. Portland, OR
Pelachaud C, Prevost S (1994) Sight and sound: generating facial expressions and spoken intonation from context. In: Proceedings of the 2nd workshop on speech synthesis. New Paltz, New York, pp 216–219
Pelachaud C, Carofiglio V, De Carolis B, de Rosis F, Poggi I (2002) Embodied contextual agent in information delivering application. In: Proceedings of the first international joint conference on autonomous agents and multiagent systems: Part 2. ACM, New York, pp 758–765
Pierrehumbert J, Hirschberg J (1990) The meaning of intonational contours in the interpretation of discourse. In: Cohen P, Morgan J, Pollack M (eds) Intentions in communication. MIT Press, Cambridge, pp 271–311
Poggi I (2007) Mind, hands, face and body. A goal and belief view of multimodal communication. Weidler Buchverlag, Berlin
Poggi I, Pelachaud C (1998) Performative faces. Speech Commun 26(1):5–21. doi:10.1016/S0167-6393(98)00047-8
Poggi I, Pelachaud C (2001) The meanings of gaze in animated faces. In: McKevitt P, Nuàllain S, Mulvihill C (eds) Language, vision and music. John Benjamins, Amsterdam, p 2001
Recanati F (2013) Content, mood, and force. Philos Compass 8(7):622–632. doi:10.1111/phc3.12045
Rizzi L (1997) On the position “int(errogative)” in the left periphery of the clause. In: Cinque G, Salvi G (eds) Current studies in Italian syntax. Elsevier, Amsterdam, pp 287–296
Sadock JM (1974) Toward a linguistic theory of speech acts. Academic Press, New York
Sbisà M (1987) Indicative mood, illocutionary force, and truth: some points for discussion. LAUD - Linguistic Agency University of Duisburg-Essen, Duisburg
Sbisà M (2007) How to read Austin. Pragmatics 17(3):461. doi:10.1075/prag.17.3.06sbi
Searle JR (1968) Austin on locutionary and illocutionary acts. Philos Rev 77(4):405–424. doi:10.2307/2183008
Searle J (1969) Speech acts. Cambridge University Press, Cambridge
Searle JR (1975) Indirect speech acts. In: Cole P, Morgan JL (eds) Syntax and semantics, vol 3., Speech actsAcademic Press, New York, pp 59–82
Searle JR, Vanderveken D (1985) The foundations of illocutionary logic. Cambridge University Press, Cambridge
Shah M, Cooper DG, Cao H, Gur RC, Nenkova A, Verma R (2013) Action unit models of facial expression of emotion in the presence of speech. In: Affective computing and intelligent interaction (ACII), 2013 Humaine association conference. IEEE Computer Society, Washington, pp 49–54
Stampe DW (1975) Meaning and truth in the theory of speech acts. In: Cole P, Morgan JL (eds) Speech acts. Academic Press, New York, pp 1–39
Acknowledgements
This research has been funded by the Italian Ministry of Education, University and Research within the three-year project SIR_2014 - EXPRESS directed by Filippo Domaneschi, project code RBSI147WM0.
Authors’ contribution
F.D. designed the research and prepared the experimental material, M.P. conducted the research and analyzed the data, C.C. helped with the statistics, F.D. wrote the paper, M.P. and C.C. revised the manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Handling editor: Stefan Kopp (Bielefeld University);
Reviewers: Nick Campbell (Trinity College Dublin), Isabella Poggi (Roma Tre University).
Appendices
Appendix 1: Experiment 1
See Table 6.
Appendix 2: Experiment 2
See Table 7.
Rights and permissions
About this article
Cite this article
Domaneschi, F., Passarelli, M. & Chiorri, C. Facial expressions and speech acts: experimental evidences on the role of the upper face as an illocutionary force indicating device in language comprehension. Cogn Process 18, 285–306 (2017). https://doi.org/10.1007/s10339-017-0809-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10339-017-0809-6